Use unlimited compilation rate limit for mustache templates in Ingest #64595
Comments
andrewkroh
added
>enhancement
needs:triage
jakelandis
added
:Data Management/Ingest Node
|
Pinging @elastic/es-core-features (:Core/Features/Ingest) |
jakelandis
added
the
:Core/Infra/Scripting
|
Pinging @elastic/es-core-infra (:Core/Infra/Scripting) |
|
@jdconrad @stu-elastic - I believe that the fix here would be in |
|
We plan on moving all non-search contexts to unlimited by 7.11. |
|
If the cache size is too small for the number of scripts that the client is running, then this can result in each script I have seen setting the compilation limit to a large number crash a cluster if the real problem is that the script cache is too small. (related support case: 00599383) |
|
All the compiled scripts are held onto by the ingest pipeline. I'll verify when I get to #62899 that templates are as well, but I believe they are. Upshot is cache is irrelevant for ingest pipelines, they are never recompiled, they stay with the pipeline for it's lifetime. |
|
Good to know, thanks! |
|
It may be necessary to delete (optional?) and upload the ingest pipeline again if the above error occurs repeatedly for a given pipeline and does not react to changing the compilation settings of the cluster. This is likely due to the error already being stored in the pipeline. |
|
If a pipeline definition hits a compilation limit, the pipeline will remember the error and will not function. The solution is to update the limit and then delete and add back the pipeline. |
Remove the compilation rate limit for ingest templates. Creates a new context, `ingest_template`, with an unlimited compilation rate limit. The `template` context is used in many places so it cannot be exempted from rate limits. Fixes: elastic#64595
* Script: no compile rate limit for ingest templates Remove the compilation rate limit for ingest templates. Creates a new context, `ingest_template`, with an unlimited compilation rate limit. The `template` context is used in many places so it cannot be exempted from rate limits. Fixes: #64595
Remove the compilation rate limit for ingest templates. Creates a new context, `ingest_template`, with an unlimited compilation rate limit. The `template` context is used in many places so it cannot be exempted from rate limits. Fixes: elastic#64595 Backport: 9370b1c
Remove the compilation rate limit for ingest templates. Creates a new context, `ingest_template`, with an unlimited compilation rate limit. The `template` context is used in many places so it cannot be exempted from rate limits. Fixes: #64595 Backport: 9370b1c
…lastic#69987) Remove the compilation rate limit for ingest templates. Creates a new context, `ingest_template`, with an unlimited compilation rate limit. The `template` context is used in many places so it cannot be exempted from rate limits. Fixes: elastic#64595 Backport: 9370b1c
Scripts in ingest pipelines may be throttled on startup or when beats starts up. This causes pipelines to be stuck in a broken state. Pipelines with many mustache templates are particularly impacted. Removes the compilation rate limit for ingest templates by creating a new context, `ingest_template`, with an unlimited compilation rate limit. Fixes: #64595 Backport: 9370b1c
* Script: no compile rate limit for ingest templates Remove the compilation rate limit for ingest templates. Creates a new context, `ingest_template`, with an unlimited compilation rate limit. The `template` context is used in many places so it cannot be exempted from rate limits. Fixes: elastic#64595
In #59267 the compilation rate limit for the
scriptandprocessor_conditionalcontexts was increased to unlimited. However some of the Ingest Node processors depend on mustache templates that are also scripts that need to be compiled. And those templates have their own separate rate limit. So users of Ingest Node pipelines can still hit limits. So I'd like to propose that templates also get an unlimited compilation rate in Ingest Node.Here's an example of the type of error users see when using Filebeat modules or packages with Fleet:
Bulk item insert failed (i=5, status=500): {\"type\":\"illegal_state_exception\",\"reason\":\"pipeline with id [logs-system.auth-0.9.0] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [logs-system.auth-0.9.0]]; nested: GeneralScriptException[Failed to compile inline script [{{system.auth.ssh.dropped_ip}}] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [{{system.auth.ssh.dropped_ip}}] using lang [mustache]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting]\"}",The text was updated successfully, but these errors were encountered: