Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest: only compile templates if the value is templated #37120

Closed
jakelandis opened this issue Jan 3, 2019 · 1 comment

Comments

Projects
None yet
2 participants
@jakelandis
Copy link
Contributor

commented Jan 3, 2019

Many of the ingest node processors support template snippets (mustache). Each field that supports template snippets requires compilation the first time the pipeline is used.

Elasticsearch has circuit breaker such that no more than 75 scripts per 5 minutes may be compiled. Every script (not only ingest node scripts) is subject to this (configurable) limit. A moderate number of processors can trip this limit, even if no templated values are defined.

For example, a single set processor always counts as 2 compilations since the field and value both support template snippets. The compilation happens even if the field and value do not define a {{ }} template.

This issue is to address the unnecessary compilation for fields that support template snippets. The goal to reduce the number of compilations to only the minimum number actually required.

This issue is more likely on newer versions of Elasticsearch since in 6.4 additional fields were added to support template snippets and 6.5 the if condition was added, which also requires a script complication (if defined).

@elasticmachine

This comment has been minimized.

Copy link

commented Jan 3, 2019

jakelandis added a commit to jakelandis/elasticsearch that referenced this issue Jan 8, 2019

ingest: compile mustache template only if field includes '{{''
Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.

This change simple checks for '{{' in the text before it attempts to compile.

fixes elastic#37120

jakelandis added a commit that referenced this issue Jan 9, 2019

ingest: compile mustache template only if field includes '{{'' (#37207)
* ingest: compile mustache template only if field includes '{{''

Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.

This change simple checks for '{{' in the text before it attempts to compile.

fixes #37120

jakelandis added a commit that referenced this issue Jan 10, 2019

ingest: compile mustache template only if field includes '{{'' (#37207)
* ingest: compile mustache template only if field includes '{{''

Prior to this change, any field in an ingest node processor that supports
script templates would be compiled as mustache template regardless if they
contain a template or not. Compiling normal text as mustache templates is
harmless. However, each compilation counts against the script compilation
circuit breaker. A large number of processors without any templates or scripts
could un-intuitively trip the too many script compilations circuit breaker.

This change simple checks for '{{' in the text before it attempts to compile.

fixes #37120
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.