Script context compilation rate changes via HTTP API do not take effect

[Cakez0r]

Elasticsearch version (bin/elasticsearch --version):

"version" : {                                               
  "number" : "7.11.2",                                      
  "build_flavor" : "default",                               
  "build_type" : "deb",                                     
  "build_hash" : "3e5a16cfec50876d20ea77b075070932c6464c7d",
  "build_date" : "2021-03-06T05:54:38.141101Z",             
  "build_snapshot" : false,                                 
  "lucene_version" : "8.7.0",                               
  "minimum_wire_compatibility_version" : "6.8.0",           
  "minimum_index_compatibility_version" : "6.0.0-beta1"     
}

Description of the problem including expected versus actual behavior:
Attempting to set the script.context.template.max_compilations_rate setting via the PUT /_cluster/settings API (persistent and transient) does not seem to affect the setting. I scanned over the ScriptCache and ScriptService code and it looks like the setting gets cached and locked in at the time the ScriptCache is created and does not have any mechanism for receiving updates when the setting changes. I'm not familiar with the elasticsearch codebase, so my naive diagnosis could be wrong here.

Steps to reproduce:

1. Cause a lot of dynamic compilations such that a warning is generated CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.context.template.max_compilations_rate] setting]
2. Update the script.context.template.max_compilations_rate setting via PUT /_cluster/settings API
3. Observe that the error message from step 1 still appears and reports that the compilation rate limit is still 75/5m
4. Observe that GET /_cluster/settings reports that script.context.template.max_compilations_rate is not 75/5m

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[stu-elastic]

Are you getting this error in an ingest processor? Those processors cache the Circuit Breaking exception, which has the old rate. The work around is to delete the pipeline and re-add it.

the setting gets cached and locked in at the time the ScriptCache is created and does not have any mechanism for receiving updates when the setting changes

You are correct, nice digging in unfamiliar code.

In fact, the entire cache for a context is replaced on a settings update via a ClusterSettingsListener.

[Cakez0r]

Hi Stu. Sorry for the slow response. This was indeed caused by an ingest processor. Just wanted to check back in and confirm that your suggested workaround worked. Thanks!

[stu-elastic]

Thanks for confirming it worked for you. As we no longer throttle ingest script compilation, I'm going to close this issue.