Make script cache and compilation limits per context

[jdconrad]

Currently, we have setting to apply a max for the script cache and settings to determine the max rate of compilation. These apply to all script contexts. We should change the settings to be per script context as each context has different needs.

Relates to #49763

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

[stu-elastic]

In ScriptService
SCRIPT_CACHE_SIZE_SETTING and SCRIPT_MAX_COMPILATIONS_RATE, SCRIPT_CACHE_EXPIRE_SETTING needs to be per context as well as Cache<CacheKey, Object> cache; along with scriptMetrics, which is where they are being tracked.

SCRIPT_MAX_COMPILATIONS_RATE needs to be able to be unlimited.

In compile we can sychronize on the context rather than this. Can add arbitrary private object used for locking.

Idea: encapsulate these into a ContextCompiler, move all compile logic into here.

[stu-elastic]

Code staging:

1. General context cache
2. Add settings for context caches. Make them dynamically updatable. Default to general compiler. Ability to transition to context caches without restart.
3. Add defaults for each context
4. Add unlimited compile rate settings
5. Default to per context settings and deprecate old general settings
6. Update stats API

Behavior:
If general settings, use general compiler and deprecate log
If no general settings use context specific
If general settings and context specific, error
-> cluster update settings for context specific and general then context specific rejected

[stu-elastic]

Should we disable caching all together if a user sets an unlimited compilation rate?

[stu-elastic]

Having script and script_cache in _node/stats is redundant. Here's the value of _node/stats in 7.x:

      "script" : {
        "compilations" : 172,
        "cache_evictions" : 72,
        "compilation_limit_triggered" : 0
      },

Here's it in 8.x right now:

      "script_cache": {                                                      
        "sum": {                      
          "compilations": 1,                                                 
          "cache_evictions": 0,                                              
          "compilation_limit_triggered": 0                                   
        },                            
        "contexts": [                                                        
          {                           
            "context": "aggregation_selector",                               
            "compilations": 0,                                               
            "cache_evictions": 0,                                            
            "compilation_limit_triggered": 0                                 
          },                                  

We will remove script_cache in 8 and hoist sum to the top level. Here's what we'll have:

      "script": {                          
        "compilations": 1,
        "cache_evictions": 0,
        "compilation_limit_triggered": 0,
        "contexts":[
          {
            "context": "aggregation_selector",                               
            "compilations": 0,                                               
            "cache_evictions": 0,                                            
            "compilation_limit_triggered": 0                                 
          },                                  

This keeps the format the same between 7 and 8 so we can easily drop script_cache. Also the sum is unnecessary indirection.