Stored scripts are not supported for runtime field

[camelia-c]

Hello,

I am interested in the following functionality which is not yet supported in Elasticsearch 7.12.1

Here is a minimal example:

curl -XPUT "$HOST_ELASTIC:9200/demoidx"

cat << EOF > /tmp/doc_1.json
{
    "f1" : "some text",
    "f2" : 130
}
EOF


curl -H "Content-Type: application/json"  \
     -XPOST "$HOST_ELASTIC:9200/demoidx/demo_doctype/" \
     -d @/tmp/doc_1.json   \
     |  python -m json.tool

and a stored script:

cat << EOF > /tmp/scr_1.json
{
    "script": {
        "source": " def f2 = doc['f2'].value; \
            return (f2 * params.factor); ",
        "lang": "painless"
    }
}
EOF


curl -H "Content-Type: application/json"  \
     -XPOST "$HOST_ELASTIC:9200/_scripts/demo_script" \
     -d "@/tmp/scr_1.json" \
     |  python -m json.tool

For now, we can successfully use the stored script to return a scripted field in the result:

cat << EOF > /tmp/q_ok.json
{
    "script_fields": {
        "f2_computed": {
            "script": {
                "id": "demo_script",
                "params": { "factor" : 10 }
            }
        }
    },
    "fields":[
        "f2_computed",
        "f2"
    ],
    "_source": false
}
EOF


curl -H "Content-Type: application/json"  \
     -XPOST "$HOST_ELASTIC:9200/demoidx/_search?pretty" \
     -d @/tmp/q_ok.json

Yet this cannot be used in filtering, because it's computed in the fetch stage of the execution plan.
The newly released runtime mappings accept inline source code, but it would be great to have them accept stored scripts too.
For now, if we try:

cat << EOF > /tmp/q_notok.json
{
    "runtime_mappings": {
        "f2_computed": {
            "type" : "long",
            "script": {
                "id": "demo_script",
                "params": { "factor" : 10 }
            }
        }
    },
    "query":{
        "bool":{
            "filter" : {
                "range": {
                    "f2_computed" : { "gt" : 1000 }
                }
            }                        
        }
    },
    "fields":[
        "f2_computed",
        "f2"
    ],
    "_source": false
}
EOF


curl -H "Content-Type: application/json"  \
     -XPOST "$HOST_ELASTIC:9200/demoidx/_search?pretty" \
     -d @/tmp/q_notok.json 

then we get error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "stored scripts are not supported for runtime field [f2_computed]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "demoidx",
        "node" : "......",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "stored scripts are not supported for runtime field [f2_computed]"
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "stored scripts are not supported for runtime field [f2_computed]",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "stored scripts are not supported for runtime field [f2_computed]"
      }
    }
  },
  "status" : 400
}

Till the support for stored scripts will be added, the query needs to be written as:

cat << EOF > /tmp/q_inline.json
{
    "runtime_mappings": {
        "f2_computed": {
            "type" : "long",
            "script": {
                "source": " def f2 = doc['f2'].value; \
                emit(f2 * params.factor) ",
            "params" : {"factor" : 10}
            }
        }
    },
    "query":{
        "bool":{
            "filter" : {
                "range": {
                    "f2_computed" : { "gt" : 1000 }
                }
            }                        
        }
    },
    "fields":[
        "f2_computed",
        "f2"
    ],
    "_source": false
}
EOF


curl -H "Content-Type: application/json"  \
     -XPOST "$HOST_ELASTIC:9200/demoidx/_search?pretty" \
     -d @/tmp/q_inline.json 

Thanks in advance,
Camelia

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[javanna]

Thanks for the clear recreation @camelia-c . I would be interested to know more about the usecase where stored scripts can be useful in the context of runtime fields.

Runtime fields are fields whose values can be computed at runtime and used in any section of a search request. Runtime fields are usually added to index mappings, but they can also be ephemeral, meaning they can be defined directly in the search request in which case the runtime field will be available only in the context of that specific search request.

When we developed runtime fields, we decided not to support stored scripts in their index mappings definition, because they are already stored within the index mappings and we did not see value in allowing to store the definition of a runtime field that points to a stored script. It is one more layer of indirection that can be confusing and add complexity while adding little value if none. I can see how it may come in handy when defining a runtime field in the search request, on the other hand given the ephemeral nature of such fields, I wonder if their scripts should/could be stored in the cluster state. Also, the flow will many times be to go from a search defined runtime field to making it permanent in the index mappings, hence adding support for stored scripts to runtime fields defined in the search request translates to supporting them also in runtime fields defined in the index mappings, which I am not sure is a good idea for the reason I stated above.

[javanna]

Closing for lack of feedback, please feel free to re-open if you are interested in this issue and have feedback around it.

[consulthys]

@javanna here is an example where it would be beneficial to allow runtime fields to access/reuse stored scripts
https://stackoverflow.com/questions/70652885/how-to-use-carriage-return-in-a-script-template-with-a-runtime-mapping-field

[whitfin]

@javanna hi!

Everything you said makes sense, but a couple of points I think are relevant:

1. Is there no benefit to having the script readily available in the cluster rather than compiled on every search request? Rather than passing them on every request, an app could register them at startup, etc.
2. If the flow is search defined runtime field -> mapping runtime field (which I agree will be most common), this currently requires copy/paste between the two, which requires someone to review/validate the full script rather than just the script name.
3. Any mistakes made during changes to a stored script are much lower impact than someone making a mistake when changing index mappings; from this perspective supporting stored scripts is an ease of maintainability. For example, automatically updating scripts at application startup sounds far less scary than automatically updating mappings.

Not necessarily saying it's the most urgent thing ever, but if it's trivial to support I would definitely see more benefit than harm. At the very least I think the docs need to clearly state that stored scripts don't work; unless I'm crazy this isn't mentioned anywhere.

[javanna]

Thanks for the feedback @whitfin , I also missed a previous reply, sorry @consulthys ! I will reopen this for discussion.

[whitfin]

Thanks @javanna!

Just one more note on this but if it is decided to keep this unsupported, the docs at https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-using.html should be updated, specifically:

Wherever scripting is supported in the Elasticsearch APIs, the syntax follows the same pattern; you specify the language of your script, provide the script logic (or source), and add parameters that are passed into the script:

"script": {
 "lang":   "...",
 "source" | "id": "...",
 "params": { ... }
}

This in particular is what led me astray in expecting this to work on runtime_mappings.