Using `Debug.explain` kills data nodes. #103018

ong-ar · 2023-12-05T19:58:18Z

Elasticsearch Version

Version: 8.11.1, Build: rpm/6f9ff581fbcde658e6f69d6ce03050f060d1fd0c/2023-11-11T10:05:59.421038163Z, JVM: 21.0.1

Installed Plugins

[]

Java Version

openjdk version "21.0.1"

OS Version

6.1.61-85.141.amzn2023.x86_64

Problem Description

I often use map_script for statistics, but when I use Debug.explain, the data nodes emit the message below and all die.

If removing Debug.explain, it works normally.

It wasn't a problem when using version 8.6.2, but since upgrading, I'm seeing that.

I need help.

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [data-03] fatal error in thread [elasticsearch[data-03][search_worker][T#14]], exiting
org.elasticsearch.painless.PainlessExplainError: null
        at org.elasticsearch.painless.api.Debug.explain(Debug.java:23) ~[?:?]
        at org.elasticsearch.painless.PainlessScript$Script.execute(  ...:66) ~[?:?]
        at org.elasticsearch.search.aggregations.metrics.ScriptedMetricAggregator$1.collect(ScriptedMetricAggregator.java:124) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:86) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.query.QueryPhaseCollector$CompositeLeafCollector.collect(QueryPhaseCollector.java:306) ~[elasticsearch-8.11.1.jar:?]
        at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:277) ~[lucene-core-9.8.0.jar:?]
        at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:236) ~[lucene-core-9.8.0.jar:?]
        at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) ~[elasticsearch-8.11.1.jar:?]
        at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38) ~[lucene-core-9.8.0.jar:?]
        at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:536) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:460) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:375) ~[elasticsearch-8.11.1.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

Steps to Reproduce

create index

PUT /test_index?pretty
{
    "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0
    },
    "mappings" : {
        "properties" : {
            "tags" : { "type" : "keyword" },
            "updated_at" : { "type" : "date" }
        }
    }
}

index a document

PUT test_index/_doc/1
{
  "tags": [
    "opster",
    "elasticsearch"
  ],
  "date": "01-01-2020"
}

search

GET test_index/_search
{
   "aggs": {
    "test":{
      "scripted_metric":{
        "init_script": """
          state.funding = new HashMap();
        """,
        "map_script": """ 
          def source = params._source;
          Debug.explain(source);
          return;
          
        """,
        "combine_script":"""
          return state;
          
        """,
        "reduce_script": """ 
          def result = new HashMap();
          
          return result;
         """
      }  
    }
  },
  "size":0
}

check

GET test_index/_search

If you use Debug.explain in map_script, an error message will appear, but if you search another index afterwards, the search will not work. The reason is that the data nodes are dead with the below message.

Logs (if relevant)

map_script result

{
  "error": {
    "root_cause": [
      {
        "type": "script_exception",
        "reason": "runtime error",
        "painless_class": "java.util.LinkedHashMap",
        "to_string": "{tags=[opster, elasticsearch], date=01-01-2020}",
        "java_class": "java.util.LinkedHashMap",
        "script_stack": [
          """Debug.explain(source);
          """,
          "              ^---- HERE"
        ],
        "script": "  ...",
        "lang": "painless",
        "position": {
          "offset": 65,
          "start": 51,
          "end": 84
        }
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "test_index",
        "node": "o39edlW3TC6VvO9Iu0bH8w",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "painless_class": "java.util.LinkedHashMap",
          "to_string": "{tags=[opster, elasticsearch], date=01-01-2020}",
          "java_class": "java.util.LinkedHashMap",
          "script_stack": [
            """Debug.explain(source);
          """,
            "              ^---- HERE"
          ],
          "script": "  ...",
          "lang": "painless",
          "position": {
            "offset": 65,
            "start": 51,
            "end": 84
          },
          "caused_by": {
            "type": "painless_explain_error",
            "reason": null
          }
        }
      }
    ]
  },
  "status": 400
}

elasticsearch log

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master-1] fatal error in thread [elasticsearch[master-1][search_worker][T#5]], exiting
org.elasticsearch.painless.PainlessExplainError: null
        at org.elasticsearch.painless.api.Debug.explain(Debug.java:23) ~[?:?]
        at org.elasticsearch.painless.PainlessScript$Script.execute(  ...:66) ~[?:?]
        at org.elasticsearch.search.aggregations.metrics.ScriptedMetricAggregator$1.collect(ScriptedMetricAggregator.java:124) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:86) ~[elasticsearch-8.11.1.jar:?]
        at org.apache.lucene.search.MatchAllDocsQuery$1$1.score(MatchAllDocsQuery.java:63) ~[lucene-core-9.8.0.jar:?]
        at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) ~[elasticsearch-8.11.1.jar:?]
        at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:38) ~[lucene-core-9.8.0.jar:?]
        at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:536) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:460) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$4(ContextIndexSearcher.java:375) ~[elasticsearch-8.11.1.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.1.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

system log

systemd-entrypoint: ERROR: Elasticsearch exited unexpectedly, with exit code 1

The text was updated successfully, but these errors were encountered:

elasticsearchmachine · 2023-12-06T10:24:51Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

In elastic#100872 Painless errors were wrapped so as to avoid throwing Errors outside scripting. However, one case was missed: PainlessExplainError which is used by Debug.explain. This commit adds the explain error to those that painless wraps. closes elastic#103018

ong-ar · 2023-12-08T03:21:21Z

@brianseeders

Can I know the expected release date for v8.11.3?

In #100872 Painless errors were wrapped so as to avoid throwing Errors outside scripting. However, one case was missed: PainlessExplainError which is used by Debug.explain. This commit adds the explain error to those that painless wraps. closes #103018

In elastic#100872 Painless errors were wrapped so as to avoid throwing Errors outside scripting. However, one case was missed: PainlessExplainError which is used by Debug.explain. This commit adds the explain error to those that painless wraps. closes elastic#103018

In #100872 Painless errors were wrapped so as to avoid throwing Errors outside scripting. However, one case was missed: PainlessExplainError which is used by Debug.explain. This commit adds the explain error to those that painless wraps. closes #103018

ong-ar · 2023-12-13T06:41:20Z

@stu-elastic

I don't think v8.11.3 contains that commit. May I know in what version it will be added?

rjernst · 2023-12-14T01:58:39Z

@ong-ar You are correct, this did not make the cutoff for 8.11.3. It will be in 8.12.0.

ong-ar added >bug needs:triage Requires assignment of a team area label labels Dec 5, 2023

craigtaverner added :Core/Infra/Scripting Scripting abstractions, Painless, and Mustache Team:Core/Infra Meta label for core/infra team v8.11.2 and removed needs:triage Requires assignment of a team area label labels Dec 6, 2023

stu-elastic self-assigned this Dec 6, 2023

brianseeders added v8.11.3 and removed v8.11.2 labels Dec 6, 2023

rjernst mentioned this issue Dec 7, 2023

Wrap painless explain error #103151

Merged

elasticsearchmachine closed this as completed in #103151 Dec 9, 2023

rjernst mentioned this issue Dec 9, 2023

Wrap painless explain error (#103151) #103234

Merged

rjernst mentioned this issue Dec 9, 2023

Wrap painless explain error (#103151) #103235

Merged

rjernst mentioned this issue Dec 9, 2023

Wrap painless explain error (#103151) #103236

Merged

DaveCTurner mentioned this issue Jan 2, 2024

Debug.explain kills Elasticsearch #103777

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using `Debug.explain` kills data nodes. #103018

Using `Debug.explain` kills data nodes. #103018

ong-ar commented Dec 5, 2023 •

edited

Loading

elasticsearchmachine commented Dec 6, 2023

ong-ar commented Dec 8, 2023

ong-ar commented Dec 13, 2023

rjernst commented Dec 14, 2023

Using Debug.explain kills data nodes. #103018

Using Debug.explain kills data nodes. #103018

Comments

ong-ar commented Dec 5, 2023 • edited Loading

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

elasticsearchmachine commented Dec 6, 2023

ong-ar commented Dec 8, 2023

ong-ar commented Dec 13, 2023

rjernst commented Dec 14, 2023

Using `Debug.explain` kills data nodes. #103018

Using `Debug.explain` kills data nodes. #103018

ong-ar commented Dec 5, 2023 •

edited

Loading