null_pointer_exception for artificial document term vector request

[rotzbouw]

I get null_pointer_exception for certain term vector requests. I narrowed it down to a reproducible request, which as follows:
{ "doc": { "entities.en": [ "Fantasy term" ] }, "fields": [ "entities.*" ], "term_statistics": true, "offsets": true, "filter": { "min_term_freq": 1 } }
as a GET request to my index using the _termvectors endpoint.
I get the following response:
{ "error": { "root_cause": [ { "type": "exception", "reason": "failed to execute term vector request" } ], "type": "exception", "reason": "failed to execute term vector request", "caused_by": { "type": "null_pointer_exception", "reason": null } }, "status": 500 }
I'm using Elasticsearch version 7.3.2 as a Docker container using two nodes (same happened for one node) and replicas set to 1.

Elasticsearch version (bin/elasticsearch --version):
7.3.2

Plugins installed: [hunspell]

JVM version (java -version):
openjdk version "12.0.2" 2019-07-16

OS version (uname -a if on a Unix-like system):
Linux elasticsearch-master-0 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 x86_64 x86_64 GNU/Linux

Provide logs (if relevant):
{"type": "server", "timestamp": "2019-10-08T09:56:54,230+0000", "level": "DEBUG", "component": "o.e.a.t.TransportTermVectorsAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "cJYEZvN_Slyj4-aLGqMkvA", "node.id": "83NfGvuxTvKezGfC2ZHilQ", "message": "null: failed to execute [org.elasticsearch.action.termvectors.TermVectorsRequest@454e4b9f]" , "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][10.42.9.29:9300][indices:data/read/tv[s]]", "Caused by: org.elasticsearch.ElasticsearchException: failed to execute term vector request", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:149) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:78) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:110) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:46) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:112) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:835) [?:?]", "Caused by: java.lang.NullPointerException", "at java.util.Objects.requireNonNull(Objects.java:221) ~[?:?]", "at org.apache.lucene.search.TermStatistics.<init>(TermStatistics.java:65) ~[lucene-core-8.1.0.jar:8.1.0 dbe5ed0b2f17677ca6c904ebae919363f2d36a0a - ishan - 2019-05-09 19:34:03]", "at org.elasticsearch.action.termvectors.TermVectorsFilter.getTermStatistics(TermVectorsFilter.java:289) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TermVectorsFilter.selectBestTerms(TermVectorsFilter.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:138) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:78) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:110) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:46) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:112) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.2.jar:7.3.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]", "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] } {"type": "server", "timestamp": "2019-10-08T09:56:54,230+0000", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "cluster.uuid": "cJYEZvN_Slyj4-aLGqMkvA", "node.id": "83NfGvuxTvKezGfC2ZHilQ", "message": "path: /contents-v2-dev-99349/_termvectors, params: {index=contents-v2-dev-99349}" , "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch-master-0][10.42.9.29:9300][indices:data/read/tv[s]]", "Caused by: org.elasticsearch.ElasticsearchException: failed to execute term vector request", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:149) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:78) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:110) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:46) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:112) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", "at java.lang.Thread.run(Thread.java:835) [?:?]", "Caused by: java.lang.NullPointerException", "at java.util.Objects.requireNonNull(Objects.java:221) ~[?:?]", "at org.apache.lucene.search.TermStatistics.<init>(TermStatistics.java:65) ~[lucene-core-8.1.0.jar:8.1.0 dbe5ed0b2f17677ca6c904ebae919363f2d36a0a - ishan - 2019-05-09 19:34:03]", "at org.elasticsearch.action.termvectors.TermVectorsFilter.getTermStatistics(TermVectorsFilter.java:289) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TermVectorsFilter.selectBestTerms(TermVectorsFilter.java:221) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:138) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.index.termvectors.TermVectorsService.getTermVectors(TermVectorsService.java:78) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:110) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.termvectors.TransportTermVectorsAction.shardOperation(TransportTermVectorsAction.java:46) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$1.doRun(TransportSingleShardAction.java:112) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.3.2.jar:7.3.2]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.3.2.jar:7.3.2]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]", "at java.lang.Thread.run(Thread.java:835) ~[?:?]"] }

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[cbuescher]

Could you share some more information about the index you are calling this on (e.g. the mappings and settings)? When I execute your request from above like this on 7.3.2 (one node, default 1 shard) I don't get the NPE you see:

PUT test

GET /test/_termvectors
{
  "doc": {
    "entities.en": [
      "Fantasy term"
    ]
  },
  "fields": [
    "entities.*"
  ],
  "term_statistics": true,
  "offsets": true,
  "filter": {
    "min_term_freq": 1
  }
}

but an empty term vector response and no errors.

{
  "_index" : "test",
  "_type" : "_doc",
  "_version" : 0,
  "found" : true,
  "took" : 0,
  "term_vectors" : { }
}

[rotzbouw]

Hi, thanks for your repliy.
Steps to reproduce in more detail:

1. Create Index.

PUT myindex
{
  "settings" : {
    "number_of_shards" : 1
  },
  "mappings" : {
    "properties" : {
      "id" : { "type" : "keyword" }
    },
    "dynamic_templates": [
      {
        "namedEntities": {
          "path_match": "entities.*",
          "mapping": {
            "type": "text",
            "term_vector": "with_offsets"
          }
        }
      }
    ]
  }
}


2. Add a document

PUT myindex/_doc/1
{
  "id": "1"
}

3. Update document

POST myindex/_update/1
{
  "script": {
    "source": "ctx._source.entities=params.entities;",
    "lang": "painless",
    "params": {
      "entities": {
        "en": [
          "dog"
        ]
      }
    }
  }
}

4. Get termvector for an artifical document

GET myindex/_termvectors
{
  "term_statistics": true,
  "doc": {
    "entities.en": [
      "cat"
    ]
  },
  "filter": {
    "max_num_terms": 1000,
    "min_term_freq": 1,
    "max_term_freq": 1000,
    "min_doc_freq": 1,
    "max_doc_freq": 1000,
    "min_word_length": 1,
    "max_word_length": 1000
  }
}