EOF with boolean query in ES 9.0.x

[benwtrent]

Elasticsearch Version

9.0.x

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

Lucene 10.1.0 on ES 9.0.x exhibits EOF for certain queries.

In particular, it must be a boolean query with a multi-claused must_not disjunction.

Steps to Reproduce

N/A (yet)

Here is an example query:

        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "filter": [
                    "< multiple term clauses>"
                  ],
                  "must_not": [
                    {
                      "terms": {
                        "<field>": [
                          "<many_values>"
                        ]
                      }
                    },
                    {
                      "terms": {
                        "<field>": [
                          "<many_values>"
                        ]
                      }
                    },
                    {
                      "term": {
                        "<field>": {
                          "value": "<value>"
                        }
                      }
                    },
                    {
                      "terms": {
                        "<field>": [
                          "<many_values>"
                        ]
                      }
                    }
                  ]
                }
              }
            ],
            "must": {
              "query": {
                "multi_match": {
                  "fields": [
                    "<many_fields>"
                  ],
                  "minimum_should_match": "custom should match",
                  "query": "<QUERY>",
                  "type": "most_fields"
                }
              }
            }
          }
        }

Logs (if relevant)

Caused by: java.io.EOFException: read past EOF (pos=2147483647): MemorySegmentIndexInput(path="<PATH>.cfs") [slice=_ou.nvd] [slice=randomaccess]
	at org.apache.lucene.core@10.1.0/org.apache.lucene.store.MemorySegmentIndexInput.handlePositionalIOOBE(MemorySegmentIndexInput.java:119)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl.readByte(MemorySegmentIndexInput.java:762)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.codecs.lucene90.Lucene90NormsProducer$3.longValue(Lucene90NormsProducer.java:399)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.TermScorer.score(TermScorer.java:93)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.DisjunctionSumScorer.score(DisjunctionSumScorer.java:43)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.DisjunctionScorer.score(DisjunctionScorer.java:176)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.FilterScorable.score(FilterScorable.java:45)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.MaxScoreBulkScorer.scoreNonEssentialClauses(MaxScoreBulkScorer.java:402)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.MaxScoreBulkScorer.scoreInnerWindowWithFilter(MaxScoreBulkScorer.java:208)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.MaxScoreBulkScorer.scoreInnerWindow(MaxScoreBulkScorer.java:147)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.MaxScoreBulkScorer.score(MaxScoreBulkScorer.java:128)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:66)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:461)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:810)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:388)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
	at org.apache.lucene.core@10.1.0/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:372)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:339)
	at org.elasticsearch.server@9.0.3/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
	... 15 more

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[benwtrent]

Reading the code paths:

• ReqExclBulkScorer#score(LeafCollector, Bits, int, int) its possible that max == NO_MORE_DOCS if that leaf slice is the entire leaf. In this case, I expect that it is as it threw on a cfs file.
• It seems possible that within MaxScoreBulkScorer#score(LeafCollector, Bits, int, int) that we actually iterate and force outerWindowMax to be NO_MORE_DOCS, just like the passed in max
• Then we come to MaxScoreBulkScorer#scoreInnerWindowWithFilter(LeafCollector, Bits, int, DisiWrapper) with the max==NO_MORE_DOCS.

This is where I get stuck, I just don't see how doc == NO_MORE_DOCS. Here is the simplified loops (removing paths we don't hit in this stack trace).

private void scoreInnerWindowWithFilter(LeafCollector collector, Bits acceptDocs, int max, DisiWrapper filter) throws IOException {
    DisiWrapper top = essentialQueue.top();
    if (top.doc < filter.doc) {
      top.doc = top.approximation.advance(filter.doc);
    }

    int innerWindowMin = top.doc;
    // if `max` == NO_MORE_DOCS, the worst this can be is `NO_MORE_DOCS`
    int innerWindowMax = (int) Math.min(max, (long) innerWindowMin + INNER_WINDOW_SIZE);

    // SEEMS TO PROTECT AGAINST NO_MORE_DOCS???
    while (top.doc < innerWindowMax) {
      if (filter.doc < top.doc) {
        filter.doc = filter.approximation.advance(top.doc);
      }

      if (filter.doc != top.doc) {
        do {
          top.doc = top.iterator.advance(filter.doc);
          top = essentialQueue.updateTop();
        } while (top.doc < filter.doc);
      } else {
        int doc = top.doc;
        boolean match =
            (acceptDocs == null || acceptDocs.get(doc))
                && (filter.twoPhaseView == null || filter.twoPhaseView.matches());
        double score = 0;
        do {
          if (match) {
            score += top.scorer.score();
          }
          top.doc = top.iterator.nextDoc();
          top = essentialQueue.updateTop();
        } while (top.doc == doc);

        if (match) {
          scoreNonEssentialClauses(collector, doc, score, firstEssentialScorer);
        }
      }
    }
  }

If we get to scoreNonEssentialClauses where doc==NO_MORE_DOCS then it explains the EOF.

As this is the advance and score code TermScorer which calls Lucene90NormsProducer#advancedExact

  @Override
  public float score() throws IOException {
    var postingsEnum = this.postingsEnum;
    var norms = this.norms;

    long norm = 1L;
    if (norms != null && norms.advanceExact(postingsEnum.docID())) {
      norm = norms.longValue();
    }
    return scorer.score(postingsEnum.freq(), norm);
  }

    @Override
    public boolean advanceExact(int target) throws IOException {
      this.doc = target;
      return true;
    }

Which just sets the doc to the target and thus when you call score() it will throw an EOF.

[benwtrent]

Related Lucene issue for investigation there: apache/lucene#15324