LUCENE-10236: Updated field-weight used in CombinedFieldQuery scoring calculation, and added a test

[zacharymorn]

Description

Updated field-weight used in CombinedFieldQuery scoring calculation

Tests

1. Added a new test from LUCENE-10061: Implements dynamic pruning support for CombinedFieldsQuery #418
2. Run ./gradlew clean; ./gradlew check -Pvalidation.git.failOnModified=false

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.

[jimczi]

LGTM, good catch @zacharymorn

[jimczi]

Could be assert duplicateCheckingSet.add(field.field) == false ?

[zacharymorn]

Ah yes. I assume you meant assert duplicateCheckingSet.add(field.field) and have updated it accordingly.

[jtibshirani]

Thanks for catching this!

[jtibshirani]

Could you explain how this test relates to the fix? Would it make sense to have a more targeted test similar to testCopyField?

[zacharymorn]

This test was actually developed in another related PR #418, and it would generate duplicated field-weight pairs in fields object to trigger the condition. As the existing tests don't currently trigger that condition, I thought this would be a good test to put in first for both PRs.

I guess if we were to have a focused test for this PR, it would still look something very similar to this test (and testCopyField), maybe something like the following, and it just ensures the test run through without assertion exception. The delta between the tests are just I removed some doc content randomizations and result comparison between TOP_SCORE and COMPLETE collection.

 public void testShouldRunWithoutAssertionException() throws IOException {
    int numDocs =
        randomBoolean()
            ? atLeast(1000)
            : atLeast(128 * 8 * 8 * 3); // make sure some terms have skip data
    int numMatchDoc = randomIntBetween(200, 500);
    int numHits = atMost(100);
    int boost1 = Math.max(1, random().nextInt(5));
    int boost2 = Math.max(1, random().nextInt(5));

    Directory dir = newDirectory();
    Similarity similarity = randomCompatibleSimilarity();

    IndexWriterConfig iwc = new IndexWriterConfig();
    iwc.setSimilarity(similarity);
    RandomIndexWriter w = new RandomIndexWriter(random(), dir, iwc);

    // adding potentially matching doc
    for (int i = 0; i < numMatchDoc; i++) {
      Document doc = new Document();

      int freqA = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqA; j++) {
          doc.add(new TextField("a", "foo", Store.NO));
        }
      }

      freqA = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqA; j++) {
          doc.add(new TextField("a", "foo" + j, Store.NO));
        }
      }

      freqA = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqA; j++) {
          doc.add(new TextField("a", "zoo", Store.NO));
        }
      }

      int freqB = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqB; j++) {
          doc.add(new TextField("b", "zoo", Store.NO));
        }
      }

      freqB = random().nextInt(20) + 1;
      if (randomBoolean()) {
        for (int j = 0; j < freqB; j++) {
          doc.add(new TextField("b", "zoo" + j, Store.NO));
        }
      }

      int freqC = random().nextInt(20) + 1;
      for (int j = 0; j < freqC; j++) {
        doc.add(new TextField("c", "bla" + j, Store.NO));
      }
      w.addDocument(doc);
    }

    IndexReader reader = w.getReader();
    IndexSearcher searcher = newSearcher(reader);
    searcher.setSimilarity(similarity);

    CombinedFieldQuery query =
        new CombinedFieldQuery.Builder()
            .addField("a", (float) boost1)
            .addField("b", (float) boost2)
            .addTerm(new BytesRef("foo"))
            .addTerm(new BytesRef("zoo"))
            .build();

    TopScoreDocCollector completeCollector =
        TopScoreDocCollector.create(numHits, null, Integer.MAX_VALUE); 
    searcher.search(query, completeCollector);

    reader.close();
    w.close();
    dir.close();
  }

I guess I'm fine either way? If you prefer a more focused test, I can replace it with the above one.

[jtibshirani]

I think starting with this more focused test makes sense for this PR.

[zacharymorn]

Sounds good. I've replaced the test with a more focused one.

[zacharymorn]

Thanks @jimczi @jtibshirani for the review and feedback!

[jtibshirani]

Looks good to me too!

[zacharymorn]

Will backport this to 8.11 and 8x.

[zacharymorn]

Will backport this to 8.11 and 8x.

Hi @jimczi @jtibshirani @mikemccand, just to confirm as I saw there was a thread on the proper handling of branch_8x, this change should be backported to branches branch_8_11 (version 8.11.2), branch_9_0 (version 9.0.1) & branch_9x (version 9.1.0), but not to branch branch_8x right?

[jpountz]

Correct, changes should no longer be backported to branch_8x.