LUCENE-8370: Reproducing TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failures

(cherry picked from commit c303c5f)

Author: Erick Erickson

lucene/CHANGES.txt

@@ -15,6 +15,10 @@ Bug Fixes:
    via LUCENE-8165 since we are now checking for offset+length going out of bounds.
    (Robert Muir, Nhat Nyugen, Simon Willnauer)
 
+* LUCENE-8370: Reproducing 
+  TestLucene{54,70}DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()
+  failures (Erick Erickson)
+
 Changes in Runtime Behavior:
 
 * LUCENE-7976: TieredMergePolicy now respects maxSegmentSizeMB by default when executing

lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java

@@ -60,15 +60,22 @@
  *  LogMergePolicy}.
  *
  *  <p><b>NOTE</b>: This policy always merges by byte size
- *  of the segments, always pro-rates by percent deletes,
- *  and does not apply any maximum segment size during
- *  forceMerge (unlike {@link LogByteSizeMergePolicy}).
+ *  of the segments, always pro-rates by percent deletes
  *
- *  <p><b>NOTE</b> As of Lucene 7.5, forceMerge (aka optimize)
- *  and expungeDeletes (findForcedMerges and
- *  findForcedDeletesMerges) respect the max segment
+ *  <p><b>NOTE</b> Starting with Lucene 7.5, there are several changes:
+ *
+ *  - findForcedMerges and findForcedDeletesMerges) respect the max segment
  *  size by default.
  *
+ *  - When findforcedmerges is called with maxSegmentCount other than 1,
+ *  the resulting index is not guaranteed to have &lt;= maxSegmentCount segments.
+ *  Rather it is on a "best effort" basis. Specifically the theoretical ideal
+ *  segment size is calculated and a "fudge factor" of 25% is added as the
+ *  new maxSegmentSize, which is respected.
+ *
+ *  - findForcedDeletesMerges will not produce segments greater than
+ *  maxSegmentSize.
+ *
  *  @lucene.experimental
  */
 

lucene/test-framework/src/java/org/apache/lucene/index/RandomIndexWriter.java

@@ -369,7 +369,9 @@ private void doRandomForceMerge() throws IOException {
           System.out.println("RIW: doRandomForceMerge(" + limit + ")");
         }
         w.forceMerge(limit);
-        assert !doRandomForceMergeAssert || w.getSegmentCount() <= limit: "limit=" + limit + " actual=" + w.getSegmentCount();
+        if (limit == 1 || (w.getConfig().getMergePolicy() instanceof TieredMergePolicy) == false) {
+          assert !doRandomForceMergeAssert || w.getSegmentCount() <= limit : "limit=" + limit + " actual=" + w.getSegmentCount();
+        }
       } else {
         if (LuceneTestCase.VERBOSE) {
           System.out.println("RIW: do random forceMergeDeletes()");

solr/solr-ref-guide/src/solr-upgrade-notes.adoc

@@ -30,7 +30,7 @@ Detailed steps for upgrading a Solr cluster are in the section <<upgrading-a-sol
 === Solr 7.5
 When upgrading to Solr 7.4, users should be aware of the following major changes from v7.3:
 
-* When using the default TieredMergePolicy (TMP), optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5,000 (5 gigaBytes). If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments=# where # is a positive integer. TMP will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively.
+* When using the default TieredMergePolicy (TMP), optimize and expungeDeletes now respect the maxMergedSegmentMB configuration parameter, which defaults to 5,000 (5 gigaBytes). If it is absolutely necessary to control the number of segments present after optimize, specify maxSegments=# where # is a positive integer. maxSegments > 1 are honored on a "best effort" basis. TMP will also reclaim resources from segments that exceed maxMergedSegmentMB more aggressively.
 
 
 === Solr 7.4

solr/solr-ref-guide/src/uploading-data-with-index-handlers.adoc

@@ -100,7 +100,7 @@ Default is `true`. Blocks until a new searcher is opened and registered as the m
 
 WARNING: expungeDeletes is "less expensive" than optimize, but the same warnings apply.
 
-`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the maxMergedSegmentMB setting. Merges the segments down to no more than this number of segments. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.
+`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the maxMergedSegmentMB setting. Makes a "best effort" attempt to merge the segments down to no more than this number of segments but does not guarantee that the goal will be achieved. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.
 
 Here are examples of <commit> and <optimize> using optional attributes: