inconsistent index options when opening pre 9.0.0 index with 9.0.0 [LUCENE-10314]

[asfimport]

We have a long-standing index with some mandatory fields and some optional
fields that has been through multiple lucene upgrades without a full
rebuild and on testing out an upgrade from version 8.11.0 to 9.0.0, when
open an IndexWriter we are hitting the exception

Exception in thread "main" java.lang.IllegalArgumentException: cannot
change field "language" from index options=NONE to inconsistent index
options=DOCS
        at
org.apache.lucene.index.FieldInfo.verifySameIndexOptions(FieldInfo.java:245)
        at
org.apache.lucene.index.FieldInfos$FieldNumbers.verifySameSchema(FieldInfos.java:421)
        at
org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:357)
        at
org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1263)
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1116)

Where language is one of our optional fields.

Presumably this is at least somewhat related to "Index options can no
longer be changed dynamically" as mentioned at
https://lucene.apache.org/core/9_0_0/MIGRATE.html although it fails before
our code attempts to update the index, and we are not trying to change any
index options.

Adding some displays to IndexWriter and FieldInfos and logging rather than
throwing the exception I see

 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=NONE
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS
 language     curr=NONE, other=DOCS

where there is one line per segment.  It logs the exception whenever
other=DOCS.  Subset with segment info:

segment _x8(8.2.0):c31753/-1:[diagnostics={timestamp=1565623850605,
lucene.version=8.2.0, java.vm.version=11.0.3+7, java.version=11.0.3,
mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
java.runtime.version=11.0.3+7,
os=Linux}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]

 language     curr=NONE, other=NONE

segment _y9(8.7.0):c43531/-1:[diagnostics={timestamp=1604597581562,
lucene.version=8.7.0, java.vm.version=11.0.3+7, java.version=11.0.3,
mergeMaxNumSegments=-1, os.version=3.1.0-1.2-desktop,
java.vendor=AdoptOpenJDK, source=merge, os.arch=amd64, mergeFactor=10,
java.runtime.version=11.0.3+7,
os=Linux}]:[attributes={Lucene87StoredFieldsFormat.mode=BEST_SPEED}]

 language     curr=NONE, other=DOCS

NOT throwing java.lang.IllegalArgumentException: cannot change field
"language" from index options=NONE to inconsistent index options=DOCS

---

Migrated from LUCENE-10314 by Ian Lea, 2 votes

[asfimport]

Michael Sokolov (@msokolov) (migrated from JIRA)

Small side note here; we have this comment in IndexOptions}:

public enum IndexOptions
  // NOTE: order is important here; FieldInfo uses this
  // order to merge two conflicting IndexOptions (always
  // "downgrades" by picking the lowest).

which is probably no longer relevant (since conflicting IndexOptions are forbidden).

[tflobbe]

This is resolved with #12326

[tflobbe]

@msokolov, updated the comment here