Update Lucene version to 9.11

[benwtrent]

No description provided.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[ChrisHegarty]

might wanna flip this on the Lucene RC build (rather than a snapshot)

[benwtrent]

OK, I tried this, and added the RC location as mvn location, but now ES is looking in that location for ALL packages, and its failing the build. Any ideas on how to simply add a secondary mvn location to look for packages instead of overriding?

[iverase]

I am investigating a possible issue in lucene 9.11, please do not merge.

[benwtrent]

While still investigating here: apache/lucene#13475

I have disabled intra-merge concurrency by always returning a same thread executor.

[benwtrent]

@martijnvg all the failures are around the new dynamic setting being changed.

Likely root cause: java.lang.IllegalArgumentException: Setting index.mapper.dynamic was removed after version 6.0.0 |  
-- | --
  | at org.elasticsearch.index.mapper.MapperService.<init>(MapperService.java:208) |  
  | at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:177) |  
  | at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.verifyIndexMetadata(IndexMetadataVerifier.java:87) |  
  | at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:279) |  
  | at org.elasticsearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:265) |  
  | at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:178) |  
  | at org.elasticsearch.node.Node.start(Node.java:1187) |  
  | at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:335) |  
  | at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:443) |  
  | at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) |  
  | at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) |  
  | at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) |  
  | at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) |  
  | at org.elasticsearch.cli.Command.main(Command.java:77) |  
  | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) |  
  | at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80)

Now, I am not sure why these particular BWC tests are running against much older versions than 7.17.22, but they are.

Here is an example failure: https://gradle-enterprise.elastic.co/s/7wyrf5fogxcqq

[martijnvg]

@benwtrent Looks like the test ran against 7.12.1, this upgrade test should only run with 7.17.latest as old version. I will update this test to skip if old version is any other version.

[martijnvg]

I opened #109574

[benwtrent]

@martijnvg I realize that it ran against old versions and its weird to me, but there must be a reason? I am pretty sure versions have to be specifically set somewhere.

Do we know why these old versions are being tested against?

[martijnvg]

Do we know why these old versions are being tested against?

I don't know. I actually surprised like you too. The PR that added these tests didn't have a test-full-bwc label, because I thought we were only testing against 7.17.latest for 7.x bwc tests and not against older 7.x minors. When I saw this test failure, I thought my assumption was just wrong. But right now I'm not sure anymore.

[martijnvg]

We do document that from 7.x to 8.x, a cluster must first be on 7.17.x before doing the actual 8.x upgrade: https://www.elastic.co/guide/en/elastic-stack/8.14/upgrading-elastic-stack.html#prepare-to-upgrade

So I don't think we need to test upgrading from 7.16 or earlier to 8.x?

[benwtrent]

@martijnvg I am not 100% a PR gets tested against 7.17 at all unless the test-full-bwc tag is added :/

[mark-vieira]

We do document that from 7.x to 8.x, a cluster must first be on 7.17.x before doing the actual 8.x upgrade: https://www.elastic.co/guide/en/elastic-stack/8.14/upgrading-elastic-stack.html#prepare-to-upgrade

So I don't think we need to test upgrading from 7.16 or earlier to 8.x?

We do, because we test the upgrade path you describe. So we don't do rolling upgrade tests, but we still run full cluster restart tests that start a 7.16 cluster, upgrade to 7.17, and then upgrade to the current version.

@martijnvg I am not 100% a PR gets tested against 7.17 at all unless the test-full-bwc tag is added :/

Every PR is tested against the HEAD of the 7.17 branch, even without that label. The label will add testing for previously released version of Elasticsearch.