OldRepositoryAccessIT.testOldRepoAccess fails on Lucene Snapshot #107168
Comments
benwtrent
added
:Search/Search
|
Pinging @elastic/es-search (Team:Search) |
benwtrent
added
low-risk
|
Its a blocker for Lucene Snapshot, but I don't know how to communicate that via labels. And I don't want folks to think its a blocker for anything else. So, low-risk for now. |
|
This is due to a recent Lucene change, however I don't think that the Lucene change is the problem but that it just makes an existing problem visible now that the numeric comparator pull a point tree eagerly in its constructor rather than lazily. We were not seeing the problem before because we're searching over less than Lucene's sorting logic assumes that if there are points, then they can be used for dynamic pruning. But this doesn't work since the old file format only supports metadata operations. A fix could consist of hiding points, e.g. by disabling them in field infos, but then the index wouldn't be able to use them for |
In order to know whether it can apply dynamic pruning using the points index, Lucene simply looks at whether a field has points. Unfortunately, this doesn't work well with our support for archive indexes, where numeric/date fields report that they have points, but they only support metadata operations on these points (min/max values, doc count), with the goal of quickly filtering out such archive indexes during the `can_match` phase. In order to address this discrepancy, dynamic pruning is now disabled when mappings report that a field is not indexed. This works because archive indexes automatically set `index: false` to make sure that filters run on doc values and not points. However, this is not a great fix as this increases our reliance on disabling dynamic pruning, which is currently marked as deprecated and scheduled for removal in the next Lucene major. So we'll need to either add it back to Lucene or find another approach. Closes elastic#107168
In order to know whether it can apply dynamic pruning using the points index, Lucene simply looks at whether a field has points. Unfortunately, this doesn't work well with our support for archive indexes, where numeric/date fields report that they have points, but they only support metadata operations on these points (min/max values, doc count), with the goal of quickly filtering out such archive indexes during the `can_match` phase. In order to address this discrepancy, dynamic pruning is now disabled when mappings report that a field is not indexed. This works because archive indexes automatically set `index: false` to make sure that filters run on doc values and not points. However, this is not a great fix as this increases our reliance on disabling dynamic pruning, which is currently marked as deprecated and scheduled for removal in the next Lucene major. So we'll need to either add it back to Lucene or find another approach. Closes #107168
CI Link
https://gradle-enterprise.elastic.co/s/5pageamepci42
Repro line
./gradlew ':x-pack:qa:repository-old-versions:javaRestTestBeforeRestart#5_0_0' --tests "org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess" -Dtests.seed=73644E41C915C560 -Dtests.locale=fi-FI -Dtests.timezone=America/Matamoros -Druntime.java=22
Does it reproduce?
Yes
Applicable branches
lucene_snapshot
Failure history
No response
Failure excerpt
But there are some interesting exceptions in the logs:
The text was updated successfully, but these errors were encountered: