BWC record checker tool

[imotov]

During the switch to BKD shapes and the recent geo_shape refactoring effort we have discovered a set of issues and inconsistencies in the way some corner cases are handled for different geometries and formats. Some examples: #43916, #43837, #43775, #43851, #35813.

We would like to fix these issues but at the same realize that fixing these bugs and inconsistencies might affect the users that are currently accidentally rely on them. Since most of these issues can be classified as bugs, we would like roll out the change that will fix these issues in one of 7.x releases instead of waiting for 8.0. However, we don't have a mechanism to notify users about breaking changes in minor releases. Moreover the current upgrade checker might not be the best suited for this particular use case since in order to check for the presence of records affected by these changes will will have to scan all records.

In the team discussion of this issue we came to the conclusion that we need to introduce a stand alone utility that can be executed by users in order to detect the records affected by the change. We would like to extend this discussion to a broader group to see if other options or solutions available.

[elasticmachine]

Pinging @elastic/es-analytics-geo

[imotov]

During the discussion in FixItFriday, we couldn't reach a conclusion that such tool is necessary and should be implemented. Concerns about overheard of maintaining this tool were raised. While we have other instances where such tool could have been beneficial (such as date checks) we typically don't provide such tools. One of the main technical concerns was practicality of such tool considering the requirement for full index scan. The overhead of running the checker on a large index would be comparable to re-indexing of such index, so a recommendation to just reindex might be a more efficient solution for users concerned that their index contain affected records.

We agreed that fixing these bugs in a minor version should be acceptable. We didn't reach a conclusion if we should maintain the old behavior for indices that were created before the bug fixes were introduced, it seems like that would create too much maintenance overhead considering that we are already maintaining "legacy" and BKD-backed shape indexing logic.

[nknize]

This is a good synopsis and I agree with the concerns raised. +1 to not worry about maintaining a checker tool and just fix the bugs in the minor version with documented breaking changes.