Rebuild version map when opening internal engine

[marregui]

With this change, we will rebuild the live version map and local checkpoint using
documents (including soft-deleted) from the safe commit when opening an internal
engine. This allows us to safely prune away _id of all soft-deleted documents as
the version map is always in-sync with Lucene index.

Relates #40741 (elastic/elasticsearch#40741)
Supersedes #42979 (elastic/elasticsearch#42979)

Port of elastic/elasticsearch#43202

[seut]

@marregui Any reasons to not backport the related test as well? https://github.com/elastic/elasticsearch/pull/43202/files#diff-8a78dedbc25426973364f21bdc81c27aR5946

[marregui]

That test uses a class that was renamed from DocIdSeqNoAndTerm to DocIdSeqNoAndSource in the context of this PR https://github.com/elastic/elasticsearch/pull/41661/files
I copied the test from ES 7.5 (and used DocIdSeqNoAndTerm) and when I run it I get

java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([795A50AD16F3298D:8D446012A0902F4C]:0)
	at org.elasticsearch.index.engine.InternalEngine.assertMaxSeqNoOfUpdatesIsAdvanced(InternalEngine.java:2779)
	at org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1100)
	at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:910)
	at org.elasticsearch.index.engine.TranslogHandler.applyOperation(TranslogHandler.java:75)
	at org.elasticsearch.index.engine.TranslogHandler.run(TranslogHandler.java:100)

I thought that at that point I would stop and reconsider.

[seut]

@marregui So why not backport elastic/elasticsearch#41661 first?
I think the mentioned test is related as it will validate the changes made by this PR.

[marregui]

Thank you, I thought about it, and that is partially the reason I stopped, to get feedback.

PR #9399

[marregui]

I believe I need to port elastic/elasticsearch#41161 as well, test assertMaxSeqNoOfUpdatesIsAdvanced defined in class InternalEngine will fail in:

        if (maxSeqNoOfUpdates == SequenceNumbers.UNASSIGNED_SEQ_NO) {
            assert config().getIndexSettings().getIndexVersionCreated().before(Version.ES_V_6_5_1);
            return true;
        }

this code is removed in PR41161

[seut]

@marregui Yes please backport elastic/elasticsearch#41161 upfront as well.

[marregui]

and that one requires this one: elastic/elasticsearch#39571 it is very simple.

[seut]

👍