Change feed mode switch error handling

[jeet1995]

Objective

The objective of this PR is to detect scenarios where the customer starts an AllVersionsAndDeletes CFP instance following stopping the LatestVersion CFP instance with an already initialized lease container (and vice-versa). This is to prevent scenarios where AllVersionsAndDeletes picks up a continuation from the lease outside its retention period or picks up a lease with StartFrom as BEGINNING which doesn't apply for AllVersionsAndDeletes change-feed mode.

Implementation

Transition 1: LatestVersion using EPK-range based lease to AllVersionAndDeletes:

1. Complete the lease store initialization flow within BootstrapperImpl.
2. Upon lease store initialization completion, query for EPK-range based lease documents whose id starts with a given lease prefix (or an empty prefix). This is done to see if the CFP instance in the bootstrapping phase will reuse leases or not.
3. If there exist leases, then extract the Mode property from the ContinuationToken of the lease document and compare it with the ChangeFeedMode value of the CFP instance in bootstrapping phase.
4. Throw an IllegalStateException if there is a mismatch in the Mode value from the fetched lease and the ChangeFeedMode value of the CFP instance in bootstrapping phase.

Transition 2: AllVersionAndDeletes to LatestVersion using EPK-range based lease:

1. Complete the lease store initialization flow within PkRangeIdVersionBootstrapperImpl.
2. Upon lease store initialization completion, query for query for EPK-range based lease documents whose id starts with a given lease prefix (or an empty prefix). This is done to see if the CFP instance in the bootstrapping phase will reuse leases or not.
3. If there exist leases, then extract the Mode property from the ContinuationToken of the lease document and compare it with the ChangeFeedMode value of the CFP instance in bootstrapping phase.
4. Throw an IllegalStateException if there is a mismatch in the Mode value from the fetched lease and the ChangeFeedMode value of the CFP instance in bootstrapping phase.

Transition 3: AllVersionAndDeletes to LatestVersion using Pk-range based lease:

• In this scenario, when LatestVersion CFP bootstraps - it can't pick up an EPK-range based lease as the CFP instance fetches leases / .info documents whose id embeds the name-based id of the database and feed container. An EPK-range based lease & its respective .info document has id which embeds the resource id (system-generated) of the database and feed container.
• Hence, in this scenario, the CFP instance will either bootstrap from pre-existing lease documents which are also PkRange-based leases (provided the lease prefix in the id of the lease document and that of the CFP instance match) or create its own Pk-range based leases.

Transition 4: LatestVersion using Pk-range based lease to AllVersionAndDeletes:

• In this scenario, when AllVersionAndDeletes CFP bootstraps - it can't pick up a PK-range based lease as the CFP instance fetches leases / .info documents whose id embeds the resource id of the database and feed container. A PK-range based lease & its respective .info document has id which embeds the name-based id (user provided) of the database and feed container.
• Hence, in this scenario, the CFP instance will either bootstrap from pre-existing lease documents which are also EPK-range based leases (provided the lease prefix in the id of the lease document and that of the CFP instance match) or create its own EPK-range based leases.

Design decisions

1. When validating change-feed modes, only 1 lease document needs to be fetched for a given lease prefix. The assumption is that all lease documents using the same lease prefix pertain to the same feed and therefore are homogeneous in terms of the value of Mode. This limits RU-usage on the lease container.
2. Say the lease left behind by a prior-running CFP instance has a null continuation token - this scenario should not error out on a change-feed mode switch as the subsequent-running CFP instance could still try and use the lease with its own ContinuationToken.

[azure-sdk]

API change check

API changes are not detected in this pull request.

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[kushagraThapar]

Missing changelog.

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

LGTM, thanks

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

Looks to be a regression in several split-based tests not finishing in time - tests pass locally. Change unrelated to splits so merging as is.

[jeet1995]

/check-enforcer override

[jeet1995]

Closes #38577