fix404ExceptionOnSparkAfterSplit

[xinlian12]

Description

When a split happens, customer may get a 404 exception.
Example stackTraces:
First getting a 410/1002

2/07/05 18:09:53 WARN TransientIOErrorsRetryingIterator: Transient failure handled in TransientIOErrorsRetryingIterator.hasNextInternal - will be retried (attempt#1) in 3586ms
{"ClassName":"GoneException","userAgent":"azsdk-java-cosmos/4.30.0 Linux/5.4.0-1080-azure JRE/1.8.0_302","statusCode":410,"resourceAddress":null,"innerErrorMessage":"Epk Range '{\"min\":\"05C1AFF91793C0\",\"max\":\"05C1B18FD52380\"}' is gone.","causeInfo":null,"responseHeaders":"{x-ms-substatus=1002}"}
	at azure_cosmos_spark.com.azure.cosmos.implementation.feedranges.FeedRangeEpkImpl.lambda$populateFeedRangeFilteringHeaders$2(FeedRangeEpkImpl.java:192)

Then getting a 404:
2/07/05 18:09:57 INFO TransientIOErrorsRetryingIterator: Attempting to cancel oldPagedFlux, Context: n/a
22/07/05 18:09:57 WARN ChangeFeedFetcher$FeedRangeContinuationSplitRetryPolicy: Exception not applicable - will fail the request. Context: n/a
{"ClassName":"NotFoundException","userAgent":"azsdk-java-cosmos/4.30.0 Linux/5.4.0-1080-azure JRE/1.8.0_302","statusCode":404,"resourceAddress":null,"innerErrorMessage":"Entity with the specified id does not exist in the system. More info: https://aka.ms/cosmosdb-tsg-not-found-java: Stale cache for collection rid 'HBpFAO8LmCQ='.","causeInfo":null,"responseHeaders":"{}"}
	at azure_cosmos_spark.com.azure.cosmos.implementation.feedranges.FeedRangeEpkImpl.lambda$populateFeedRangeFilteringHeaders$2(FeedRangeEpkImpl.java:180)
	at azure_cosmos_spark.reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)
	at azure_cosmos_spark.reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)

In this PR, it includes two fixes:

1. Return the correct shouldRetryResult after FeedRangeContinuationSplitRetryPolicy.handleSplit, so the process will continues instead of throwing back to TransientIOErrorsRetryingIterator.
2. The reason we are getting 404 exception is because after the the FeedRangeContinuationSplitRetryPolicy.handleSplit,there is a range(Min = Max, isMinInclusive = true, isMaxInclusive = false) being generated, which causing no overlapping ranges in theFeedRangeEpkImpl.populateFeedRangeFilteringHeaders`, hence a client side 404 is being generated. The fix is to use the correct isMinInClusive, isMaxInclusive flag.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes - Included Tests SparkE2EChangeFeedSplitITestfrom PR: Adding a unit test to reproduce the NotFoundException in change feed after partition split #29980

[azure-sdk]

API change check

API changes are not detected in this pull request.

[FabianMeiswinkel]

Thanks Annie!

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[FabianMeiswinkel]

Annie - please ping me offline. Want to understand whether after the split we still have three ranges - one with min==max (but proper flags) - if so, that is a bug we should address as well.

[xinlian12]

Annie - please ping me offline. Want to understand whether after the split we still have three ranges - one with min==max (but proper flags) - if so, that is a bug we should address as well.

By passing the proper flags, there is no range with min=max any more

[FabianMeiswinkel]

LGTM - thanks!

[xinlian12]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[xinlian12]

/azp run java - cosmos - spark

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).