[SPARK-53794][SS] Add option to limit deletions per maintenance operation associated with rocksdb state provider

[anishshri-db]

What changes were proposed in this pull request?

Add option to limit deletions per maintenance operation associated with rocksdb state provider

Why are the changes needed?

We see some instances where the changelog deletion can take a really long time. This means that for that partition, we also cannot upload full snapshots which affects recovery/replay scenarios. This problem is much more apparent on resource constrained clusters. So, we add an option to allow for incremental cleanup per maintenance operation invocation.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests

[info] Run completed in 17 seconds, 591 milliseconds.
[info] Total number of tests run: 8
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 8, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

Was this patch authored or co-authored using generative AI tooling?

No

[anishshri-db]

cc - @HeartSaVioR - PTAL, thx !

[liviazhu]

LGTM, thank you!

[dongjoon-hyun]

Hi, @anishshri-db , just a question, do you have any reason not to wait for an approval from the Apache Spark committers?

For some urgent cases, Apache Spark release managers can do the merging their PRs to unblock the release (and, of course, expects a late post-merge approval), but I'm just curious if you have those kind of exceptional cases in your Apache Spark 4.1.0 PR.

BTW, according to the PR code, I can give my +1 to meet the Apache Spark community requirements (if you missed the approval mistakenly). So, this is just a question and there is no big deal here.

[anishshri-db]

Hi @dongjoon-hyun - thanks for the pointer. Sorry not intentional, was not sure whether we always need committer stamp for committer authored PRs as well. Could you please stamp this one for me ? Will make sure to get committer stamp henceforth before merging. Thanks !

[dongjoon-hyun]

+1, LGTM.

No problem at all. I just wanted to remove any such an ASF issue from this PR . Thank you, @anishshri-db .