Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider #41099
[SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider #41099
Changes from 9 commits
da54bb5
5bda37f
8bc8552
be7846b
0512806
974a47e
2d81dc2
9df1340
1b0f94c
bf30cf1
19b8355
4ff442f
b3cc436
0ef2fc9
e59d43f
1e46adc
5f53f49
5910fb7
0ee93c1
7c65cac
b2ead71
36d9ae2
4aa2605
ff4cff9
82e7168
4109c29
573e0e9
fc8c1bd
6480621
590f21c
bb58556
14d7b91
99f5e0a
b1d3809
050f214
f723840
da7aa99
4f9b0a7
7d52ed5
91d0075
5732fbd
6cb6d0b
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is case class preferred pattern for something like this ? cc - @HeartSaVioR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1000 entries feel relatively small to trigger a snapshop upload. Consider to remove the limit, or make it very large, such as 1 million.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This does not trigger a snapshot upload, it simply flush and create a local checkpoint. I can increase it to 10K, which guarantee that the changelog replay between every 2 snapshot is < 50k records.