Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable receiver to parse chunked file snapshot chunks #19490

Merged
merged 47 commits into from
Jun 26, 2024

Conversation

EuroLew
Copy link
Contributor

@EuroLew EuroLew commented Jun 18, 2024

Description

Built on the changes in #19361 as new chunk fields are needed.

  • Removed file existence checks as they will fail for chunked files.
  • Now writes file using the chunk position so order of chunks doesn't matter (for writing files will still fail checksums)
  • Now support partially building metadata if file is chunked.
  • Now enforces order of chunks using chunkId in PassiveRole

Related issues

closes #12795

@github-actions github-actions bot added the component/zeebe Related to the Zeebe component/team label Jun 18, 2024
@EuroLew EuroLew marked this pull request as draft June 18, 2024 10:33
@EuroLew EuroLew marked this pull request as ready for review June 18, 2024 12:51
Base automatically changed from el-12795-snapshot-file-chunks to main June 18, 2024 18:45
@EuroLew EuroLew force-pushed the el-12795-snapshot-file-receive-chunks branch from 692c311 to 8396cc6 Compare June 19, 2024 08:51
@EuroLew EuroLew marked this pull request as draft June 19, 2024 09:13
@EuroLew EuroLew force-pushed the el-12795-snapshot-file-receive-chunks branch from a0f5efe to 016edeb Compare June 19, 2024 10:55
@EuroLew EuroLew marked this pull request as ready for review June 19, 2024 14:09
@github-actions github-actions bot added component/operate Related to the Operate component/team component/tasklist Related to the Tasklist component/team labels Jun 20, 2024
@EuroLew EuroLew force-pushed the el-12795-snapshot-file-receive-chunks branch from 6b85129 to 0834636 Compare June 24, 2024 10:29
Copy link
Member

@lenaschoenburg lenaschoenburg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Let's not close #12795 yet, we need to find a sensible default value that's lower than 1GiB

@EuroLew EuroLew force-pushed the el-12795-snapshot-file-receive-chunks branch from 25c0fd3 to f199c5a Compare June 24, 2024 15:49
@EuroLew EuroLew added this pull request to the merge queue Jun 25, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 25, 2024
@EuroLew EuroLew added this pull request to the merge queue Jun 26, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 26, 2024
@EuroLew EuroLew added this pull request to the merge queue Jun 26, 2024
Merged via the queue into main with commit 126b96c Jun 26, 2024
49 checks passed
@EuroLew EuroLew deleted the el-12795-snapshot-file-receive-chunks branch June 26, 2024 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/operate Related to the Operate component/team component/tasklist Related to the Tasklist component/team component/zeebe Related to the Zeebe component/team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Split files into smaller chunks when replicating snapshots
2 participants