introduce resumable backups #492
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vroldanbet
force-pushed
the
backup-improvements
branch
from
26afc6f to
f33f408
Compare
vroldanbet
force-pushed
the
backup-improvements
branch
from
f33f408 to
4eb56d4
Compare
derives the file name from the zed context name
adds a flag that allows the user defining the number of relationships they'd like to be sent per bulk export page
vroldanbet
force-pushed
the
backup-improvements
branch
from
4eb56d4 to
01a2035
Compare
this commit changes the backup create command so that a canceled backup can be resumed. A new marker file is added with the last written bulk export cursor. This was not added to the OCF file because OCF container is not meant for in-place updates, but streaming append operations. The marker is updated every time a bulk export page is successfully written. For convenience the command now also supports creating a backup without a file name: in that case it will derive the backup file name from the zed context name. The new logic detects several scenarios like: - a backup exists, but no marker exists (meaning it completed) - backup does not exist, but marker is left behind (it gets truncated) The marker file will be removed only when the backup completes. This new logic does not guarantee relationships may not get duplicated. When terminated gracefully the system should write the last page received and close and flush the files. But if the process was abruptly terminated (e.g. SIGKILL) it could lead to relationships being written to the OCF file, but the marker not being updated.
vroldanbet
force-pushed
the
backup-improvements
branch
from
01a2035 to
b13830c
Compare
alecmerdler
reviewed
Apr 22, 2025
Contributor
There was a problem hiding this comment.
Not a blocker, but I was able to get an error by running against SpiceDB seeded using the test data:
ERR terminated with errors error="error receiving relationships: %!w(<nil>): set flag --max-message-size=9793576 to increase the maximum allowable size
- print backup stats always, regardless termination reason - improve handling of ResourceExhausted error, instead of fixing all exit paths, wrap the command function so the error is always evaluated. - fixes malformed error message due to error variable overlap - made resilient to existing backup with empty marker file
Contributor
Author
@alecmerdler this is a known issue, which can be addressed by setting the I fixed the confusing error message in cea939a, thanks for reporting it! |
alecmerdler
reviewed
Apr 23, 2025
| module github.com/authzed/zed | ||
|
|
||
| go 1.23.8 | ||
| go 1.24 |
Contributor
There was a problem hiding this comment.
Should this be in a separate pull request?
Contributor
Author
There was a problem hiding this comment.
no because I needed it for t.Context()
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR modifies the backup create command to allow an incomplete backup to be resumed.
A new marker file is added with the last written bulk export cursor. This was not added to the OCF file because OCF container is not meant for in-place updates, but streaming append operations. The marker is updated every time a bulk export page is successfully written.
For convenience, the command now also supports creating a backup without a file name; in this case, it will derive the backup file name from the Zed context name. A new
page-limitflag has also been added to allow users to specify the number of relationships to ingest per page. This may be useful as a mechanism to throttle the number of relationships read from the datastore.The new logic detects several scenarios like:
The marker file will be removed only when the backup completes.
This new logic does not guarantee that relationships will not be duplicated. When terminated gracefully, the system should write the last page received, close, and flush the files. But if the process was abruptly terminated (i.e. SIGKILL) it could lead to relationships being written to the OCF file, but the marker not being updated.