Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New compact format for snapshots #197

Closed
3 tasks
Tracked by #191
ohsayan opened this issue Aug 8, 2021 · 2 comments
Closed
3 tasks
Tracked by #191

New compact format for snapshots #197

ohsayan opened this issue Aug 8, 2021 · 2 comments
Labels
A-independent Architecture independent issue/PR C-enhancement New feature or request C-reliability This issue/PR relates to reliability C-storage Relating to storage C-tests This is related to the test suite D-server Related to the server L-Medium Requires a "medium" level of work P-high High priority

Comments

@ohsayan
Copy link
Member

ohsayan commented Aug 8, 2021

We'll need to implement a new format for compressing and storing snapshots since storing a directory, and transmitting it over a network might introduce additional maintenance complexity for sysadmins. We don't want that to happen and instead implement a single-file snapshot format for easy backups.

Tasks:

  • Implement new snapshot engine
  • Implement new snapshot compression format
  • Add tests
@ohsayan ohsayan mentioned this issue Aug 8, 2021
8 tasks
@ohsayan ohsayan added A-independent Architecture independent issue/PR C-enhancement New feature or request C-reliability This issue/PR relates to reliability C-storage Relating to storage C-tests This is related to the test suite D-server Related to the server L-Medium Requires a "medium" level of work P-high High priority labels Aug 8, 2021
@ohsayan
Copy link
Member Author

ohsayan commented Aug 8, 2021

We'll avoid linking against something like tar (through flate2 for example) due to a lack of customizability and the level of fine grained control that we need. At the same time, linking against a library statically might increase binary sizes or linking dynamically might introduce the infamous "library not found" issue across platforms. For all these reasons, we will implement our own portable format.

@ohsayan
Copy link
Member Author

ohsayan commented Aug 9, 2021

This is not a very performant way to export/dump data. Hence, we'll proceed to using something optimal: dump the files in the directory like we currently do

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-independent Architecture independent issue/PR C-enhancement New feature or request C-reliability This issue/PR relates to reliability C-storage Relating to storage C-tests This is related to the test suite D-server Related to the server L-Medium Requires a "medium" level of work P-high High priority
Projects
None yet
Development

No branches or pull requests

1 participant