Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split the chunk offset tables to reduce memory usage while dumping #5

Closed
ph0llux opened this issue Apr 7, 2023 · 0 comments
Closed

Comments

@ph0llux
Copy link
Collaborator

ph0llux commented Apr 7, 2023

To reduce the memory usage while dumping a very large amount of data (e.g. a physical disk with 20TB) to a single segment, the chunk offset table should be split in multiple tables - this tables should be written periodically into the Zff container (currently, the full table is cached in memory while dumping the data - a single table entry needs 2*8 bytes, so you need 500MB memory space for each TB of data by using a chunk size of 32kB).
At the end of the segment there should be an additional table which contains the appropriate offsets to the chunk-offset tables.
Due this tables are sorted HashMaps (or BTreeMaps), they can be variable in their size.

@ph0llux ph0llux added the Zffv3 label Apr 7, 2023
@ph0llux ph0llux added this to the Zffv3 milestone Apr 7, 2023
@ph0llux ph0llux closed this as completed Dec 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant