Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Allow larger files #6
As of #1 and its implementation in #2, the
Increase addressable archive size bye enforce write alignment
If we align the start of each file written to the archive by
Using more bits for addressing
Instead of splitting the 64bit integer into two 32bit integers, we might as split it into, for example 40bit and 24bit -- shifting the limits to 1TB archives containing files up to 16MB. This should work very well for the rustdoc use case.
This can of course be combined with the alignment option described above, to yield (2^36)*(2^4)=1TB archive files containing 4 byte aligned files up to 2^28=268MB.
Patching fst to allow other value types
The fst docs mention that in the future, it should be possible to map to something other than a
Please correct my math, it's late and I had a few beers.
There is another solution not mentioned here: simply store the address and length separately. When writing the file contents into the archive, prepend the contents with an 8 byte integer representing the length.
Edit: also, you have a typo, just letting you know!
Ohhh, how could I forget to mention that? Thank you! This trades in a bit of bit shift logic for reading these 8 bytes and using them,at probably no performance cost. This approach works best after we specified the archive file format a bit more for future compatibility (and sanity).…
On Mon, 12 Nov 2018, 04:04 Josh Leverette, ***@***.***> wrote: There is another solution not mentioned here: simply store the address and length separately. When writing the file contents into the archive, prepend the contents with an 8 byte integer representing the length. When fst hands you back the u64, let that u64 just represent the absolute byte position of the file content in the archive. Seek to that position in the file, read the 8 bytes representing the length, then read the content. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABOX33Y7bB5w7EGTJsSoDtpYhA1txxzks5uuOVHgaJpZM4X0BF6> .