Memory-friendly cartridge format #37
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cute-the-niini
added
enhancement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This required some messing around with the cartridge format (again "Orz), but at least now it should stabilise (famous last words ahaha :'>). This is a breaking change as the parsers for previous versions was removed --- so existing cartridges need to be re-packaged with the new version to be installed.
There are quite a few things that this patch takes care of, though it doesn't solve all memory issues in Kate just yet:
Cartridges now have no (practical) size limitation, however individual files must fit entirely in memory, so there are practical limitations on those. The reason for this is because the browser cryptographic APIs that Kate uses for verifying the cartridge's contents do not (yet?) support streaming use cases, so the whole data needs to be load in memory anyway for integrity checking. The old 1.4GB arbitrary limit on the KART file size has been removed as a result of this. Installation and cartridge parsing requires O(max(f1.size, ..., fN.size)) space --- so the highest size of each individual file.
Theoretically file sizes for Kate on web are limited to 2**53, but that's 8192 TB so no real concern there. Cartridge offsets are Uint64 by the specification, but web APIs only support 64-bit floats for offsets and sizes.
Cartridge format is now friendly to stream-writing and random-access. By placing files first and the location header last in the file we don't have to back-patch any references and we have all necessary data to build a proper file location table with specific offsets into the binary. That means we can run a cartridge with a file reference without having to install it first, however Kate will continue to install cartridges as before.
That said, the importer doesn't make use of any of this yet because it has no disk storage to write data to. The partitioned file storage has to be reified as a user-land API in order for the importer (and the future publisher) to be able to create cartridges with bounded memory usage. Likewise, the installation API needs to change to take a file reference instead of arbitrary bytes.
There's some work that still needs to be done with signatures (they aren't verified yet! and you can't sign cartridges yet!). This does prepare things a bit. Cartridges can now be signed without relocating any files by overwriting only the metadata and header sections. Note that the metadata section is not fixed-size, so both sections need to be re-written if a cartridge is to be signed after-the-fact. Kate supports multiple signatures to be attached and they all sit in the same metadata section. Files are not signed directly, but the file offset table (with the locations and integrity hashes) is signed. This also avoids needing to keep files around for signing given web crypto API limitations.