Implement streaming decompression#26
Draft
LordMike wants to merge 5 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial implementation of streaming decompression, written by OpenAI Codex. This is following my suggestion in #25, and the following benchmarks of compression, where zlib stands out as good. To combat the memory requirements though, we need to use a smaller Window Size, and we need to restructure the uzlib api a bit to support it. No API we could use, has a low state size while also offering streaming APIs, so we've adapted uzlib to get that.
od_zlib_stream_push,od_zlib_stream_poll, ..OPENDISPLAY_DECOMPRESSION_CHUNK_SIZEOPENDISPLAY_ZLIB_WINDOW_BITSTODO
ZIP, likeSTREAMING-ZIP, would signal clients that we support 512 byte window sizes.ZIP + STREAMING-ZIPmeans any window size is possible.STREAMING-ZIP, we know it supports deflate, but only at a 512 byte window size. Old clients will not understandSTREAMING-ZIP, and will assume compression is not possible (as today)windowSize=9when compressing forSTREAMING-ZIP-only devices; and to not early exit if the compressed bytes would be too large.TARGET_LARGE_MEMORY(?)Fixes #25