Skip to content

Implement streaming decompression#26

Draft
LordMike wants to merge 5 commits into
OpenDisplay:mainfrom
LordMike:streaming-zlib-vendor-uzlib
Draft

Implement streaming decompression#26
LordMike wants to merge 5 commits into
OpenDisplay:mainfrom
LordMike:streaming-zlib-vendor-uzlib

Conversation

@LordMike
Copy link
Copy Markdown
Contributor

@LordMike LordMike commented May 8, 2026

Initial implementation of streaming decompression, written by OpenAI Codex. This is following my suggestion in #25, and the following benchmarks of compression, where zlib stands out as good. To combat the memory requirements though, we need to use a smaller Window Size, and we need to restructure the uzlib api a bit to support it. No API we could use, has a low state size while also offering streaming APIs, so we've adapted uzlib to get that.

  • Pulls in uzlib as code, rather than a github reference
  • Patches uzlib a lot, removing the one-shot API in favor of api methods that mimic heatshrink's API: od_zlib_stream_push, od_zlib_stream_poll, ..
  • Replaces the decompression of both the compressed full image and compressed partial image writing, with this new API
  • Reduces memory for decompression to about 2 KB (for 512 byte window size):
    • 1192 bytes for the state
    • 256 bytes for the output buffer (when reading from our decompressed stream), controlled by OPENDISPLAY_DECOMPRESSION_CHUNK_SIZE
    • 512 to 32768 bytes for window size, controlled by OPENDISPLAY_ZLIB_WINDOW_BITS

TODO

  • Find a way to signal support for smaller window sizes. Adding a new capability flag like ZIP, like STREAMING-ZIP, would signal clients that we support 512 byte window sizes.
    • Clients could also infer that ZIP + STREAMING-ZIP means any window size is possible.
    • If a device only offers STREAMING-ZIP, we know it supports deflate, but only at a 512 byte window size. Old clients will not understand STREAMING-ZIP, and will assume compression is not possible (as today)
  • Tweak py-opendisplay to support the above, and then use windowSize=9 when compressing for STREAMING-ZIP-only devices; and to not early exit if the compressed bytes would be too large.
  • Decide on which build targets get which build constants, and if some MCUs now support zip, where they didn't before.
  • Remove constants like TARGET_LARGE_MEMORY (?)

Fixes #25

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support streaming decompression

1 participant