-
Notifications
You must be signed in to change notification settings - Fork 83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decompression of a stream in output chunks without dictionary #32
Comments
As an update: I found a way to decompress only per chunk of X bytes to RAM, then write that to flash memory (my use-case is a compressed firmware image for OTA firmware upgrades), then reset the RAM output buffer ( By detecting whenever the A first proof-of-concept is implemented here: maximevince@c528e63 After testing this, I realized there are many use-cases for this: But there might be a cleaner way to implement this. Instead of introducing the extra Let me know what you think, and what kind of interface you would prefer, and I can send a pull-request. |
For the exact same purpose, I had initiated a similar job, FWIW, not proposed as pull-request. I implemented your idea through macros ALIGN_READ / ALIGN_WRITE that have default definitions for legacy behavior but that can be redefined to something else (reading from flash) when dictionary is disabled. This functionality you propose would be indeed very useful on small targets. This idea was abandoned because the esp8266 arduino OTA second stage happens at reboot time when there's still plenty/enough of temporary ram to be able to uncompress and reflash the FW. I'd like to thanks once again @pfalcon for this library. Credits are already given in the esp8266 arduino main readme. |
It's been a while, but I believe this Pull Request accomplishes what you are looking for - "deflate a stream without a memory buffer". It allows for reading compressed input, and writing the decompressed output, a single byte at a time. It's not a very clean PR, but I did put this into a production device for reading a compressed factory firmware image from SPI flash, and uncompressing it a word at a time to write to internal program flash, and it's been working great for 4+ years. I should probably clean up the PR and re-submit it at some point. |
Apparently @mikevoyt your idea is welcome by owner but further work seems to be required for your PR to be accepted. |
There was a change which I had in my git stash for quite some time, and which I now cleaned up and pushed: 2aeab6c. If you look at it carefully, you'll notice that what it would take to support "copy LZ strings from a backing store [instead of memory]" - is to allow to override memcpy there with a user-defined function. If someone would like to prepare a patch, please follow https://github.com/pfalcon/change-control-manifesto . Thanks. |
Oh, and I would like to remind that zlib supports configurable dictionary size, so it doesn't have to be "32KB". Powers of 2 from 512 to 32KB are supported. As an example, Pycopy packages use 4K dictionary: https://github.com/pfalcon/pycopy-lib/blob/master/sdist_upip.py#L34 |
Nice, I just rebased on top of your new commits. |
Thanks for your submission. However there was no (further) activity
Thanks for your understanding! |
Closing due to inactivity. |
From the README, I gather:
That leaves me with:
Option 3: What I am trying to do is to decompress 128 bytes (or any other value) of bytes at a time (the 'chunk' size) from a stream. (Streaming the input in 128b chunks works fine!)
This 128 byte output works, once.
After that, if I re-init
d.dest_start = d.dest = uz_data_dest;
,this will cause an error -3 when decompressing the next series of bytes.
I found out that this is because of the LZSS offset (
lzOff
) looking for previous occurrences of the same data, relative to the current position of the output buffer (henced->dest[0] = d->dest[d->lzOff]
). But since the output buffer has been re-initialized, these relative offsets are not valid anymore. So that sounds logical.I gather that's why in this case, a
dict
is needed. Sure enough, I added a 32kB rambuffer to be used as dict, and it worked.However, on the target embedded system, I don't have 32kB of RAM...
So the question is:
With some explanation about the exact usage of these case, I would be glad to provide some example code to be included in the repo, if that would help as documentation for the project!
Thanks for the project! 👍
The text was updated successfully, but these errors were encountered: