Compress 0x55aaf33f blobs better

[hvenev]

It looks like recent EEPROM versions contain blobs (magic 55aaf33f) compressed with an algorithm similar to LZJB. However, the files are not compressed optimally. For example, memsys01.bin is currently 12132 bytes compressed and 20492 bytes uncompressed. However, it can be compressed down to 11814 bytes. In total, over 4K can be saved in the entire EEPROM image.

I've provided an optimal (in terms of compression ratio; runtime and memory usage are not great, but it should not matter for 512K files) implementation of the compression program here: https://git.venev.name/hristo/rpi-eeprom-compress/tree/compress.c

Alternatively, you can switch to lz4, which is also quite easy to decompress, and provides a better compression ratio (for example, 10663 bytes for the file I mentioned above).

[timg236]

Thanks for the suggestion.

I'm open to to the idea of improving the compression algorithms and LZ4 was originally considered but the current algorithm won because the code size is smaller which significant when the bootloader is run from the L2 cache.

Changing the algorithm now would mean that new bootloader images are incompatible with current start.elf which also reads some of those blobs so it's hard to justify that for a 4K saving.

Leaving it open as a future enhancement

N.B. The license would probably need to be 2-clause compatible BSD because we would want anyone who uses bootloader related code to be able to statically link code.

[hvenev]

N.B. The license would probably need to be 2-clause compatible BSD because we would want anyone who uses bootloader related code to be able to statically link code.

Static linking is fine if you provide the code together with the binaries, right?

Edit: by "the code", I mean all the code that goes into the statically linked binary, together with build scripts and configs and so on.

Edit 2: I think the algorithm I provided is compatible with the current decompressor. I have tested it against my decompressor implementation, which reads the original EEPROM just fine. The savings from LZ4 would be 39K (not counting decompressor code).

[timg236]

"Derivatives works (including modifications or anything statically linked to the library) can only be redistributed under LGPL, but applications that use the library don't have to be."
https://tldrlegal.com/license/gnu-lesser-general-public-license-v3-(lgpl-3)

I'm not sure whether a single object file constitutes a library but IANAL and that's detail :)

[hvenev]

Just one quick question: Is the algorithm you used well-known? The closest thing I could find was LZJB, which encodes a 12-bit offset and a 4-bit length, whereas your algorithm uses 8-bit offsets and 8-bit lengths.

[cleverca22]

why are the 8 different memsys files?
ive noticed that they are fairly similar, could you just store one compressed copy and 7 binary patches?

which files does start4.elf read from the eeprom? could those still use the old algo (at the cost of a larger bootcode.bin to handle both)

[hvenev]

I can confirm that my Pi 4 boots when the blobs are re-compressed with https://git.venev.name/hristo/rpi-eeprom-compress/tree/compress.c.

[timg236]

Closing. Not changing the compression now. In future we might move to something like LZ4.