zlib compressed updates via flashz lib

[vortigont]

This is proof-of-concept implementation of zlib-compressed images updating via esp32-flashz lib.
It allows transparent decompression and flashing with either compressed or uncompressed images.
A full featured implementation would require more work, especially with signed+compressed images.
Not sure if this idea is of any use for esp32FOTA project's roadmap. So any feedback is much appreciated.

[tobozo]

hey @vortigont thanks for that POC, also perfect timing 👍

I've been experimenting with another decompression library (esp32-targz), and it's very comforting to see the similarities with your integration.

I'm still evaluating the benefits of having the stream source overridable e.g. optionnally hook some open/read/error callbacks from the sketch and have the callbacks use HTTP (default), esp32-FlashZ, esp32-targz, fs::File, Serial or any other valid stream source during the update.

So far the most complex scenario is with .tar.gz format, where a single decompression stream can handle separate contexts (spiffs and firmware) while the partitions order in the archive isn't controlled by esp32fota.

Thanks to your POC I now have two decompression libraries to experiment with and much less choice paralysis for the implementation 🥳

[vortigont]

Hi @tobozo you are welcome!
I was thinking to get some feedback before deep diving into this :) tnx for your reply.
Pls, feel free to share your ideas if that could of any use here. esp32-flashz is lightweight and transparent, although it's just a decompressor, not a full featured OTA solution. Supporting tar.gz could give more features as I think, but it's an overcomplication for firmware IMHO :) Unless there are plans to implement FS restoration from the archive not from the image :)
I like the idea to have separate manifest in this lib, so it is pretty easy to specify multiple firmware files there.

[tobozo]

esp32-targz needs some work before being usable with esp32fota as it bundles its own Update logic without using inheritance, so I'll leave this for later

I haven't tested yet (still adjusting the CI workflow) but the optional use of esp32-flashz is possible, these are the macros I came up with to keep the impact on the code mininal:

#if __has_include(<flashz.hpp>) || defined ESP32_FLASHZ_H
  #include <flashz.hpp>
  #define F_Update FlashZ::getInstance()
  #define F_UseZlib (_stream->peek() == ZLIB_HEADER)
  #define F_canBegin() mode_z ? F_Update.beginz(updateSize, partition) : F_Update.begin(updateSize, partition)
  #define F_abort() if (mode_z) F_Update.abortz()
  #define F_writeStream() mode_z ? F_Update.writezStream(*stream, contentLength) : F_Update.writeStream(*stream)
#else
  #define F_Update Update
  #define F_UseZlib false
  #define F_canBegin() F_Update.begin( updateSize, partition )
  #define F_abort() { }
  #define F_writeStream() F_Update.writeStream( *_stream );
#endif

https://github.com/chrisjoyce911/esp32FOTA/tree/0.2.3

[vortigont]

Should be fine for testing I suppose.
This one #define F_abort() { } must be #define F_abort() { Update.abort(); } I suppose.
It so frustrating that UpdaterClass methods are not virtual, things could be much easy that way.

[tobozo]

got some satisfying result after a few changes (published on 0.2.3 branch), and reduced the friction in the sketch to this:

#include <flashz.hpp>
#include <esp32FOTA.hpp>

⚠️ the archive in the manifest should be named with .z extension e.g. firmware.z and be served with the application/x-compress mimetype.

I'm not sure this choice is appropriate though, please advise if you think of a better extension/mimetype.

apparently files created with gzip don't work with flashZ library, for some reason I thought zlib would handle gzipped content natively, is there a big difference between those formats?

[vortigont]

actually it should be just fine with default application/octet-stream, same way as .bin file. I used .zz extension as produced with pigz tool. File format is auto-detected via first magic byte. So it is OK to specify either .bin or .zz in manifest.
gzip format is using same compression, but it also contains a header with source file metadata. It is not supported by miniz in esp32 ROM.

Need to think about signed updates though. I'm not that deep into this. As I can tell from the code a signature is in first 512 bytes of a .bin file. Is that a standardization of some kind or just an implementation in esp32FOTA? Looks weird, usually a signature is being placed at the end or even in a detached file.

[tobozo]

thanks, your explanation makes sense

I may have an idea for sig-checking archive: instead of being concatenated, the signature file is optionnally left in a separate file firmware.sig, so if this .sig file isn't specified in the json meta then there is no (flashz + check_sig) support.

[edit] it was also suggested to provide the signature via a http header

the signature logic happening after the partition is flashed still doesn't feel right though

[vortigont]

IMHO a detached signature sounds as the most flexible solution. It could be specified either as separate file in manifest or even in the manifest itself (much better approach than HTTP header as for me). And it could be on user's decision either to use it mandatory, or opportunistically. Any way it is good to have direct access to a plain firmware file, it could downloaded/flashed via serial, etc... without the need for specific update lib capable of signature verification.

I was under impression that IDF has some embedded features to keys management, encrypting/verifying firmware but I've never researched this area much yet. It might worth investigating so not to reinvent the wheel :)

[tobozo]

I don't know esp-idf or even cryptographic practices well enough to tell but any change affecting a CONFIG_* macro from the sdkconfig may be a pain in the ass to land in arduino-esp32, and I haven't seen an Arduino (vanilla) example demonstrating that yet.

Using a http header to send the signature was suggested in the past though.

Given all the suggested options I could find in the issue tracker, the signature can be:

• Not required
• Prepended to the binary (problematic with flashz)
• In a separate file (adds an extra http request)
• A JSON property (can create JSON growth and dynamic buffer problems)
• A HTTP header value (not applicable on all webservers)

// signature validation available methods
enum SigType_t
{
  FOTA_SIG_PREPEND,       // signature is contatenated with file (default)
  FOTA_SIG_FILE,          // signature is in a separate file
  FOTA_SIG_JSON_PROPERTY, // signature is a JSON property
  FOTA_SIG_HTTP_HEADER    // signature is a HTTP header
};


struct FOTAConfig_t
{
  const char*  name { nullptr };
  const char*  manifest_url { nullptr };
  SemverClass  sem {0};
  bool         check_sig { false };
  SigType_t    type_sig {FOTA_SIG_PREPEND};
  bool         unsafe { false };
  bool         use_device_id { false };
  CryptoAsset* root_ca { nullptr };
  CryptoAsset* pub_key { nullptr };
  FOTAConfig_t() = default;
};

The tradeoffs are suggesting that all or none of those situations should be implemented, maybe the signature check needs to be abstracted away from the main class ?

Anyway I'm still busy researching how to add gz support so I probably won't be playing with signatures any time soon.

[vortigont]

IMO implementing all options is an overkill :) considering that ESP-IDF has native secure boot (maybe too complex but more feature reach).
I can only suggest to create a poll and get some feedback from community. Maybe the simplest one with detached sig file could be just good enough.

[tobozo]

I can only suggest to create a poll

good point, I'm not sure if github supports polls but there's an upvote/downvote system with the discussions

@chrisjoyce911 is it possible to enable the discussions tab on this repository?

@vortigont I managed to get gz support to work by using a customized copy of the FlashZ class with some esp32-targz static methods 🥳 thanks for your inspiration! 🙇

[vortigont]

@vortigont I managed to get gz support to work by using a customized copy of the FlashZ class with some esp32-targz static methods partying_face thanks for your inspiration! bow

Cool! Glad it was useful in adjacent areas :)

[tobozo]

closing this PR as esp32-flashz is now integrated