PlatformIO

[Finn-Land]

In your project you mention issues with PlatformIO, what were those problems? I'm planning to use PlatformIO but if it's not feasible to use I would hope that you could tell me!

Thanks in advance!

[thinkty]

It's been a while so I might be wrong, the ESP8266_RTOS_SDK used in PlatformIO was a little outdated so some of the features in the latest SDK were missing. However, I think they have updated since then because I see ESP8266 OTA in the docs now.

[Finn-Land]

One last question about the BSdiff, I see on your project website that you were using BSdiff but decided not to and switching to Detools. But Detools also uses BSdiff as algorithm right?

[thinkty]

But Detools also uses BSdiff as algorithm right?

Yes, but iirc, it's a modified version of bsdiff since the original bsdiff uses dynamic memory allocation. Using detools with heatshrink for compression allowed achieving the same result with static memory allocation.

[Finn-Land]

Okay very clear, thank you for the answer! I appreciate you taking the time to reply. I will now close this issue, have a good day!

[Finn-Land]

I have one more question, I'm currently working on the implementation using the Arduino framework. Since it's built on the Espressif-IDF I thought it could be worth to try. I am able to successfully apply the patch file using the same detools function you're using. However during the esp_ota_end() and esp_set_boot_partition() functions the checksum of the new partition fails, did you also encounter this issue? If so, how did you resolve it?

[thinkty]

However during the esp_ota_end() and esp_set_boot_partition() functions the checksum of the new partition fails

Based on what I'm seeing from the ESP8266_RTOS_SDK source code, is esp_ota_end() returning with ESP_ERR_OTA_VALIDATE_FAILED? If so, is the error coming from esp_image_load() or esp_ota_verify_binary()?

If it's coming from esp_image_load(), you might want to look into that function and compare your logs since there seems to be a lot of ways for things to go bad.

If it's coming from esp_ota_verify_binary(), it might be a problem with the partitions (try increasing the size to fit the new image?) assuming that the new image was received and decompressed without any errors.

did you also encounter this issue?

I do not recall encountering this issue. In fact, I do not recall much from this project but pain...

[Finn-Land]

How can you tell if the error is coming from esp_image_load() or esp_ota_verify_binary()? Did you achieve this with actual debugging or enabling some verbose outputs? Because I'm currently going through the process of attempting to enable debugging using PlatformIO but it's not looking pretty. Could also be because I'm using the ESP32-C3

[Finn-Land]

It is returning with ESP_ERR_OTA_VALIDATE_FAILED btw, but it unfortunately returns that in more scenarios yes

[thinkty]

How can you tell if the error is coming from esp_image_load() or esp_ota_verify_binary()? Did you achieve this with actual debugging or enabling some verbose outputs?

I don't have an ESP on my hand at the moment, so I just looked at the source code for ESP8266_RTOS_SDK. When working on this project, I mostly relied on the serial logs (ex.ESP_LOGI) coming from the ESP8266. Perhaps check the serial outputs?

Because I'm currently going through the process of attempting to enable debugging using PlatformIO but it's not looking pretty. Could also be because I'm using the ESP32-C3

I would first try to setup a proper debugger if possible. https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/jtag-debugging/using-debugger.html#jtag-debugging-using-debugger-command-line seems like a good place to start. I remember using GDB in more serious bugs to set breakpoints and executing step by step to see exactly what is going on.

[Finn-Land]

I'm currently trying to just use your example to see if I can actually get it to work before saving endless amount of hours debugging. I'm using the ESP32-C3 to test your code, so far I've got everything to built. One question about the bridge project, I'm trying to use it in WSL since I'm using Windows. I can get it to run but how does one open a terminal with the "bridge-tester" to publish messages? I did manage to get the terminal running

[thinkty]

how does one open a terminal with the "bridge-tester" to publish messages?

I'm not sure if I understood your question. If you're referring to the demo video, I used a terminal multiplexer called tmux to open several terminals in the same window. So, bridge and bridge-tester are actually two separate applications. I just wanted to show them in the same window.
I don't have much experience with WSL so I'm not sure if opening another WSL window will work.

To send using the bridge-tester, you need to download the source, build it, and run the following command :

./publisher <ip> <port> <topic> <file>

Publisher application requires the ip and port of the Bridge server and also the topic of interest to publish to. The last argument file is the name of the file that contains the message to publish to the Bridge server. On a successful operation, the program will output the total number of bytes sent.

I'm using the ESP32-C3 to test your code, so far I've got everything to built.

Hmm I'm not sure if the code based on ESP8266_RTOS_SDK (made for ESP8266) will also work on the ESP32 platform.

https://www.esp32.com/viewtopic.php?t=11315

At the moment, the ESP8266 RTOS SDK has been refactored to be similar to ESP-IDF (similar configuration and build system, some component APIs are the same, etc). However it's not the same as ESP-IDF.

[Finn-Land]

Thank you very clear, I have setup bridge and tester both via Docker now, I am figuring out the port forwarding at the moment. However I have one last question, the firmware files for the ESP, do you extract them from the ESP using the esptool on the partitions to retrieve their data?

For my implementation I didn't include the ESP8266_RTOS_SDK, since well I have the C3. But your code still manages to built and run, right now I'm just trying to get that delta file over. If that works then I think your implementation also works on different ESP's.

[thinkty]

Sorry for the late reply.

the firmware files for the ESP, do you extract them from the ESP using the esptool on the partitions to retrieve their data?

Nope. I just used the binary (= build output from the compiler) that's on my PC since it's the one being flashed to the ESP, thus they will be identical. Then, I would make a copy somewhere else before I create a new image. Using detools, I generated the delta file containing the difference between the old image and the new one.

detools create_patch --compression heatshrink <old-image> <new-image> <patch-name>