OTA updates support

[callmephilip]

I am trying to figure out how to add OTA updates support to demo using this as the reference. Here are the steps:

Updated partitions.csv with 2 ota slots:

# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x4000,
otadata,  data, ota,     0xd000,  0x2000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000,  1M,
ota_0,    app,  ota_0,   0x110000, 1M,
ota_1,    app,  ota_1,   0x210000, 1M,

Updated sdkconfig.defaults to include as seen here

CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
CONFIG_PARTITION_TABLE_TWO_OTA=y

I intend to use this API once I have firmware update downloaded, however I am confused as to what exactly to download. Output found in target/xtensa-esp32-espidf/debug/rust-esp32-std-demo is a 34.6MB binary. This seems to exceed the size of the flash on the chip, as far as I can tell. Does this output need to processed further before being downloaded used for the update?

[brianmay]

Normally espflash does this.

From the README:

• You can also flash with the esptool.py utility which is part of the Espressif toolset
• Use the instructions below only if you have flashed successfully with espflash at least once, or else you might not have a valid bootloader and partition table!
• The instructions below only (re)flash the application image, as the (one and only) factory image starting from 0x10000 in the partition table!
• Install esptool using Python: pip install esptool
• (After each cargo build) Convert the elf image to binary: esptool.py --chip [esp32|esp32s2|esp32c3] elf2image target/xtensa-esp32-espidf/debug/rust-esp32-std-demo
• (After each cargo build) Flash the resulting binary: esptool.py --chip [esp32|esp32s2|esp32c3] -p /dev/ttyUSB0 -b 460800 --before=default_reset --after=hard_reset write_flash --flash_mode dio --flash_freq 40m --flash_size 4MB 0x10000 target/xtensa-esp32-espidf/debug/rust-esp32-std-demo.bin

I think this 2nd last step "Convert the elf image to binary" might be what you are looking for.

[ivmarkov]

Indeed. Note however that espflash now also has an option to convert the elf image to bin and then output the bin.

[svenstaro]

I think it'd be great if a proper OTA example could be integrated into this repo in some way.

[callmephilip]

Ok, so i think i got one step further on this. To get a proper binary file for OTA, I did the following:

espflash ./target/xtensa-esp32-espidf/debug/rust-esp32-std-demo save-image esp32 ./target/xtensa-esp32-espidf/debug/rust-esp32-std-demo firmware.bin

I then used otatool.py that ships with esp-idf to upload update to a device to test it out

./otatool.py write_ota_partition --name ota_0 --input firmware.bin

This seems to work. Gonna try updating the app using API now

[callmephilip]

Here's a basic working firmware update. Update occurs when accessing /api/ota of the demo server

at("/api/ota").get(move |_| {
  use embedded_svc::http::{self, client::*, status, Headers, Status};
  use embedded_svc::io::Bytes;
  use esp_idf_svc::http::client::*;
  use embedded_svc::ota::{Ota, OtaSlot, OtaUpdate};
  use esp_idf_svc::ota::{EspOta};
  use embedded_svc::io::{Write};

  let mut ota = EspOta::new().unwrap();
  let mut client = EspHttpClient::new_default()?;
  let response = client.get(&String::from("URL_OF_THE_FIRMWARE_BIN"))?.submit()?;
  let mut ota_update = ota.initiate_update().unwrap();
  let mut firmware_update_ok = true;

  loop {
      // download firmware in batched of 10K
      let bytes_to_take = 10 * 1024;
      let body: Result<Vec<u8>, _> = Bytes::<_, 64>::new(response.reader()).take(bytes_to_take).collect();
      let body = body?;

      match ota_update.do_write(&body) {
          Ok(buff_len) => info!("wrote update: {:?}", buff_len),
          Err(err) => {
              info!("failed to write update with: {:?}", err);
              firmware_update_ok = false;
              break;
          }
      }

      if body.len() < bytes_to_take {
          break;
      }
  }

  if firmware_update_ok {
      ota_update.complete().unwrap();
  } else {
      ota_update.abort().unwrap();
  }

  Ok(embedded_svc::httpd::Response::from("Ok"))
})?

This seems to work, as far as I can tell. However, I cannot find a way to restart firmware from inside the app. There is this in the original API ref. Is it a recommended way of restarting the app?

[ivmarkov]

This seems to work, as far as I can tell.

Heh. So nice to see that my never-ever-tested-or-ran-even-once code actually works! The power of Rust, I guess! Argue with the compiler inside VSCode for a day and then the thing just works.

However, I cannot find a way to restart firmware from inside the app. There is this in the original API ref. Is it a recommended way of restarting the app?

Should be OK I guess. Though I would cleanly shutdown the whole app (as in shutting down the http server, wifi etc. etc.) and once all of these are dropped, I would call esp_restart. Though not strictly necessary, I guess.

[ivmarkov]

One more important detail: Somewhere at the beginning of your main() function, you have to call one of these two:

• EspOta::mark_running_slot_valid()
  or
• EspOta::mark_running_slot_invalid_and_reboot()

basically once you've rebooted, you would be running the new firmware. At some point, you have to tell the system where this firmware is OK or not OK. If it is not OK, ESP-IDF will mark the partition as invalid and will reboot into the old firmware.

[callmephilip]

Had to wrestle around with partitions table to make sure things fit into a 4MB flash. The current setup is:

# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x4000,
otadata,  data, ota,     0xd000,  0x2000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000,  0x140000,
ota_0,    app,  ota_0,   0x150000, 0x140000,
ota_1,    app,  ota_1,   0x290000, 0x140000,

Firmware updates found here are about 1.08MB. Trying to load a release bundle using the following code

let mut client = EspHttpClient::new(&EspHttpClientConfiguration {
  crt_bundle_attach: Some(esp_idf_sys::esp_crt_bundle_attach),
  ..Default::default()
})?;
let response = client.get("https://github.com/bakery/rust-esp32-std-demo/releases/download/0.24.50/firmware-0.24.50.bin")?.submit()?;

I am now getting the following error which only occurs with this custom partitions table:

E (104150) HTTP_CLIENT: Out of buffer
W (104160) esp_idf_svc::httpd: Request handled with status 500 (Some("ESP_FAIL"))

It seems like I am maybe misunderstanding how these partitions are working

[ivmarkov]

I don't think this error has anything to do with the partitions. It is in http client - says it does not have buffer. Need to check the esp idf http client code exactly when this happens

[ivmarkov]

Ok so your url / query string is too long and you need to increase the http client TX buffer size. Unfortunately, I've only exposed conf for the rx buffer but forgot about the TX one: https://github.com/esp-rs/esp-idf-svc/blob/e96ebb78cd7f6f63a82c9091680d10a6ca758d56/src/http/client.rs#L103

I'll expose the TX one in the next days, or you can just file a PR yourself.

[ivmarkov]

It seems default TX and RX buf size is 512 bytes which is not enough for the first line HTTP request line: https://github.com/espressif/esp-idf/blob/a82e6e63d98bb051d4c59cb3d440c537ab9f74b0/components/esp_http_client/include/esp_http_client.h#L20

[callmephilip]

Nice catch. GH does redirect to this which is 530+ chars default buffer is 512.

[maximeborges]

Hey, I've been working on OTA update over UART for our application, but for some reasons after restarting the app seems to be stuck and I can't get any log or anything, and the watchdog just triggers after a while.

I've tried to reimplemented the exact C commands used in the IDF example with the same result, and I can confirm that the verification of the binary is OK once flashed.

I can switch back to the previous slot with otatool.py.

At the moment I'm doing the following:

cargo build +esp
cargo espflash save-image ota.bin
# Or `esptool.py --chip esp32 elf2image target/xtensa-esp32-espidf/debug/rust-ota-demo` and using the resulting `rust-ota-demo.bin`, same result

Then executing my app that is doing the ota.initiate_update(), ota_update.do_write() in a loop, and ota_update.complete() without any error.

I guess I'm missing a step or something to make the binary...

[callmephilip]

@maximeborges i don't know if this helps, but here is a little demo update i managed to get to work with @ivmarkov's help

as far as builds go, I have been using Sergio's container for builds. with the following command

cargo +esp espflash --partition-table ./partitions.csv save-image ota-firmware-VERSION-NUMBER-HERE.bin

We've also built a couple of GH actions to help integrate this into a CI workflow. There is an example of how to use it here

[maximeborges]

@callmephilip In the end I just had some crap in my sdkconfig... But thanks for the code, it definitly helped me catch up differences with my project!

[maximeborges]

For reference, here is my demo repo for flashing OTA partition via UART.

[wzhd]

Had to wrestle around with partitions table to make sure things fit into a 4MB flash.

Unless you need the factory partition for something, I find it useful to get rid of it and just have two ota partitions when working with a small flash. Esp will try to boot from an ota partition if a factory app can't be found.

[faern]

I just published esp-ota and then I found this issue right after. This issue has lots of good links and references. esp-idf-svc even has an OTA implementation?! 🤯. I wish I had found this issue earlier.