Update SPIFFS

[meditant]

Hello,

You work like very elegant ! I juste have a question.
It is possible to update SPIFFS ?
I need to update FW and SPIFFS.

Best,

[chrisjoyce911]

This function is not party off the aim of this library , Maybe look at UDHttp ?

name=UDHttp
version=1.0.0
author=iotsharing.com
maintainer=https://github.com/nhatuan84
sentence=ESP32 upload and download file
paragraph=ESP32 upload and download file
category=Communication
url=iotsharing.com
architectures=esp32

`#include "UDHttp.h"

int wdataf(uint8_t *buffer, int len){
//write downloaded data to file
return root.write(buffer, len);
}

void progressf(int percent){
Serial.printf("%d\n", percent);
}

Serial.print("Initializing SD card...");
if (!SD.begin(32, 14, 12, 27)) {
Serial.println("initialization failed!");
return;
}
Serial.println("initialization done.");
SD.remove("test.pdf");
{
UDHttp udh;
//open file on sdcard to write
root = SD.open("test.pdf", FILE_WRITE);
if (!root) {
Serial.println("can not open file!");
return;
}
//download the file from url
udh.download("http://www.smart-words.org/linking-words/linking-words.pdf", wdataf, progressf);
root.close();
Serial.printf("done downloading\n");
}`

[tobozo]

SPIFFS could eventually be brought into the scope of this library with little effort: the second argument of Update.begin() is the partition type.

So it would only be a matter of adding a json entry (e.g. "{ spiffs" : "/path/to/spiffs.bin }") and a bit of logic to the Update process to implement that.

[scubachristopher]

Why not join forces?

I recommend esp32FOTA integrates what @tobozo has created: taking a gzipped binary and decompressing the stream directly to the OTA partition. Minimizes payload, no SPIFFS / persistence of the firmware before update.

esp32FOTA could handle the wrap-around http/https, optional checking if fw version is > than what's currently there (if the developer wants to put a version json file -- or not).

I wanted to protect my gzipped binary in transit, so I have an https server with a complex basic auth pw. I DO store my cert in SPIFFS but it could be baked into flash. I do NOT have a json file specifying version, but that could be trivially added. esp32FOTA could take the cert and optional creds as a params to a secure function for users:

#define CLOUD_DOMAIN            "MYDOMAIN.com"
#define OTA_URL                         "https://" CLOUD_DOMAIN

void ProcessFirmware(char *firmwareFilename)
{
    HTTPClient http;
    WiFiClientSecure clientForOta;
    int httpCode;
    char buff[(2 * MAX_FIRMWARE_FILENAME) + 1];

    tcpip_adapter_init();
    esp_task_wdt_reset();

    http_cert = lriLoadSPIFFSIntoMem("/httpcert.pem");
    clientForOta.setCACert((const char *) http_cert);
    clientForOta.setTimeout(30);
    strcpy(buff, OTA_URL);
    strcat(buff, "/");
    strcat(buff, firmwareFilename);
    strcat(buff, ".gz");

    Serial.printf("Attempting to connect to %s\n", buff);
    vTaskDelay(pdMS_TO_TICKS(1000));

    if (!http.begin(clientForOta, buff)) {
      Serial.printf("Http.begin failed\n");
    }
    else {
      Serial.printf("HTTP.begin successful for %s\n", buff);
      String auth = base64::encode(HTTP_BASIC_AUTH_USERNAME ":" HTTP_BASIC_AUTH_PASSWORD);
      http.addHeader("Authorization", "Basic " + auth);
      httpCode = http.GET();
      if (httpCode > 0) {
        Serial.printf("HTTP.get successful for %s, size of file reported as %d\n", buff, http.getSize());
        //
        // Read from the stream and decompress to the free OTA partition, then reset
        //
        if (!gzStreamUpdater(http.getStreamPtr(), UPDATE_SIZE_UNKNOWN) ) {
          Serial.printf("gzHTTPUpdater failed with return code #%d\n", tarGzGetError());
        }
      }
      else {
        Serial.printf("HTTP.get UNSUCCESSFUL for %s, http returned %d\n", buff, httpCode);
      }
    }
    //
    // If we got here, we failed to update. 
    Serial.printf("\nFirmware update failed.  Rebooting...\n");
    delay(3000);
    ESP.restart();
}

You could also provide examples of how to prop a simple http server and gzip the bin. It's pretty trivial but daunting for many.

On ubuntu, for example, I have a gist that automates this:

// Setup
DEBIAN_FRONTEND=noninteractive sudo apt-get -y update 2>/dev/null 1>/dev/null
DEBIAN_FRONTEND=noninteractive sudo apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef --allow-downgrades --allow-remove-essential --allow-change-held-packages upgrade 2>/dev/null 1>/dev/null
sudo su - pi -c "curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - 2>/dev/null 1>/dev/null"
sudo su - pi -c "sudo apt-get install -y nodejs build-essential 2>/dev/null 1>/dev/null"
sudo npm install http-server -g 2>/dev/null 1>/dev/null

// Run http server
sudo /usr/bin/http-server "/home/ubuntu/firmware/" -p 443 -g --log-ip --no-dotfiles -S -C "/home/ubuntu/http/httpcert.pem" -K "/home/ubuntu/http/httpkey.pem" -d false --username [something complex] --password [something complex]

// To compress a bin, put it in /home/ubuntu/firmware
gzip --keep --best --force blah.bin

This entire effort could be a great solution: trivial calls to esp32FOTA for secure and insecure, compressed and not compressed, version checking or not, example of how to provision a server to serve gzipped binaries, securely.

Both project are great work. Perhaps esp32FOTA leveraging esp32-targz is a marriage made in heaven?

[tobozo]

@scubachristopher sure let's join forces :-)

here are some implementation ideas based on your input:

• decouple the HTTP client logic from execOTA and execHTTPcheck so it can be optionnaly overloaded with HTTPS client logic
• get the existing bin-based Update logic out of execOTA so it can be optionnaly overloaded with gz-based logic.
• create esp32FOTA::execOTA( Stream *StreamIn, []( Stream* StreamOut) ), where *StreamIn is an already established HTTP(S) connexion with the incoming file, and the lambda function is either the existing bin-based update logic, or the gz-based.
• Figure out how to handle both HTTP and HTTPS in this library without creating messy/redundant code
• following HTTP(S) 301/302 redirects

[chrisjoyce911]

I think its a plan , at the moment I don't have the time to assist much

[tobozo]

Just implemented the code so it compiles without errors and warnings, I'll keep this on my fork until this is properly tested but it should already be usable:

https://github.com/tobozo/esp32FOTA

Usage:

{
    "type": "esp32-fota-http",
    "version": 2,
    "host": "192.168.0.100",
    "port": 80,
    "bin": "/fota/esp32-fota-http-2.ino.bin",
    "spiffs": "/fota/esp32-fota-http-2.spiffs.bin"
}

I also had those random ideas (for the far-far-away future) while working on the code:

• There's room for merging HTTP with HTTPS logic and improve maintainability: a bool useSecureConnexion could be used as a dispatcher. By doing so the class methods from esp32FOTA can become virtual and be extended by secureEsp32FOTA instead of being dissociated.
• Migrating connectivity+stream creation into plugins could enable seamless Update.writeStream support for HTTP, HTTPS, UDP, SD, SD_MMC, Ethernet, LoRa, CAN.

[tobozo]

see #47

[ludvigaldrin]

Hey @tobozo ! I tried your version today since I have both SPIFFS and BIN that I want to be able to update. Im not 100% sure about your version. If I put in both a bin and spiffs path. It will only update SPIFFS. Removing SPIFFS path will update BIN.

The major issue with this is that updating SPIFFS do not increment the bin version so it will send the update to a loop downloading the SPIFFS over and over again.

Am I missing something or how is this supposed to work? I guess I need a SPIFFS version and a BIN version somewhere so it know which one to update? Any good idea?

[tobozo]

hey @ludvigaldrin

whoops totally forgot I had a fork in progress :)

as far as I remember, this mod used .tar.gz files instead of .gz, so a single archive would deploy both SPIFFS and the firmware.

I should probably restart from scratch using a fresh fork, will see what I can do in September

[tobozo]

ping @scubachristopher last commit on the ongoing PR #92 implements the injection of custom http headers during the queries as per your earlier suggestion.

Should this PR be conclusive, the next milestone will consist in getting esp32fota to work with gz/targz files, which will bring me to these tasks:

• Decouple the HTTP client logic from execOTA and execHTTPcheck
• Use Stream* to seamlessly overload with HTTP, HTTPS, File or gzStream source
• Explore and test the new logical cases this unlocks e.g. firmare + spiffs joined in tar.gz or separately gzipped, plus signature, plus certificate, plus extra headers

[tobozo]

Implemented in 0.2.0.

Keeping the issue open as gz/targz support is still being worked on (0.2.1 version coming sooon ^^)

[tobozo]

bump

gzip and zlib support are implemented in 0.2.3, and there won't be any .tar.gz support as it does not make sense with the security design of this library.