OTA update from github url

[mmiscool]

Hello,

Git hub seems to do every thing over https and i was hoping to use the ota functionality to download from there. Is there a way to make ota work over https or a methos to get a regular http download from git hub?

        t_httpUpdate_return  ret = ESPhttpUpdate.update("github.com", 80, "/esp8266/Basic/raw/master/Flasher/Build/4M/ESP8266Basic.cpp.bin");

        switch (ret) {
          case HTTP_UPDATE_FAILD:
            Serial.println("HTTP_UPDATE_FAILD");
            break;

          case HTTP_UPDATE_NO_UPDATES:
            Serial.println("HTTP_UPDATE_NO_UPDATES");
            break;

          case HTTP_UPDATE_OK:
            Serial.println("HTTP_UPDATE_OK");
            break;
        }

[sticilface]

I think the short answer is no.
I was trying to download an 18Kb file from github, and it failed, @igrr debugged it failed as the TLS fragment size is too large. I reckon OTA is a no go. Something i wanted to try to do too.

[Links2004]

github.io shut work it allow http requests.

more info here:
https://pages.github.com/

[igrr]

@mmiscool yup, pretty much what @sticilface said.
My suggestion is to either:

• set up TLS proxy (i.e. Apache Traffic Server) and configure it for smaller fragment size, or
• host a file with SHA sum + link to you OTA binary on github, and then download OTA binary over plain HTTP. Verify SHA sum as you are downloading.

Sorry but i don't think we can do anything about this issue given the RAM available to us on the ESP8266. Also unfortunately most TLS servers don't support TLS fragment size negotiation (that's an extension to TLS protocol), so we can't make them switch to smaller fragment sizes.

[igrr]

Okay, spoke too soon.
I've improved TLS library so now it can handle full 16k fragments.
Here's an example which does OTA over HTTPS from github:
https://gist.github.com/igrr/24dd2138e9c8a7daa1b4

[mmiscool]

Just tested this.

Turned the code in to function. Minimal modifications to remove the wifi connecting stuff as that is already handled in the sketch.

Added this function.

'''
const char* host = "raw.githubusercontent.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char* fingerprint = "B0 74 BB EF 10 C2 DD 70 89 C8 EA 58 A2 F9 E1 41 00 D3 38 82";

const char* url = "/esp8266/Basic/master/Flasher/Build/4M/ESP8266Basic.cpp.bin";

bool BasicOTAupgrade()
{
configTime(3 * 3600, 0, "pool.ntp.org");

// Use WiFiClientSecure class to create TLS connection
WiFiClientSecure client;
Serial.print("connecting to ");
Serial.println(host);
if (!client.connect(host, httpsPort)) {
Serial.println("connection failed");
return 0;
}

if (client.verify(fingerprint, host)) {
Serial.println("certificate matches");
} else {
Serial.println("certificate doesn't match");
return 0;
}

Serial.print("Starting OTA from: ");
Serial.println(url);

auto ret = ESPhttpUpdate.update(client, host, url);
// if successful, ESP will restart
Serial.println("update failed");
Serial.println((int) ret);
}
'''

I get the following output on the serial terminal.

connecting to raw.githubusercontent.com

Soft WDT reset

ctx: cont
sp: 3fff5c80 end: 3fff61b0 offset: 01b0

stack>>>
3fff5e30: 3fff68c0 00000003 00002000 40211c94
3fff5e40: 4000050c 3fffc278 40105038 40234594
3fff5e50: 00000022 00011757 000010ac 00000003
3fff5e60: 00000080 00000040 3fffbd60 40234689
3fff5e70: 40235245 3fffb178 00000003 3ffe9138
3fff5e80: 00000261 00000000 4022eaab 3fff5f70
3fff5e90: ffffffff 3fffc6fc 0000005c 00000000
3fff5ea0: 00000000 00000000 3fffb6e8 3fffb698
3fff5eb0: 3fffb648 3fffb2f0 3fffb5b8 3fffb528
3fff5ec0: 000001df 3fffbd60 3fffbd60 000000ff
3fff5ed0: 3fffb180 00000006 00000040 00000040
3fff5ee0: 00000080 00000000 00000000 3fffbdc8
3fff5ef0: 3fffbd50 00000006 3fffb140 3fffb178
3fff5f00: 00000003 00000261 3fffa9c0 4022ed44
3fff5f10: 3fffb230 3fffb234 00000006 40234eda
3fff5f20: 3fffb2e0 00000261 00000000 40235205
3fff5f30: 3ffe9138 3fffb234 3fffb230 40221bae
3fff5f40: 3fffb168 40107116 3fffaee0 00020000
3fff5f50: 3ffec324 3fffb178 00000000 40235245
3fff5f60: 00000000 00000000 3fffb178 4022eab0
3fff5f70: 3fffaed0 402149be 3fff6010 3fff4f04
3fff5f80: 3fff4f08 3fff6010 3fffaee0 40211aa2
3fff5f90: 8527eb17 3fff3220 40214a9c 3fff3220
3fff5fa0: 3ffeb3bc 000001bb 3fff6010 40211d52
3fff5fb0: 3ffebf80 8527eb17 3ffeb8a0 3fff3220
3fff5fc0: 3ffeb3bc 000001bb 3fff6010 402110ad
3fff5fd0: 3ffebf80 8527eb17 3ffebf80 8527eb17
3fff5fe0: 3ffeb888 3fff6040 3fff6204 4021b87c
3fff5ff0: 3fff604c 3ffe8344 3fff6204 40211d68
3fff6000: 3ffeb3bc 3ffe8344 3fff6204 4020d99e
3fff6010: 3ffeb990 00000000 000003e8 0000000f
3fff6020: 3fff6160 3fffb140 00000000 4021c16c
3fff6030: 3ffeb3bc 00000001 3fff44f0 4020dc01
3fff6040: 3fffb140 0000000f 00000006 3fffb140
3fff6050: 0000000f 0000000c 3fffb120 0000000f
3fff6060: 00000000 3fffb100 0000000f 00000000
3fff6070: 3fffafd0 0000000f 00000000 3fffaec0
3fff6080: 0000000f 00000003 3fffa6b8 0000000f
3fff6090: 00000008 3fffa698 0000000f 00000005
3fff60a0: 3fffb800 0000054f 00000545 4021c1ba
3fff60b0: 3fff77a8 00000009 3fff6100 3fff452c
3fff60c0: 3fff4cf0 3fffa9d8 3fffa9d8 4020df4c
3fff60d0: 3fff4cf0 3fffa9d8 3fff6100 402126d2
3fff60e0: 0000000f 3fffc6fc 3fffa9d8 40212702
3fff60f0: 3fffaec0 0000000f 3fff44f0 402128a8
3fff6100: 3fff7868 0000000f 00000009 0000005f
3fff6110: 00000009 3fff7868 0000000f 00000003
3fff6120: 3fffb230 0000007f 00000000 3fff5038
3fff6130: 402126dc 00000001 40214a9c 0000000f
3fff6140: 40201646 00000001 00000001 3fff5180
3fff6150: 3fffdc20 000003e6 3fff44f0 402129bc
3fff6160: 3ffeb950 00000000 000003e8 0000bd26
3fff6170: 3fff4510 3fffaf28 00000016 40101f01
3fff6180: 3fffdc20 00000000 3fff5178 402107cb
3fff6190: 3fffdc20 00000000 3fff5178 40214ae5
3fff61a0: 00000000 00000000 3fff5190 40100114
<<<stack<<<

[igrr]

Looks like you are running out of RAM. Check with nothing except this function in your sketch (i.e. no SPIFFS, no web server).

[Links2004]

check out the new API of ESPhttpUpdate

now it shut work with one line like this:
ESPhttpUpdate.update("https://github.com/esp8266/Basic/raw/master/Flasher/Build/4M/ESP8266Basic.cpp.bin");

https://github.com/Links2004/Arduino/blob/httpClient/libraries/ESP8266httpUpdate/examples/httpUpdate/httpUpdate.ino