New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpClient: HTTPS request without specifying certificate fingerprint? #3417

Closed
tuxedo0801 opened this Issue Jul 12, 2017 · 21 comments

Comments

Projects
None yet
@tuxedo0801
Contributor

tuxedo0801 commented Jul 12, 2017

Hi there,

I have to push some data from ESP8266 to an IoT Cloud system. But this requires HTTPS.

In the examples I found, I always need to specify the fingerprint of the https-certficate.

This binds my source directly to the cloud system. If certificate changes, the request fails I guess...
So I have to update my sketch with every certificate-change, which is not that easy for my use-case.

Is there a possibility to accept whatever certificate is running behind https?

For my use-case it would be okay to use http instead of https, but the cloud system does only provide https. So I have to stick to it.

br,
Alex

@SnakeDoc

This comment has been minimized.

SnakeDoc commented Jul 12, 2017

Not the best solution, but could you proxy the requests through a service/server you control? I suppose you'd have the same issue once your service/server's certificate expires though...

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Jul 13, 2017

This issue relates to #1851 ...

Maybe I add a php-script inbetween to decouple https-request from esp8266, but this is more a workaround than a real solution.

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Jul 13, 2017

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Jul 14, 2017

Currently I'm testing the solution provided in #1851, but I would like to see, that I can make https-calls, without specifying a certificate-finger-print and without providing the root-ca-certificate to do certificate validation. Just "accept all certificates and ignore the complete certificate-check", like you can do on any other desktop-programming-language ...

Is this in general possible? Or is this technically on the esp8266 not possible?

@JamesGKent

This comment has been minimized.

JamesGKent commented Jul 25, 2017

you might be able to do it if you set the methods that check the cert to return as if they matched at all times, from a brief look you need to:

  • replace the return with a return true at line 485 in wificlientsecure.cpp
  • replace return with return true at line 518 in wificlientsecure.cpp
  • replace return with return true at line 546 in wificlientsecure.cpp

I would suggest that to do what you want you'll want to move the returns closer to the start of each function, but after checking that you have an ssl context, so i'd suggest just adding return true before each of these lines:

  • 465
  • 494
  • 524
  • 554 (needed because call to verify cert chain will fail)

can always give this a try.

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Aug 15, 2017

Sounds like copy&paste&modify of Wificlientsecure.cpp/.h to create a quick'n'easy workaround...

Did someone test this already?

@MikaelSchultz

This comment has been minimized.

MikaelSchultz commented Aug 30, 2017

I just had the same problem. Just ignore the result of the client.verify() method, or just dont run it at all. It's up to you what to do if the certificate verification fails.
Works for me. No need to change the source code of WiFiClientSecure.cpp

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Nov 24, 2017

Finally, I have some feedback:

When I use this sample as a starting point: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

and just ignore lines 52-56:

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

then https communication is working in principle. My Code:

void sendMessageHTTPS(String msg) {

  
  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to : '");
  Serial.print(host);
  Serial.println("'");
  
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

//  if (client.verify(fingerprint, host)) {
//    Serial.println("certificate matches");
//  } else {
//    Serial.println("certificate doesn't match");
//  }

  Serial.print("requesting URL: '");
  Serial.print(url);
  Serial.println("'");

  client.print(String("POST ") + url + " HTTP/1.1\r\n" +
          "Host: " + host + "\r\n" +
          "Connection: close\r\n" +
          "Accept: */*\r\n" +
          "User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n" +
          "Content-Type: application/json;charset=utf-8\r\n" +
          "Authorization: Bearer "+cloudDeviceToken+"\r\n" + 
          "Content-Length: "+msg.length()+"\r\n" +
          "\r\n" + 
          msg + "\r\n");

  Serial.println("request sent");
  unsigned long timeout = millis();
  while (client.available() == 0) {
    if (millis() - timeout > 5000) {
      Serial.println(">>> Client Timeout !");
      client.stop();
      return;
    }
  }
  
  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
    String line = client.readStringUntil('\r');
    Serial.print(line);
  }
}

but I face some ESP crashes after a few https requests. Still need to investigate what goes wrong.

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Nov 24, 2017

decoded stacktrace:

Exception 29: StoreProhibited: A store referenced a page mapped with an attribute that does not permit stores
Decoding 29 results
0x402167d2: memset at /Users/igrokhotkov/e/newlib-xtensa/xtensa-lx106-elf/newlib/libc/machine/xtensa/../../../../.././newlib/libc/machine/xtensa/memset.S line 133
0x402231d6: more_comps at crypto/bigint.c line 672
0x40223368: alloc at crypto/bigint.c line 672
0x402235e9: regular_multiply at crypto/bigint.c line 672
0x4020eaa8: esp_yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 87
0x40223368: alloc at crypto/bigint.c line 672
0x4022409f: bi_barrett at crypto/bigint.c line 1289
0x402242f4: bi_mod_power at crypto/bigint.c line 1414
0x40223238: trim at crypto/bigint.c line 672
0x40225425: RSA_public at crypto/rsa.c line 252
:  (inlined by) RSA_encrypt at crypto/rsa.c line 286
0x40220e9c: send_client_key_xchg at ssl/tls1_clnt.c line 409
0x40221331: do_clnt_handshake at ssl/tls1_clnt.c line 123
0x40208200: ClientContext::_consume(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) ClientContext::read(char*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src\include/ClientContext.h line 253
0x4020eb20: __yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 96
0x40220ca4: do_handshake at ssl/tls1.c line 2005
:  (inlined by) basic_read at ssl/tls1.c line 1481
0x40221080: do_client_connect at ssl/tls1_clnt.c line 168
0x401004d8: malloc at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266\umm_malloc/umm_malloc.c line 1664
0x40220e0c: ssl_read at ssl/tls1.c line 2005
0x4020868d: SSLContext::connect(ClientContext*, char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) WiFiClientSecure::_connectSSL(char const*) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 321
0x40207cf2: WiFiClient::connect(IPAddress, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClient.cpp line 324
0x402088ad: WiFiClientSecure::connect(char const*, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 313
0x402038c4: sendMessageHTTPS(String) at C:\dev\git-repos\TestFW\Test/Test.ino line 489
0x4020dec3: String::changeBuffer(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df0f: String::reserve(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df41: String::copy(char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020e048: String::operator=(String const&) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x40203c74: loop at C:\dev\git-repos\TestFW\Test/Test.ino line 584
0x4020eaf4: loop_wrapper at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 124
0x4010070c: cont_norm at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/cont.S line 109

Hmm ...

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Nov 24, 2017

@igrr
Any ideas what's going wrong?

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Nov 24, 2017

Initially, the request was sent every 5 sec. It fails every 5-8 times, then I get stacktrace, device restarts.

I changed interval from 5sec to 60sec. This did not change anything. Crashed after 5 requests.

Beside the https-request, the device is serving a small webpage (which has not been accessed during my tests), queries NTP time and reads power-measurement data from HLW8012 chip via interrupt.
Everything works unless I activate HTTPS.
HTTP worked quite well, was extremly stable (no crash so far). And according to the stacktrace, it has something to do with https/rsa/crypto ...

Would be great if someone can shed some light on this issue.

@tuxedo0801

This comment has been minimized.

Contributor

tuxedo0801 commented Nov 24, 2017

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

@cimba007

This comment has been minimized.

cimba007 commented Dec 11, 2017

Any chance to get this working with HTTPClient? My responses could become lager and I don't want to deal with chunked responses manually...

@odelot

This comment has been minimized.

odelot commented Feb 9, 2018

a noob question: this fingerprint changes over time?

I know you are trying not to use them, but I have a valid one and I was thinking to hardcode it on my device.

@romkey

This comment has been minimized.

romkey commented Mar 20, 2018

@odelot That's a good question.

Yes, fingerprints do change over time. They'll change any time the web server's SSL certificate changes.

So if a web server's SSL certificate expires on March 21, 2018 then it'll need a new certificate then and the new certificate will have a new fingerprint.

The server's certificate might easily change before the expiration date, too.

It's even possible for a web server which is load balanced by multiple servers to use different certificates on different servers, so you might see different fingerprints for the same HTTPS endpoint. That's not a likely scenario - it's probably the result of an odd or mis-configuration - but it's possible.

Using the fingerprint to verify the server isn't ideal, but the alternative is maintaining a list of certificates for the root certificate authorities - this is what more capable computers do. Unfortunately these need to be updated too, and their size well exceeds the storage available on an ESP8266.

Unfortunately I don't have a good alternative to hardcoding the fingerprint on your device.

@MikeS159

This comment has been minimized.

MikeS159 commented Mar 21, 2018

@odelot @romkey
There is an example here which lets you use the root certificates fingerprint. Root certs have much longer life spans, since they are hard coded into browser and operating systems, making them hard to update. The root certificate for GitHub (used in the example) is valid until 10 November 2031, 00:00:00 GMT.

To get a root certificate you can view it in a browser (click the green padlock), view certificate, select the root certificate and export it as a .der file. I wrote a little python script to format .der files as a hex string you can past straight into the CACert.ino file.

image

@davidverweij

This comment has been minimized.

davidverweij commented Jun 3, 2018

@tuxedo0801 , I have a nearly identical error (stacktrace) - what do you mean by 2.4.0-rc2? My NodeMCU v1.0 crashes at the first attempt.

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

@davidverweij

This comment has been minimized.

davidverweij commented Jun 3, 2018

Apologies - after fiddeling with the variables I found I needed to add the 'https://' to the host adress in order for it to work. I was attempting a POST request a custom Google Cloud functions https endpoint. Somehow I could not find documentation on a Firebase/store request but not Cloud Functions. (e.g. .cloudfunctions.net). I did had a near similar error - so perhaps that might help? Cheers!

@devyte

This comment has been minimized.

Collaborator

devyte commented Jul 13, 2018

Closing due to bearssl merge (axtls will be deprecated).

@devyte devyte closed this Jul 13, 2018

@C0rn3j

This comment has been minimized.

C0rn3j commented Sep 20, 2018

https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WiFi/examples

I can only find an example HTTPSRequest which still uses the fingerprint, am I looking in the wrong place?

@d-a-v

This comment has been minimized.

Collaborator

d-a-v commented Sep 20, 2018

@C0rn3j check BearSSL_Validation.ino in this examples repository.
also check this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment