Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpClient: HTTPS request without specifying certificate fingerprint? #3417

Closed
tuxedo0801 opened this issue Jul 12, 2017 · 32 comments
Closed

HttpClient: HTTPS request without specifying certificate fingerprint? #3417

tuxedo0801 opened this issue Jul 12, 2017 · 32 comments

Comments

@tuxedo0801
Copy link
Contributor

@tuxedo0801 tuxedo0801 commented Jul 12, 2017

Hi there,

I have to push some data from ESP8266 to an IoT Cloud system. But this requires HTTPS.

In the examples I found, I always need to specify the fingerprint of the https-certficate.

This binds my source directly to the cloud system. If certificate changes, the request fails I guess...
So I have to update my sketch with every certificate-change, which is not that easy for my use-case.

Is there a possibility to accept whatever certificate is running behind https?

For my use-case it would be okay to use http instead of https, but the cloud system does only provide https. So I have to stick to it.

br,
Alex

@SnakeDoc
Copy link

@SnakeDoc SnakeDoc commented Jul 12, 2017

Not the best solution, but could you proxy the requests through a service/server you control? I suppose you'd have the same issue once your service/server's certificate expires though...

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Jul 13, 2017

This issue relates to #1851 ...

Maybe I add a php-script inbetween to decouple https-request from esp8266, but this is more a workaround than a real solution.

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Jul 13, 2017

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Jul 14, 2017

Currently I'm testing the solution provided in #1851, but I would like to see, that I can make https-calls, without specifying a certificate-finger-print and without providing the root-ca-certificate to do certificate validation. Just "accept all certificates and ignore the complete certificate-check", like you can do on any other desktop-programming-language ...

Is this in general possible? Or is this technically on the esp8266 not possible?

@JamesGKent
Copy link

@JamesGKent JamesGKent commented Jul 25, 2017

you might be able to do it if you set the methods that check the cert to return as if they matched at all times, from a brief look you need to:

  • replace the return with a return true at line 485 in wificlientsecure.cpp
  • replace return with return true at line 518 in wificlientsecure.cpp
  • replace return with return true at line 546 in wificlientsecure.cpp

I would suggest that to do what you want you'll want to move the returns closer to the start of each function, but after checking that you have an ssl context, so i'd suggest just adding return true before each of these lines:

  • 465
  • 494
  • 524
  • 554 (needed because call to verify cert chain will fail)

can always give this a try.

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Aug 15, 2017

Sounds like copy&paste&modify of Wificlientsecure.cpp/.h to create a quick'n'easy workaround...

Did someone test this already?

@ghost
Copy link

@ghost ghost commented Aug 30, 2017

I just had the same problem. Just ignore the result of the client.verify() method, or just dont run it at all. It's up to you what to do if the certificate verification fails.
Works for me. No need to change the source code of WiFiClientSecure.cpp

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Nov 24, 2017

Finally, I have some feedback:

When I use this sample as a starting point: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

and just ignore lines 52-56:

  if (client.verify(fingerprint, host)) {
    Serial.println("certificate matches");
  } else {
    Serial.println("certificate doesn't match");
  }

then https communication is working in principle. My Code:

void sendMessageHTTPS(String msg) {

  
  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to : '");
  Serial.print(host);
  Serial.println("'");
  
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

//  if (client.verify(fingerprint, host)) {
//    Serial.println("certificate matches");
//  } else {
//    Serial.println("certificate doesn't match");
//  }

  Serial.print("requesting URL: '");
  Serial.print(url);
  Serial.println("'");

  client.print(String("POST ") + url + " HTTP/1.1\r\n" +
          "Host: " + host + "\r\n" +
          "Connection: close\r\n" +
          "Accept: */*\r\n" +
          "User-Agent: Mozilla/4.0 (compatible; esp8266 Lua; Windows NT 5.1)\r\n" +
          "Content-Type: application/json;charset=utf-8\r\n" +
          "Authorization: Bearer "+cloudDeviceToken+"\r\n" + 
          "Content-Length: "+msg.length()+"\r\n" +
          "\r\n" + 
          msg + "\r\n");

  Serial.println("request sent");
  unsigned long timeout = millis();
  while (client.available() == 0) {
    if (millis() - timeout > 5000) {
      Serial.println(">>> Client Timeout !");
      client.stop();
      return;
    }
  }
  
  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
    String line = client.readStringUntil('\r');
    Serial.print(line);
  }
}

but I face some ESP crashes after a few https requests. Still need to investigate what goes wrong.

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Nov 24, 2017

decoded stacktrace:

Exception 29: StoreProhibited: A store referenced a page mapped with an attribute that does not permit stores
Decoding 29 results
0x402167d2: memset at /Users/igrokhotkov/e/newlib-xtensa/xtensa-lx106-elf/newlib/libc/machine/xtensa/../../../../.././newlib/libc/machine/xtensa/memset.S line 133
0x402231d6: more_comps at crypto/bigint.c line 672
0x40223368: alloc at crypto/bigint.c line 672
0x402235e9: regular_multiply at crypto/bigint.c line 672
0x4020eaa8: esp_yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 87
0x40223368: alloc at crypto/bigint.c line 672
0x4022409f: bi_barrett at crypto/bigint.c line 1289
0x402242f4: bi_mod_power at crypto/bigint.c line 1414
0x40223238: trim at crypto/bigint.c line 672
0x40225425: RSA_public at crypto/rsa.c line 252
:  (inlined by) RSA_encrypt at crypto/rsa.c line 286
0x40220e9c: send_client_key_xchg at ssl/tls1_clnt.c line 409
0x40221331: do_clnt_handshake at ssl/tls1_clnt.c line 123
0x40208200: ClientContext::_consume(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) ClientContext::read(char*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src\include/ClientContext.h line 253
0x4020eb20: __yield at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 96
0x40220ca4: do_handshake at ssl/tls1.c line 2005
:  (inlined by) basic_read at ssl/tls1.c line 1481
0x40221080: do_client_connect at ssl/tls1_clnt.c line 168
0x401004d8: malloc at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266\umm_malloc/umm_malloc.c line 1664
0x40220e0c: ssl_read at ssl/tls1.c line 2005
0x4020868d: SSLContext::connect(ClientContext*, char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 609
:  (inlined by) WiFiClientSecure::_connectSSL(char const*) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 321
0x40207cf2: WiFiClient::connect(IPAddress, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClient.cpp line 324
0x402088ad: WiFiClientSecure::connect(char const*, unsigned short) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\libraries\ESP8266WiFi\src/WiFiClientSecure.cpp line 313
0x402038c4: sendMessageHTTPS(String) at C:\dev\git-repos\TestFW\Test/Test.ino line 489
0x4020dec3: String::changeBuffer(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df0f: String::reserve(unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020df41: String::copy(char const*, unsigned int) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x4020e048: String::operator=(String const&) at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/WString.cpp line 519
0x40203c74: loop at C:\dev\git-repos\TestFW\Test/Test.ino line 584
0x4020eaf4: loop_wrapper at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/core_esp8266_main.cpp line 124
0x4010070c: cont_norm at C:\Users\MyUser\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.4.0-rc1\cores\esp8266/cont.S line 109

Hmm ...

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Nov 24, 2017

@igrr
Any ideas what's going wrong?

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Nov 24, 2017

Initially, the request was sent every 5 sec. It fails every 5-8 times, then I get stacktrace, device restarts.

I changed interval from 5sec to 60sec. This did not change anything. Crashed after 5 requests.

Beside the https-request, the device is serving a small webpage (which has not been accessed during my tests), queries NTP time and reads power-measurement data from HLW8012 chip via interrupt.
Everything works unless I activate HTTPS.
HTTP worked quite well, was extremly stable (no crash so far). And according to the stacktrace, it has something to do with https/rsa/crypto ...

Would be great if someone can shed some light on this issue.

@tuxedo0801
Copy link
Contributor Author

@tuxedo0801 tuxedo0801 commented Nov 24, 2017

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

@cimba007
Copy link

@cimba007 cimba007 commented Dec 11, 2017

Any chance to get this working with HTTPClient? My responses could become lager and I don't want to deal with chunked responses manually...

@odelot
Copy link

@odelot odelot commented Feb 9, 2018

a noob question: this fingerprint changes over time?

I know you are trying not to use them, but I have a valid one and I was thinking to hardcode it on my device.

@romkey
Copy link

@romkey romkey commented Mar 20, 2018

@odelot That's a good question.

Yes, fingerprints do change over time. They'll change any time the web server's SSL certificate changes.

So if a web server's SSL certificate expires on March 21, 2018 then it'll need a new certificate then and the new certificate will have a new fingerprint.

The server's certificate might easily change before the expiration date, too.

It's even possible for a web server which is load balanced by multiple servers to use different certificates on different servers, so you might see different fingerprints for the same HTTPS endpoint. That's not a likely scenario - it's probably the result of an odd or mis-configuration - but it's possible.

Using the fingerprint to verify the server isn't ideal, but the alternative is maintaining a list of certificates for the root certificate authorities - this is what more capable computers do. Unfortunately these need to be updated too, and their size well exceeds the storage available on an ESP8266.

Unfortunately I don't have a good alternative to hardcoding the fingerprint on your device.

@MikeS159
Copy link

@MikeS159 MikeS159 commented Mar 21, 2018

@odelot @romkey
There is an example here which lets you use the root certificates fingerprint. Root certs have much longer life spans, since they are hard coded into browser and operating systems, making them hard to update. The root certificate for GitHub (used in the example) is valid until 10 November 2031, 00:00:00 GMT.

To get a root certificate you can view it in a browser (click the green padlock), view certificate, select the root certificate and export it as a .der file. I wrote a little python script to format .der files as a hex string you can past straight into the CACert.ino file.

image

@davidverweij
Copy link

@davidverweij davidverweij commented Jun 3, 2018

@tuxedo0801 , I have a nearly identical error (stacktrace) - what do you mean by 2.4.0-rc2? My NodeMCU v1.0 crashes at the first attempt.

switches from 2.4.0-rc1 to rc2... looks better now. already running with >20requests without any crash.

@davidverweij
Copy link

@davidverweij davidverweij commented Jun 3, 2018

Apologies - after fiddeling with the variables I found I needed to add the 'https://' to the host adress in order for it to work. I was attempting a POST request a custom Google Cloud functions https endpoint. Somehow I could not find documentation on a Firebase/store request but not Cloud Functions. (e.g. .cloudfunctions.net). I did had a near similar error - so perhaps that might help? Cheers!

@devyte
Copy link
Collaborator

@devyte devyte commented Jul 13, 2018

Closing due to bearssl merge (axtls will be deprecated).

@devyte devyte closed this Jul 13, 2018
@C0rn3j
Copy link

@C0rn3j C0rn3j commented Sep 20, 2018

https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WiFi/examples

I can only find an example HTTPSRequest which still uses the fingerprint, am I looking in the wrong place?

@d-a-v
Copy link
Collaborator

@d-a-v d-a-v commented Sep 20, 2018

@C0rn3j check BearSSL_Validation.ino in this examples repository.
also check this

@emitterpark
Copy link

@emitterpark emitterpark commented Jan 12, 2019

To use https and ignore fingerprint validation, that worked for me.

Modified the verify function from the line 73 in ESP8266HTTPClient.cpp ,a different way.

bool verify(WiFiClient& client, const char* host) override
{
auto wcs = static_castaxTLS::WiFiClientSecure&(client);
//return wcs.verify(_fingerprint.c_str(), host);
wcs.verify(_fingerprint.c_str(), host);
return true;
}

Then in sketch when to begin client, Use a fake fingerprint, e.g. all Zeroes.

#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

void setup() {

....
HTTPClient https;

// this goes to: bool HTTPClient::begin(String url, String httpsFingerprint) or char array
https.begin("https://your full url..", "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00");
...

}

@d-a-v
Copy link
Collaborator

@d-a-v d-a-v commented Jan 12, 2019

You should not patch libraries but use their api (or propose patches/fixes):
What you need is available in this example
The improved API also prefers you give a WiFiClient as the first parameter

@emitterpark
Copy link

@emitterpark emitterpark commented Jan 13, 2019

For I am not a programmer, Sorry for the anomalous way in recent.
As you recommended, I restored ESP8266HTTPClient.cpp

Then using HTTPClient wrapper in this way could send an insecure get request on port 443.
Thank you so much.

`#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

const char *ssid = "my_ssid";
const char *pass = "my_password";

const char * host = "api.github.com";
const uint16_t port = 443;
const char * path = "/";

void setup() {

Serial.begin(115200);

Serial.println("conntecting to network..");
WiFi.begin(ssid, pass);
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
Serial.println("conntected to network..");

BearSSL::WiFiClientSecure client;
client.setInsecure();
HTTPClient https;

Serial.println("conntecting to server..");
if (https.begin(client, host, port, path)) {
int httpsCode = https.GET();
if (httpsCode > 0) {
Serial.println(httpsCode);
if (httpsCode == HTTP_CODE_OK) {
Serial.println(https.getString());
}
} else {
Serial.print("failed to GET");
}
} else {
Serial.print("failed to connect to server");
}

}
void loop() {

}`

@cyberphobia
Copy link

@cyberphobia cyberphobia commented Mar 17, 2019

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

@psychonaute
Copy link

@psychonaute psychonaute commented May 21, 2019

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

Thanks for this golden info :)

@nobilix
Copy link

@nobilix nobilix commented May 26, 2019

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

yes it kinda works (I have only some problems with Let's Encrypt Wildcard Certs), but it runs 10x longer. In my case it was 20 seconds using BearSSL certificate store vs 2 seconds using direct fingerprint string.

@technorainbows
Copy link

@technorainbows technorainbows commented Mar 25, 2020

Note from mtnr: This message is about esp32 and not applicable here

I found that simply by not setting the certificate you can get around the whole verify thing. This is because in the ssl_client library, which is used by WiFiClientSecure, if no certificate is provided it will default to not trying to verify it. See here

` // MBEDTLS_SSL_VERIFY_REQUIRED if a CA certificate is defined on Arduino IDE and
// MBEDTLS_SSL_VERIFY_NONE if not.

if (rootCABuff != NULL) {
    log_v("Loading CA cert");
    mbedtls_x509_crt_init(&ssl_client->ca_cert);
    mbedtls_ssl_conf_authmode(&ssl_client->ssl_conf, MBEDTLS_SSL_VERIFY_REQUIRED);
    ret = mbedtls_x509_crt_parse(&ssl_client->ca_cert, (const unsigned char *)rootCABuff, strlen(rootCABuff) + 1);
    mbedtls_ssl_conf_ca_chain(&ssl_client->ssl_conf, &ssl_client->ca_cert, NULL);
    //mbedtls_ssl_conf_verify(&ssl_client->ssl_ctx, my_verify, NULL );
    if (ret < 0) {
        return handle_error(ret);
    }
} else if (pskIdent != NULL && psKey != NULL) {
    log_v("Setting up PSK");
    // convert PSK from hex to binary
    if ((strlen(psKey) & 1) != 0 || strlen(psKey) > 2*MBEDTLS_PSK_MAX_LEN) {
        log_e("pre-shared key not valid hex or too long");
        return -1;
    }
    unsigned char psk[MBEDTLS_PSK_MAX_LEN];
    size_t psk_len = strlen(psKey)/2;
    for (int j=0; j<strlen(psKey); j+= 2) {
        char c = psKey[j];
        if (c >= '0' && c <= '9') c -= '0';
        else if (c >= 'A' && c <= 'F') c -= 'A' - 10;
        else if (c >= 'a' && c <= 'f') c -= 'a' - 10;
        else return -1;
        psk[j/2] = c<<4;
        c = psKey[j+1];
        if (c >= '0' && c <= '9') c -= '0';
        else if (c >= 'A' && c <= 'F') c -= 'A' - 10;
        else if (c >= 'a' && c <= 'f') c -= 'a' - 10;
        else return -1;
        psk[j/2] |= c;
    }
    // set mbedtls config
    ret = mbedtls_ssl_conf_psk(&ssl_client->ssl_conf, psk, psk_len,
             (const unsigned char *)pskIdent, strlen(pskIdent));
    if (ret != 0) {
        log_e("mbedtls_ssl_conf_psk returned %d", ret);
        return handle_error(ret);
    }
} else {
    mbedtls_ssl_conf_authmode(&ssl_client->ssl_conf, MBEDTLS_SSL_VERIFY_NONE);
    log_i("WARNING: Use certificates for a more secure communication!");
}`
@d-a-v
Copy link
Collaborator

@d-a-v d-a-v commented Mar 25, 2020

@technorainbows Are you working on esp32 ?
esp8266 (this repository) does not use mbedtls.

@JeanVanDenderFlume
Copy link

@JeanVanDenderFlume JeanVanDenderFlume commented Apr 18, 2020

Currently I'm testing the solution provided in #1851, but I would like to see, that I can make https-calls, without specifying a certificate-finger-print and without providing the root-ca-certificate to do certificate validation. Just "accept all certificates and ignore the complete certificate-check", like you can do on any other desktop-programming-language ...

Is this in general possible? Or is this technically on the esp8266 not possible?

Do you know if setInSecure with a valid certificate at the backend, not expired allow encryption of the connection line? I want toe accept several backends on several domains but i dont want to specifiy fingerprint or Cert Authority certificate with setCACert

@earlephilhower
Copy link
Collaborator

@earlephilhower earlephilhower commented Apr 18, 2020

setInsecure still encrypts. You just can't be sure you're talking to the remote site you think you are since the certificate is not checked.

@Bharath-M-R
Copy link

@Bharath-M-R Bharath-M-R commented May 18, 2020

Here's a write-up that explains how to use HttpClient while checking against an actual certificate store (rather than specifying a certificate fingerprint):

https://medium.com/@dfa_31434/doing-ssl-requests-on-esp8266-correctly-c1f60ad46f5e

very useful
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.