Latest Core Breaks WiFiClientSecure Insecure HTTPS

[bwjohns4]

Hardware:

Board: ESP32 Dev Module
Core Installation version: 1.0.6
IDE name: Platform.io
Flash Frequency: 40Mhz
PSRAM enabled: unsure
Upload Speed: PlatformIO default
Computer OS: Windows 10

Description:

WiFiClientSecure doesn't work when I upgrade to latest 1.0.6 Arduino Core (via PlatformIO Espressif32 3.2.0 Release). I'm not validating any certificates or thumbprints on the other end, just using it as insecure HTTPS. It works fine on 1.0.4 (via PlatformIO Espressif 3.0.0 Release), but breaks when upgrading to 1.0.6 (unsure if problem still exists in 1.0.5). I know that in 1.0.4 there is no .setInsecure() method, but that method has recently been added. I did try including that in my code, but that didn't make any recognizable difference.

I have so far traced it back to start_ssl_client() within ssl_client.cpp where it fails at the first check and returns -1, but haven't made it any further:

int start_ssl_client(sslclient_context *ssl_client, const char *host, uint32_t port, int timeout, const char *rootCABuff, const char *cli_cert, const char *cli_key, const char *pskIdent, const char *psKey, bool insecure)
{
    char buf[512];
    int ret, flags;
    int enable = 1;
    log_v("Free internal heap before TLS %u", ESP.getFreeHeap());

    if (rootCABuff == NULL && pskIdent == NULL && psKey == NULL && !insecure) {
        return -1; //***************It Fails Here**************************************
    }

Sketch: (leave the backquotes for code formatting)

int httpCheck(char *domainName){
  delay(5000);
  WiFiClientSecure secureClient;
  String requestString = "https://" + String(domainName) + "/";
  secureClient.setTimeout(20000);
  delay(0);
  BWJ_DEBUG_PRINTLN_FLASH("Starting TCP Connect");
  if(!secureClient.connect(domainName, uint16_t(443))){
    delay(1000); //!!!!!!!!!!!! This is absolutely required. Works with only 100ms but put 1000ms for overkill. Lots of troubleshooting to find error in WiFiclientSecure that requires extra delay upon failed connection. Otherwise temporarily blocks interrupts or just Serial.println, or more. Unknown. 2-2-21
    return -1;
  }
  delay(0);
  HTTPClient http;
  http.setTimeout(20000);
  BWJ_DEBUG_PRINTLN_FLASH("Starting HTTP: Begin()");
  http.begin(secureClient, requestString);
  BWJ_DEBUG_PRINTLN_FLASH("Running GET()");
  int httpResponseCode = http.GET();
  if (httpResponseCode>0) {
    BWJ_DEBUG_PRINT_FLASH("HTTP Response code: ");
    BWJ_DEBUG_PRINTLN(httpResponseCode);
    //String payload = http.getString();
  }
  else {
    BWJ_DEBUG_PRINT_FLASH("Error code: ");
    BWJ_DEBUG_PRINTLN(httpResponseCode);
  }
  http.end();
  secureClient.stop();
  return httpResponseCode;
  
}

Debug Messages:

09:44:21.330 > [V][HTTPClient.cpp:245] beginInternal(): url: https://my.domain.name/stuff/moreStuff
09:44:21.464 > [D][HTTPClient.cpp:293] beginInternal(): protocol: https, host: my.domain.name port: 443 url: /stuff/moreStuff
09:44:23.667 > [D][HTTPClient.cpp:579] sendRequest(): request type: 'GET' redirCount: 0
09:44:23.737 >
09:44:23.737 > [V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 259264
09:44:23.804 > [E][WiFiClientSecure.cpp:133] connect(): start_ssl_client: -1
09:44:23.867 > [V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.
09:44:23.963 > [D][HTTPClient.cpp:1118] connect(): failed connect to my.domain.name:443
09:44:24.037 > [W][HTTPClient.cpp:1417] returnError(): error(-1): connection refused
09:44:24.130 > [E][HTTPUpdate.cpp:231] handleUpdate(): HTTP error: connection refused
09:44:24.204 >
09:44:24.204 > [D][HTTPClient.cpp:400] disconnect(): tcp is closed
09:44:24.267 >
09:44:24.267 > [V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.
09:44:24.337 > [V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection. 

[me-no-dev]

you need to call secureClient.setInsecure(); before connect() :)

https://github.com/espressif/arduino-esp32/blob/master/libraries/WiFiClientSecure/src/WiFiClientSecure.h#L66

[bwjohns4]

@me-no-dev I already tried this but that does not fix it. I will try it again. In 1.0.4 .setInsecure() was not yet available, but since 1.0.6 had implemented this I went ahead and tried that without success.

[me-no-dev]

you did call it before secureClient.connect(domainName, uint16_t(443)) right? I guarantee you that it works ;)

[bwjohns4]

@me-no-dev Just tried it again with .setInsecure() added as shown below and it continues to fail. If I switch back to 1.0.4 if works as expected (just have to remove the .setInsecure()):

  String requestString = "https://" + String(domainName) + "/";
  secureClient.setTimeout(20000);
  secureClient.setInsecure();
  delay(0);
  BWJ_DEBUG_PRINTLN_FLASH("Starting TCP Connect");
  if(!secureClient.connect(domainName, uint16_t(443))){
    delay(1000);
    return -1;
  }

[lbernstone]

Unless you intend to interact directly with the client connection, there is no reason to include it. HTTPClient provides all the functionality you need.

[me-no-dev]

@bwjohns4 can you post the debug serial output again?

[bwjohns4]

@me-no-dev Well.... I spoke too soon. I have a few different WiFiClientSecure functions and I added the .setInsecure() to one that doesn't get called first, so it confused the test. After adding it to all calls, it appears to work fine. Apologies!

[bwjohns4]

Apparently after 1.0.4 (unsure of 1.0.5) you MUST use .setInsecure() whereas with 1.0.4 and before you did NOT include that (nor was it available)

[SheetLightning]

I have ran into the same issues. Arduino IDE 1.8.19, ESP32 library version 1.0.6.

Please forgive my confusion, but why would one use a HTTPS client only to then use a method called .setInsecure()? In that case what's the point of using a https library in the first place?

.setInsecure() does allow me to establish a connection, but when I do a subsequent GET request I get:

[E][ssl_client.cpp:36] _handle_error(): [data_to_read():287]: (-29184) SSL - An invalid SSL record was received
The URL I am trying works perfectly fine in the browser on the PC.

I also tried reverting to 1.0.4. It works without .setInsecure(), but I get:

[I][ssl_client.cpp:156] start_ssl_client(): WARNING: Use certificates for a more secure communication!

The subsequent GET request still returns the same:

[E][ssl_client.cpp:33] _handle_error(): [data_to_read():270]: (-29184) SSL - An invalid SSL record was received

[SheetLightning]

The referenced fix by effitient does not appear to have fixed the problem. The problem still continues.