GET following secureClient.setInsecure(); results in SSL - An invalid SSL record was received

[SheetLightning]

Board

ESP32 WROOM

Device Description

ESP32 Lolin

Hardware Configuration

No

Version

v1.0.6

IDE Name

Arduino IDE version 1.8.19

Operating System

Linux Mint

Flash frequency

40MHz

PSRAM enabled

no

Upload speed

115200

Description

New issue to reference issue 4992 which is still not fixed and has been closed.

#4992

I am still having the problem with the latest version of the ESP32 library for the Arduino IDE.

Sketch

None provided at this stage

Debug Message

[E][ssl_client.cpp:36] _handle_error(): [data_to_read():287]: (-29184) SSL - An invalid SSL record was received

Other Steps to Reproduce

Set secureClient.setInsecure(); then perform a get request.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[lbernstone]

You must provide a certificate authority cert (CAcert) for the site if you want to connect to a protected site. You can set setInsecure before the connection, which tells WiFiClientSecure to accept whatever certificate is presented by the server. This is not actually secure, since there is no authority backing up the server's claim of authenticity.

[podaen]

I think your site you want to connect to doesn't support insecure requests. Some sites doesn't simply approve that kind of connection.

[SuGlider]

Not being authenticated is the reason for receiving the message SSL - An invalid SSL record was receive
As said by @lbernstone and @podaen - it most likely is an error of certificates.

Not an issue of Arduino. Please try #4992 (comment) in order to bypass it, but be aware of the warning in the commentary below:

https://github.com/espressif/arduino-esp32/blob/master/libraries/WiFiClientSecure/src/WiFiClientSecure.h#L68

[podaen]

Put your verbose log on and you would be more informed about what happens.

[SheetLightning]

Unfortunately verbose doesn't give me much than debug mode. I am trying to connect with a web app set up on Google script. I have no problem with this on an ESP8266 but unfortunately the same does not work on an ESP32.

Google's SSL certificate is very unlikely to the invalid. I have spent considerable time (more than a couple of hours) looking for how to set up the appropriate certificate chain so that the ESP32 Web Client might be able to read Google's SSL certificate, but have not really found anything useful. My research unfortunately ended in frustration. I tried a few online examples but did not really get the confidence that any of the information and examples I found were actually appropriate to this situation. Nothing actually worked. I just kept getting the same error. If anyone knows a reliable source that demonstrates how to set up the ESP32 to work with Google Calendar and Google spreadsheet I would be grateful for a link. Most of the projects I found were using the ESP8266, probably with good reason.

I appreciate that using setInsecure() means that the connection is not SSL encrypted and therefore not secure.

In the meantime, this is the log with log level 'debug' enabled:

[I][ssl_client.cpp:127] start_ssl_client(): WARNING: Skipping SSL Verification. INSECURE!
Connected to Google
Start Request...
[I][ssl_client.cpp:127] start_ssl_client(): WARNING: Skipping SSL Verification. INSECURE!
Connected.
Fetching events from: 
https://script.google.com:/macros/s/AKfycbz7O9aCiFl7IEiRLxYe5TjMN-ssR07fZhIwBqUZEHrA6G2zibw/exec
[E][ssl_client.cpp:36] _handle_error(): [data_to_read():287]: (-29184) SSL - An invalid SSL record was received

This the the log with 'verbose' level enabled:

Start Request...
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 216492
[V][ssl_client.cpp:65] start_ssl_client(): Starting socket
[V][ssl_client.cpp:104] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:113] start_ssl_client(): Setting up the SSL/TLS structure...
[I][ssl_client.cpp:127] start_ssl_client(): WARNING: Skipping SSL Verification. INSECURE!
[V][ssl_client.cpp:197] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:212] start_ssl_client(): Performing the SSL/TLS handshake...
[V][ssl_client.cpp:233] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:242] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:257] start_ssl_client(): Free internal heap after TLS 179960
Fetching events from: 
https://script.google.com:/macros/s/AKfycbz7O9aCiFl7IEiRLxYe5TjMN-ssR07fZhIwBqUZEHrA6G2zibw/exec
[V][ssl_client.cpp:295] send_ssl_data(): Writing HTTP request with 133 bytes...
[E][ssl_client.cpp:36] _handle_error(): [data_to_read():287]: (-29184) SSL - An invalid SSL record was received
[V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.

This is my calendar fetching function. connectGoogleHost() works fine. bool getCalendar() which is executed when connectGoogleHost() returns true (connected) successfully connects but the subsequent GET request fails.

bool connectGoogleHost(){
  // Use HTTPSRedirect class to create a new TLS connection
  gclient = new HTTPSRedirect(httpsPort);
  gclient->setInsecure();
//  gclient->setCACert(google_com_root_ca);
  gclient->setPrintResponseBody(true);
  gclient->setContentTypeHeader("application/json");

  Serial.print("Connecting to ");
  Serial.println(googleHost);

  // Try to connect for a maximum of 3 times
  bool flag = false;
  for (int i = 0; i < 3; i++) {
//    Serial.print(F("Attempt: "));
//    Serial.println(i);
    int retval = gclient->connect(googleHost, httpsPort);
    if (retval == 1) {
      flag = true;
      break;
    }else{
      Serial.println("Connection failed. Retrying...");
    }
  }

  if (!flag) {
    Serial.print("Could not connect to server: ");
    Serial.println(googleHost);
//    Serial.println("Exiting...");
//    ESP.reset();
    return false;
  }
  Serial.println("Connected to Google");
  return true;
}


bool getCalendar() {
//  HTTPSRedirect gclient;
//  gclient = new HTTPSRedirect(httpsPort);
//  gclient->setInsecure();
//  gclient->setPrintResponseBody(true);
//  gclient->setContentTypeHeader("application/json");

  String calendarData = "";

  Serial.println("Start Request...");
//  unsigned long getCalenderEntry = millis();

  // Try to connect for a maximum of 3 times
  bool flag = false;
//  gclient->setCACert(google_com_root_ca);
  for (int i = 0; i < 3; i++) {
    int retval = gclient->connect(googleHost, httpsPort);
    
    if (retval == 1) {
      flag = true;
      Serial.println(F("Connected."));
      break;
    }else{
      Serial.println(F("Connection failed. Retrying..."));
      delay(2000);  // Wait a while before re-trying
    }
  }
  if (!flag) {
    Serial.print("Could not connect to server: ");
    Serial.println(googleHost);
//    Serial.println("Exiting...");
//    ESP.reset();
    return false;
  }
  //Fetch Google Calendar events
  String url = String("/macros/s/") + GScriptIdRead + "/exec";
  Serial.println(F("Fetching events from: "));
  Serial.print("https://");
  Serial.print(googleHost);
  Serial.print(":");
  Serial.println(url);
  gclient->GET(url, googleHost);
  Serial.println(F("Performed GET"));
  calendarData = gclient->getResponseBody();
  Serial.println(calendarData);

  if (calendarData.length() > 0) parseEvents(calendarData);

  printEvents();
  
//  Serial.print("Calendar Data---> ");
//  Serial.println(calendarData);
//  calenderUpToDate = true;
  yield();
  return true;
}

[podaen]

I appreciate that using setInsecure() means that the connection is not SSL encrypted and therefore not secure.

You will keep running the ssl library and get probably an x.509 certificate. That's normal, but you want to connect to google. That is something else. I tried it too a while a go but it didn't work for me ether. I don't think it possible with the original tcp on the esp, because google is base64 encoded and so on and so on, if I am remembering it good.

[me-no-dev]

I appreciate that using setInsecure() means that the connection is not SSL encrypted and therefore not secure.

This is not what it means :) Your connection IS encrypted, BUT the certificate is not validated if authentic. This allows you to connect to places where you do not have certificates for (or have not added proper ROOT certificates), but it also allows you to connect to malicious site, pretending to be someone else.

[SheetLightning]

Well now that is interesting. So the setting basically causes the request to ignore the certificate chain and validation? Thanks for explaining that. However, if that is the case, then the ball surely rests squarely in Esspressif's court? If my (and the original authors) sketch works flawlessly on an ESP8266, then why shouldn't it work using the equivalent web client libraries (which use identical syntax - no changes required to code, only the names of libraries in the #include statements) on the ESP32?

This is the project I am following:
https://github.com/SensorsIot/Reminder-with-Google-Calender

I would be happy to put the proper certificate chain in place to make this secure if I could find some documentation to show me how? As I previously mentioned, I can find information on how to add a single certificate, but regarding how to add multiple certificates and public keys as would be required in a certificate chain where, for example, intermediate certificates are required.

[me-no-dev]

@SheetLightning previous behavior is very insecure by default. Now it is not. ESP32 is used in many projects that do require proper security. You can add the following snippet before client.connect(); and it will work on both platforms

#ifdef ARDUINO_ARCH_ESP32
client.setInsecure();
#endif
client.connect();