"Chain could not be linked to a trust anchor" using mongoDB stitch

[AugustoCiuffoletti]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [Wemos D1-mini]
• Core Version: [2.5.2]
• Development Env: [Arduino IDE]
• Operating System: [Ubuntu]

Settings in IDE

• Module: [Wemos D1 mini r2]
• Flash Mode: [dio?]
• Flash Size: [4MB]
• lwip Variant: [v2 Lower Memory]
• Reset Method: [?]
• Flash Frequency: [?]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [921600] (serial upload only)

I am trying to interface a Wemos with a mongoDB stitch using HTTPS with fingerprint authentication, and I meet the problem in the title. I am successfully running a similar interface with mLab, another database service from the same provider.

To reproduce my problem, start from the HTTPSRequest example in ESP8266WiFi, and replace

const char* host = "api.github.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char fingerprint[] PROGMEM = "5F F1 60 31 09 04 3E F2 90 D2 B0 8A 50 38 04 E8 37 9F BC 76";

with

const char* host = "eu-west-1.aws.webhooks.mongodb-stitch.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char fingerprint[] PROGMEM = "73 5D 6B A2 F7 ED 7C 72 74 AC A3 F5 67 F0 56 6B 68 3B 4B 47";

Adding the following lines just before the "connection failed" printout around line 60 we have some debugging info. The output on Serial is the following:

...
WiFi connected
IP address: 
192.168.113.133
connecting to eu-west-1.aws.webhooks.mongodb-stitch.com
Using fingerprint '73 5D 6B A2 F7 ED 7C 72 74 AC A3 F5 67 F0 56 6B 68 3B 4B 47'
Chain could not be linked to a trust anchor.
connection failed

Replacing the ''fingerprint()'' method with ''setInsecure()'' everything works fine with no errors.

Inspecting the SSL protocol with

openssl s_client -connect <host>:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin

when is eu-west-1.aws.webhooks.mongodb-stitch.com I obtain (only the chain"):

Certificate chain
 0 s:/C=US/ST=New York/L=New York/O=MongoDB, Inc./CN=*.mongodb.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

With the mLab server api.mlab.com (which does not exhibit the problem) the chain is similar:

Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=ObjectLabs Corporation/CN=api.mlab.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

My suspect is that either the library or the certificate are not exactly compliant. I started asking you.

Thank you for your attention.

[earlephilhower]

FWIW the URL for the eu-west-1 stitch can't negotiate a proper connection with Firefox, either, but that might just be some regional restrictions in place so I can't get to it from US-West.

Fingerprints don't actually check chains, they just take the SHA fingerprint of the server and the one you send in and compare. I imagine there are load balancers in between you and the service, so it might be that you're getting the cert for the load balancer which is not always going to be the same as you bounce around.

A quick OpenSSL run on my own server seems to match your fingerprint, though.

The other thing that could be happening is that, technically, the SHA1 fingerprint is taken on a specially formatted ASN.1 document (since reordering elements could cause a different fingerprint). BearSSL doesn't support reordering elements inside the ASN.1 since it could require 10s of KB of RAM and breaks the very cool stream setup they have.

Since you're tying yourself to a specific cert with specific pub/private key already with the fingerprint, I would suggest just extracting the public key from the cert and using it as a KnownKey. You're just as secure as before, because you can't negotiate with anyone who doesn't have that server's private key, anyway.

[AugustoCiuffoletti]

In fact, I was wondering about the solution in the last lines of your reply.
After posting I noticed that the Common Name in the chain does not match the FQDN (mongodb-stitch.com vs *.mongodb.com)? How strict (or needed) is the match?

[earlephilhower]

Fingerprint doesn't check anything in the certificate, only that the final hash of its contents matches what is expected. So SNI is ignored, expiry dates, etc. Using knownkey is simple enough and should get you what you want without doing a full validation.

[AugustoCiuffoletti]

I guess your reply closes the issue.
Thank you very much!

[AugustoCiuffoletti]

My guess was wrong.

I have tried to use the pubkey as suggested, but the problem is still there. From a reply on the thread I opened on MongoDB community, your words, and a number of posts that I found googling around, my new guess is that the problem is that SNI thing, which is not implemented in BearSSL. Broadly speaking, it corresponds to the "load balancer" situation you mentioned. In the case of MongoDB-Stitch, which is a serverless cloud provision, it is totally justified and, IMHO, unavoidable.

From the other thread (https://jira.mongodb.org/browse/SUPPORT-2675, but you need to be registered to have access) I obtain the following reply to a direct answer:

"Hi Augusto Ciuffoletti, we don't believe there is an issue as we require clients to support SNI such that the right certificate is served. I believe this would be an issue on the Arduino side."

Is there a way out?

[earlephilhower]

BearSSL supports SNI https://bearssl.org/gitweb/?p=BearSSL;a=blob;f=inc/bearssl_ssl.h;h=f2ecc3782828bb54a18e625b72db46aefb061c5b;hb=HEAD#l2814 which means the connect(host, port) host needs to correspond to the hostname that SNI will present.

We call the API with the hostname you used in your connect() call at

Arduino/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp

Line 1075 in 5a47cab

if (!br_ssl_client_reset(_sc.get(), hostName, _session?1:0)) {

[earlephilhower]

@d-a-v, was there a way to get packet dumps from the 8266? This would be very simple to diagnosis using Wireshark or even plain hex dumps of the SSL handshake.

[AugustoCiuffoletti]

Use a (Linux) PC as WiFi hotspot and use tcpdump on the wifi iface? Il giorno ven 28 giu 2019 alle ore 22:38 Earle F. Philhower, III < notifications@github.com> ha scritto:

…

@d-a-v <https://github.com/d-a-v>, was there a way to get packet dumps from the 8266? This would be very simple to diagnosis using Wireshark or even plain hex dumps of the SSL handshake. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#6209?email_source=notifications&email_token=AGEP724LVKZNTCYU65XN363P4ZZELA5CNFSM4HZB7N22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3DM6Y#issuecomment-506869371>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGEP722HDK64WJZBX5RXTR3P4ZZELANCNFSM4HZB7N2Q> .

-- Augusto Ciuffoletti Dipartimento di Informatica Università di Pisa 56100 - Pisa (Italy)

[d-a-v]

Use a (Linux) PC as WiFi hotspot and use tcpdump on the wifi iface?

Yes, or this from inside the esp if you have a closed firmware router: https://github.com/d-a-v/EspGoodies/tree/master/examples/netdump (check also the tcpdump example) Or @hreinkte fork on my repository

[earlephilhower]

@AugustoCiuffoletti , any luck yet? Again, since it's against your private shard I can't reproduce anything locally so without packet traces I don't think we're going to be able to get very far here...

[AugustoCiuffoletti]

Sorry, but it is not a priority for me right now, as I prefer to finish the project, which aims at research, not production. I'll try to provide you a trace and details during next week, the task is in my todo list now. Il giorno sab 6 lug 2019 alle ore 06:22 Earle F. Philhower, III < notifications@github.com> ha scritto:

…

@AugustoCiuffoletti <https://github.com/AugustoCiuffoletti> , any luck yet? Again, since it's against your private shard I can't reproduce anything locally so without packet traces I don't think we're going to be able to get very far here... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6209?email_source=notifications&email_token=AGEP727UXO23ZHFRHJITA43P6AMWLA5CNFSM4HZB7N22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZKSEWA#issuecomment-508895832>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGEP727XEXF23QSLHYVLIPLP6AMWLANCNFSM4HZB7N2Q> .

-- Augusto Ciuffoletti Dipartimento di Informatica Università di Pisa 56100 - Pisa (Italy)

[AugustoCiuffoletti]

Dear @earle and others,

I've collected the trace for the fingerprint case, which is the initial issue in this thread.

This is the story, useful to reproduce:

• I created a webhook on my MongoDB service that responds with the string "Hello world" to a GET. It is the skeleton example that is initialized upon webhook creation;

• I modified as follows the HTTPSRequest example referenced above:

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>
#include <NetDump.h>
#include "secret.h"

const char* ssid = STASSID;
const char* password = STAPSK;

const char* host = "eu-west-1.aws.webhooks.mongodb-stitch.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate
const char fingerprint[] PROGMEM = "73 5D 6B A2 F7 ED 7C 72 74 AC A3 F5 67 F0 56 6B 68 3B 4B 47";
//const char fingerprint[] PROGMEM = "5F F1 60 31 09 04 3E F2 90 D2 B0 8A 50 38 04 E8 37 9F BC 76";

void dump (int netif_idx, const char* data, size_t len, int out, int success) {
  (void)success;
  Serial.print(out ? F("out ") : F(" in "));
  Serial.printf("%d ", netif_idx);

  // optional filter example: if (netDump_is_ARP(data))
  {
    //netDump(Serial, data, len);
    netDumpHex(Serial, data, len);
  }
}

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

//  phy_capture = dump;

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);

  Serial.printf("Using fingerprint '%s'\n", fingerprint);
  client.setFingerprint(fingerprint);
//  client.setInsecure();

  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }

  String url = "/api/client/v2.0/app/manage-jjtug/service/exampleHTTPS/incoming_webhook/webhook0";
  Serial.print("requesting URL: ");
  Serial.println(url);

  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "User-Agent: BuildFailureDetectorESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');
  if (line.startsWith("{\"state\":\"success\"")) {
    Serial.println("esp8266/Arduino CI successfull!");
  } else {
    Serial.println("esp8266/Arduino CI has failed");
  }
  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println("closing connection");
}

void loop() {
}

Notes: I pasted in the "dump" function from @d-a-v examples and added the definition of the phy_capture hook, moved the WiFi credential in a secret.h file, changed the definition of host, url and fingerprint.

• I uploaded the sketch to a WEMOS D1 R1 using the Arduino IDE (V1.8.9) and tested with the setInsecure method first, and then with the setFingerprint. With dump disabled in the first case I obtained the expected response:

WiFi connected
IP address: 
192.168.113.122
connecting to eu-west-1.aws.webhooks.mongodb-stitch.com
Using fingerprint '73 5D 6B A2 F7 ED 7C 72 74 AC A3 F5 67 F0 56 6B 68 3B 4B 47'
requesting URL: /api/client/v2.0/app/manage-jjtug/service/exampleHTTPS/incoming_webhook/webhook0
request sent
headers received
esp8266/Arduino CI has failed
reply was:
==========
"Hello World!"
==========
closing connection

while using the fingerprint:

WiFi connected
IP address: 
192.168.113.122
connecting to eu-west-1.aws.webhooks.mongodb-stitch.com
Using fingerprint '73 5D 6B A2 F7 ED 7C 72 74 AC A3 F5 67 F0 56 6B 68 3B 4B 47'
connection failed

• I ran the fingerprint case with dump enabled to obtain a trace that I imported in Wireshark so to remove unwanted traffic, with the result shown in the screenshot

• Finally, from Wireshark I saved the trace in a pcap file that you can analyze

I leave the thread open, right?

Thank you for your help,

Augusto