Skip to content
This repository has been archived by the owner on Jan 6, 2023. It is now read-only.

ESP32-lwmqtt example does not works with latest WiFiClientSecure library #221

Open
Yaaximus opened this issue Apr 8, 2021 · 41 comments
Open

Comments

@Yaaximus
Copy link

Yaaximus commented Apr 8, 2021

With latest changes in the past few months in WIFIClientSecure library (https://github.com/espressif/arduino-esp32/tree/master/libraries/WiFiClientSecure) which is a prerequisite for getting ESP32-lwmqtt example functional.
The changes ensures that there is a root certificate or else it does not continue and return with an error code of "-1" as shown in code below from file ssl_client.cpp in WIFIClientSecure

int start_ssl_client(sslclient_context *ssl_client, const char *host, uint32_t port, int timeout, const char *rootCABuff, const char *cli_cert, const char *cli_key, const char *pskIdent, const char *psKey, bool insecure)
{
    char buf[512];
    int ret, flags;
    int enable = 1;
    log_v("Free internal heap before TLS %u", ESP.getFreeHeap());

    if (rootCABuff == NULL && pskIdent == NULL && psKey == NULL && !insecure) {
        return -1;
    }

What I have noticed is that as for now root certificate defined in ciotc_config.h file by the variable "root_cert" in line 63 is not used anywhere in the implementation which causes the secure connection to fail with the given error as shown from output below

Waiting on time sync...
checking wifi...Connecting...
Refreshing JWT
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 270164
[E][WiFiClientSecure.cpp:133] connect(): start_ssl_client: -1
[V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.
not connected
Settings incorrect or missing a cyper for SSL
Connect with mqtt.2030.ltsapis.goog:8883
ClientId: projects/my-project-name/locations/us-central1/registries/my-registry-name/devices/my-device-name
Waiting 60 seconds, retry will likely fail

Please let me know if my understanding is not right but if it is then Are we planning to use the root certificate in future?

For anyone else who is facing the same problem please revert back to the following commit on Jan 21, 2020 of WIFIClientSecure repository. Commit ID:"ed59ae6482f6b94dd36848c50f0c65727c26a02d" (https://github.com/espressif/arduino-esp32/tree/ed59ae6482f6b94dd36848c50f0c65727c26a02d/libraries/WiFiClientSecure)where the above mentioned changes are not made and at-least in my case with this commit I am able to establish connection with Clout IOT Core

@raphael-bmec-co
Copy link

If you call this line wifiClientSecure->setInsecure(); does it resolve the issue?

@Yaaximus
Copy link
Author

Yaaximus commented Apr 8, 2021

No it does not work because in esp32-mqtt.h file line 16 Clint.h is included

line:16 #include <Client.h>

netClient pointer, pointing to address of class Client is diclared

line:35 Client *netClient;

netClient is instentiated as an object of Class WIFIClientSecure

line:112 netClient = new WiFiClientSecure();

setInsecure() is defined in WIFIClientSecure

class WiFiClientSecure : public WiFiClient
{
public:
    WiFiClientSecure *next;
    WiFiClientSecure();
    WiFiClientSecure(int socket);
    ~WiFiClientSecure();
    int connect(IPAddress ip, uint16_t port);
    int connect(IPAddress ip, uint16_t port, int32_t timeout);
    int connect(const char *host, uint16_t port);
    int connect(const char *host, uint16_t port, int32_t timeout);
    int connect(IPAddress ip, uint16_t port, const char *rootCABuff, const char *cli_cert, const char *cli_key);
    int connect(const char *host, uint16_t port, const char *rootCABuff, const char *cli_cert, const char *cli_key);
    int connect(IPAddress ip, uint16_t port, const char *pskIdent, const char *psKey);
    int connect(const char *host, uint16_t port, const char *pskIdent, const char *psKey);
	int peek();
    size_t write(uint8_t data);
    size_t write(const uint8_t *buf, size_t size);
    int available();
    int read();
    int read(uint8_t *buf, size_t size);
    void flush() {}
    void stop();
    uint8_t connected();
    int lastError(char *buf, const size_t size);
    void setInsecure(); // Don't validate the chain, just accept whatever is given.  VERY INSECURE!


but Client class doesn't have the declaration for setInsecure();
as shown below from file Client.h

class Client: public Stream
{
public:
    virtual int connect(IPAddress ip, uint16_t port) =0;
    virtual int connect(const char *host, uint16_t port) =0;
    virtual size_t write(uint8_t) =0;
    virtual size_t write(const uint8_t *buf, size_t size) =0;
    virtual int available() = 0;
    virtual int read() = 0;
    virtual int read(uint8_t *buf, size_t size) = 0;
    virtual int peek() = 0;
    virtual void flush() = 0;
    virtual void stop() = 0;
    virtual uint8_t connected() = 0;
    virtual operator bool() = 0;
protected:
    uint8_t* rawIPAddress(IPAddress& addr)
    {
        return addr.raw_address();
    }
};

Which gives the error

'class Client' has no member named 'setInsecure'

error message

Arduino: 1.8.13 (Linux), Board: "ESP32 Dev Module, Disabled, Default 4MB with spiffs (1.2MB APP/1.5MB SPIFFS), 240MHz (WiFi/BT), QIO, 80MHz, 4MB (32Mb), 921600, None"

In file included from /home/yasim/Esp32-lwmqtt-working/Esp32-lwmqtt-working.ino:39:0:
sketch/esp32-mqtt.h: In function 'void setupCloudIoT()':
esp32-mqtt.h:113:14: error: 'class Client' has no member named 'setInsecure'
   netClient->setInsecure();
              ^
Multiple libraries were found for "WiFi.h"
 Used: /home/yasim/.arduino15/packages/esp32/hardware/esp32/1.0.6/libraries/WiFi
 Not used: /home/yasim/Downloads/arduino-1.8.13/libraries/WiFi
exit status 1
'class Client' has no member named 'setInsecure'


This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

@raphael-bmec-co
Copy link

and if you cast netClient to type WIFIClientSecure:

WiFiClientSecure& wifiClientSecure= static_cast<WiFiClientSecure&>(netClient);
wifiClientSecure->setInsecure();

@Yaaximus
Copy link
Author

Yaaximus commented Apr 8, 2021

Then changes are also to be made in the following files "CloudIoTCoreMqtt.h,CloudIoTCoreMqtt.cpp"

Changes made:
In line 35 file esp32-mqtt.h

//Client *netClient;
WiFiClientSecure *netClient;

CloudIoTCoreMqtt.cpp from line 24 to line 32

///////////////////////////////
// MQTT common functions
///////////////////////////////
CloudIoTCoreMqtt::CloudIoTCoreMqtt(
    MQTTClient *_mqttClient, WiFiClientSecure *_netClient, CloudIoTCoreDevice *_device){
  this->mqttClient = _mqttClient;
  this->netClient = _netClient;
  this->device = _device;
}

include WIFIClientSecure in CloudIoTCoreMqtt.h file

#include <WiFiClientSecure.h>

In the same file CloudIoTCoreMqtt.h replace "Client" by "WiFiClientSecure" in line 35 and 39

    MQTTClient *mqttClient;
    WiFiClientSecure *netClient;
    CloudIoTCoreDevice *device;

  public:
    CloudIoTCoreMqtt(MQTTClient *mqttClient, WiFiClientSecure *netClient, CloudIoTCoreDevice *device);

After netClient = new WiFiClientSecure(); in line 113 of file esp32-mqtt.h

netClient = new WiFiClientSecure();
netClient->setInsecure();

Removing the use of Client Class and by setting the connection to InSecure, After making the following changes the connection is established. But the point remains by doing the above changes the connection has been made Insecure and it is still not using the root certificate, So basically its still not working with WIFIClientSecure connection rather with WIFIClientInsecure

@raphael-bmec-co
Copy link

@Yaaximus I am not an expert in this field but it is my understanding that there is a degree of security provided by the JWT.

After casting you should be able to set the certificate on the netClient?

@Yaaximus
Copy link
Author

Yaaximus commented Apr 9, 2021

Yes JWT adds a degree of security for sure.

After casting certificate can be set using the setCACert() but so far I was not able to establish secure connection using Long-term support (LTS) as mentioned here (https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#iot-core-mqtt-auth-run-python).
by using the instructions provided in the ciot_config.h file line 56

openssl s_client -showcerts -connect mqtt.2030.ltsapis.goog:8883

When I run the above command this is the output I get

CONNECTED(00000003)
depth=1 C = US, O = Google Trust Services LLC, CN = GTS LTSX
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.2030.ltsapis.goog
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.2030.ltsapis.goog
   i:C = US, O = Google Trust Services LLC, CN = GTS LTSX
-----BEGIN CERTIFICATE-----
MIIDDDCCArKgAwIBAgIURjzskJE39lD3SX0IkTs5jt4noWYwCgYIKoZIzj0EAwIw
RDELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
TEMxETAPBgNVBAMTCEdUUyBMVFNYMB4XDTIwMDcxMzAwMDAwMFoXDTIxMDcxMjAw
MDAwMFowbTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNV
BAcMDU1vdW50YWluIFZpZXcxEzARBgNVBAoMCkdvb2dsZSBMTEMxHDAaBgNVBAMM
EyouMjAzMC5sdHNhcGlzLmdvb2cwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARh
fb8NcIBEcwbwSJKKlnjJ7cDwKMHiHeQd1xgE4PWzqEFxauPG30WniyoFZQDtgqft
Tq6iImzs1iP4cR5xN9Hao4IBVzCCAVMwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDgYD
VR0PAQH/BAQDAgeAMB4GA1UdEQQXMBWCEyouMjAzMC5sdHNhcGlzLmdvb2cwDAYD
VR0TAQH/BAIwADAfBgNVHSMEGDAWgBSzK6ugSBx+E4rJCMRAQiKiNlHiCjBpBggr
BgEFBQcBAQRdMFswLwYIKwYBBQUHMAKGI2h0dHA6Ly9wa2kuZ29vZy9ndHNsdHNy
L2d0c2x0c3guY3J0MCgGCCsGAQUFBzABhhxodHRwOi8vb2NzcC5wa2kuZ29vZy9H
VFNMVFNYMCEGA1UdIAQaMBgwDAYKKwYBBAHWeQIFAzAIBgZngQwBAgIwMAYDVR0f
BCkwJzAloCOgIYYfaHR0cDovL2NybC5wa2kuZ29vZy9HVFNMVFNYLmNybDAdBgNV
HQ4EFgQU3VXinuGqY4eo6dBUviU9iLPkWTcwCgYIKoZIzj0EAwIDSAAwRQIhAO4t
PT9rr/pZxYx3KSzXRK0l0bxpVELE2WS43FIYvWYzAiBwEHAunJZ5oBXfPXH6sJkr
Xt+ea9ZwVMYLXWQnGvKQ+Q==
-----END CERTIFICATE-----
 1 s:C = US, O = Google Trust Services LLC, CN = GTS LTSX
   i:C = US, O = Google Trust Services LLC, CN = GTS LTSR
-----BEGIN CERTIFICATE-----
MIIC0TCCAnagAwIBAgINAfQKmcm3qFVwT0+3nTAKBggqhkjOPQQDAjBEMQswCQYD
VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzERMA8G
A1UEAxMIR1RTIExUU1IwHhcNMTkwMTIzMDAwMDQyWhcNMjkwNDAxMDAwMDQyWjBE
MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
QzERMA8GA1UEAxMIR1RTIExUU1gwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARr
6/PTsGoOg9fXhJkj3CAk6C6DxHPnZ1I+ER40vEe290xgTp0gVplokojbN3pFx07f
zYGYAX5EK7gDQYuhpQGIo4IBSzCCAUcwDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1Ud
DgQWBBSzK6ugSBx+E4rJCMRAQiKiNlHiCjAfBgNVHSMEGDAWgBQ+/v/MUuu/ND49
80DQ5CWxX7i7UjBpBggrBgEFBQcBAQRdMFswKAYIKwYBBQUHMAGGHGh0dHA6Ly9v
Y3NwLnBraS5nb29nL2d0c2x0c3IwLwYIKwYBBQUHMAKGI2h0dHA6Ly9wa2kuZ29v
Zy9ndHNsdHNyL2d0c2x0c3IuY3J0MDgGA1UdHwQxMC8wLaAroCmGJ2h0dHA6Ly9j
cmwucGtpLmdvb2cvZ3RzbHRzci9ndHNsdHNyLmNybDAdBgNVHSAEFjAUMAgGBmeB
DAECATAIBgZngQwBAgIwCgYIKoZIzj0EAwIDSQAwRgIhAPWeg2v4yeimG+lzmZAC
DJOlalpsiwJR0VOeapY8/7aQAiEAiwRsSQXUmfVUW+N643GgvuMH70o2Agz8w67f
SX+k+Lc=
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.2030.ltsapis.goog

issuer=C = US, O = Google Trust Services LLC, CN = GTS LTSX

---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1849 bytes and written 434 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
read:errno=0

In the output I can see that there is an error

verify error:num=20:unable to get local issuer certificate

And even if I add the above certificate in var root_cert by following the instructions provided in the file ciot_config.h line 57 and 58

Copy the certificate (all lines between and including ---BEGIN CERTIFICATE---
// and --END CERTIFICATE--) to root.cert and put here on the root_cert variable.
const char *root_cert =

      "-----BEGIN CERTIFICATE-----\n"
      "MIIDDDCCArKgAwIBAgIURjzskJE39lD3SX0IkTs5jt4noWYwCgYIKoZIzj0EAwIw\n"
      "RDELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM\n"
      "TEMxETAPBgNVBAMTCEdUUyBMVFNYMB4XDTIwMDcxMzAwMDAwMFoXDTIxMDcxMjAw\n"
      "MDAwMFowbTELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNV\n"
      "BAcMDU1vdW50YWluIFZpZXcxEzARBgNVBAoMCkdvb2dsZSBMTEMxHDAaBgNVBAMM\n"
      "EyouMjAzMC5sdHNhcGlzLmdvb2cwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARh\n"
      "fb8NcIBEcwbwSJKKlnjJ7cDwKMHiHeQd1xgE4PWzqEFxauPG30WniyoFZQDtgqft\n"
      "Tq6iImzs1iP4cR5xN9Hao4IBVzCCAVMwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDgYD\n"
      "VR0PAQH/BAQDAgeAMB4GA1UdEQQXMBWCEyouMjAzMC5sdHNhcGlzLmdvb2cwDAYD\n"
      "VR0TAQH/BAIwADAfBgNVHSMEGDAWgBSzK6ugSBx+E4rJCMRAQiKiNlHiCjBpBggr\n"
      "BgEFBQcBAQRdMFswLwYIKwYBBQUHMAKGI2h0dHA6Ly9wa2kuZ29vZy9ndHNsdHNy\n"
      "L2d0c2x0c3guY3J0MCgGCCsGAQUFBzABhhxodHRwOi8vb2NzcC5wa2kuZ29vZy9H\n"
      "VFNMVFNYMCEGA1UdIAQaMBgwDAYKKwYBBAHWeQIFAzAIBgZngQwBAgIwMAYDVR0f\n"
      "BCkwJzAloCOgIYYfaHR0cDovL2NybC5wa2kuZ29vZy9HVFNMVFNYLmNybDAdBgNV\n"
      "HQ4EFgQU3VXinuGqY4eo6dBUviU9iLPkWTcwCgYIKoZIzj0EAwIDSAAwRQIhAO4t\n"
      "PT9rr/pZxYx3KSzXRK0l0bxpVELE2WS43FIYvWYzAiBwEHAunJZ5oBXfPXH6sJkr\n"
      "Xt+ea9ZwVMYLXWQnGvKQ+Q==\n"
      "-----END CERTIFICATE-----\n";

With verbose on I am still getting this error

Waiting on time sync...
checking wifi...Connecting...
Refreshing JWT
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 270168
[V][ssl_client.cpp:65] start_ssl_client(): Starting socket
[V][ssl_client.cpp:104] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:113] start_ssl_client(): Setting up the SSL/TLS structure...
[V][ssl_client.cpp:129] start_ssl_client(): Loading CA cert
[V][ssl_client.cpp:197] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:212] start_ssl_client(): Performing the SSL/TLS handshake...
[E][ssl_client.cpp:36] _handle_error(): [start_ssl_client():216]: (-9984) X509 - Certificate verification failed, e.g. CRL, CA or signature check failed
[E][WiFiClientSecure.cpp:133] connect(): start_ssl_client: -9984
[V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.
not connected
Settings incorrect or missing a cyper for SSL
Connect with mqtt.2030.ltsapis.goog:8883
ClientId: projects/my-projetc-name/locations/us-central1/registries/my-registry/devices/my-device-name
Waiting 60 seconds, retry will likely fail

@raphael-bmec-co
Copy link

raphael-bmec-co commented Apr 9, 2021

I am not sure that you have the correct certificate - I suspect you have the server certificate (again I may well be mistaken).

Can you try generate the certificates as follows:

/// Download root CAs here https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#downloading_mqtt_server_certificates:
/// Open in windows, select details tab, select Copy to File..., select Base-65 encoded X.509 (.CER).
/// Open resulting .cer files in text editor and copy contents. Ensure to terminate with \n.

@Yaaximus
Copy link
Author

Yaaximus commented Apr 10, 2021

I was able to connect to Cloud IOT Core while using root certificate by following these steps:

Download primary and secondary crt files from the following link:
(https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#downloading_mqtt_server_certificates)

Convert these '.crt' files in '.pem' file using the following command

openssl x509 -inform DER -in gtsltsr.crt -out primary.pem -text
openssl x509 -inform DER -in GSR4.crt -out secondary.pem -text

And then copying the content from both these file which end up looking something like this in ciotc_config.h file

const char *root_cert =

    "-----BEGIN CERTIFICATE-----\n"
    "MIIBxTCCAWugAwIBAgINAfD3nVndblD3QnNxUDAKBggqhkjOPQQDAjBEMQswCQYD\n"
    "VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzERMA8G\n"
    "A1UEAxMIR1RTIExUU1IwHhcNMTgxMTAxMDAwMDQyWhcNNDIxMTAxMDAwMDQyWjBE\n"
    "MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM\n"
    "QzERMA8GA1UEAxMIR1RTIExUU1IwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATN\n"
    "8YyO2u+yCQoZdwAkUNv5c3dokfULfrA6QJgFV2XMuENtQZIG5HUOS6jFn8f0ySlV\n"
    "eORCxqFyjDJyRn86d+Iko0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw\n"
    "AwEB/zAdBgNVHQ4EFgQUPv7/zFLrvzQ+PfNA0OQlsV+4u1IwCgYIKoZIzj0EAwID\n"
    "SAAwRQIhAPKuf/VtBHqGw3TUwUIq7TfaExp3bH7bjCBmVXJupT9FAiBr0SmCtsuk\n"
    "miGgpajjf/gFigGM34F9021bCWs1MbL0SA==\n"
    "-----END CERTIFICATE-----\n"
    "-----BEGIN CERTIFICATE-----\n"
    "MIIB4TCCAYegAwIBAgIRKjikHJYKBN5CsiilC+g0mAIwCgYIKoZIzj0EAwIwUDEk\n"
    "MCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBDQSAtIFI0MRMwEQYDVQQKEwpH\n"
    "bG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTEyMTExMzAwMDAwMFoX\n"
    "DTM4MDExOTAzMTQwN1owUDEkMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBD\n"
    "QSAtIFI0MRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWdu\n"
    "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEuMZ5049sJQ6fLjkZHAOkrprlOQcJ\n"
    "FspjsbmG+IpXwVfOQvpzofdlQv8ewQCybnMO/8ch5RikqtlxP6jUuc6MHaNCMEAw\n"
    "DgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFSwe61F\n"
    "uOJAf/sKbvu+M8k8o4TVMAoGCCqGSM49BAMCA0gAMEUCIQDckqGgE6bPA7DmxCGX\n"
    "kPoUVy0D7O48027KqGx2vKLeuwIgJ6iFJzWbVsaj8kfSt24bAgAXqmemFZHe+pTs\n"
    "ewv4n4Q=\n"
    "-----END CERTIFICATE-----\n";

And obviously setting root_cert in esp32-mqtt.h in line 116

void setupCloudIoT(){

  device = new CloudIoTCoreDevice(
      project_id, location, registry_id, device_id,
      private_key_str);

  setupWifi();
  
  netClient = new WiFiClientSecure();
  netClient->setCACert(root_cert);
  mqttClient = new MQTTClient(512);
  mqttClient->setOptions(180, true, 1000); // keepAlive, cleanSession, timeout
  mqtt = new CloudIoTCoreMqtt(mqttClient, netClient, device);
  mqtt->setUseLts(true);
  mqtt->startMQTT();
}

Serial Monitor output with core debug level set to verbose

Waiting on time sync...
checking wifi...Connecting...
Refreshing JWT
[V][ssl_client.cpp:60] start_ssl_client(): Free internal heap before TLS 270088
[V][ssl_client.cpp:66] start_ssl_client(): Starting socket
[V][ssl_client.cpp:105] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:114] start_ssl_client(): Setting up the SSL/TLS structure...
[V][ssl_client.cpp:130] start_ssl_client(): Loading CA cert
[V][ssl_client.cpp:198] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:213] start_ssl_client(): Performing the SSL/TLS handshake...
[V][ssl_client.cpp:234] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:243] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:258] start_ssl_client(): Free internal heap after TLS 230832
[V][ssl_client.cpp:296] send_ssl_data(): Writing HTTP request with 313 bytes...
connected

Library connected!

@raphael-bmec-co
Copy link

Fantastic! Have you verified that both the primary and backup certificates are working? Maybe by putting an error in each on in turn?

@Yaaximus
Copy link
Author

Yaaximus commented Apr 10, 2021

Still able to connect to Google Cloud IOT core even after putting an error in backup certificate but with error added in Primary certificate it fails to verify certificate.
What I was thinking that here (https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#downloading_mqtt_server_certificates)

Google's minimal root CA set (<1 KB) for mqtt.2030.ltsapis.goog. The minimal root CA set includes a primary and backup certificate.
    - This set is for devices with memory constraints, like microcontrollers, and establishes the chain of trust to communicate with Cloud IoT Core only.
    - Devices with the minimal root CA set communicate with the Cloud IOT Core via long-term support domains.
    - This set is fixed through 2030 (the primary and backup certificates won't change). For added security, Google Trust Services may switch between the primary and backup certificates at any time without notice.

It is mentioned that Google Trust Services may switch between the primary and backup certificate at any time means that its up to Google trust service to decide with which certificate its going to verify if the certificate is correct or not either primary or backup. Or should it work in any case if one of the certificate is correct?

@raphael-bmec-co
Copy link

Your point is very valid and google may only be accepting the primary at the moment. My concern is that I am not sure if you can concatenate both certificates into one string like that. Do you know? Maybe invert the order so that the backup is first and see if it still connects. In our implementation we manually change to the backup cert if the primary does not work. Your approach is much more elegant, provided it works.

@Yaaximus
Copy link
Author

Yaaximus commented Apr 10, 2021

Changed the order without putting error and it worked.
In the order changed basically backup certificate first, added error in backup certificate and it worked.
In the order changed added error in primary certificate and it did not worked.

I guess it can be safely be said that order does not matter.
I think there is no need to manually change to the backup cert if the primary does not work. Just concatenate both certificates int o one string without order and it should work.

@raphael-bmec-co
Copy link

Fantastic! Thanks for discovering that. I think it would be a good idea to leave this issue open as there is scope for improving the documentation related to this.

@Yaaximus
Copy link
Author

Your very much Welcome, Thanks for you guidance as well.
I was also thinking the same thing that if these changes should be made into this repository or have the developers of the repository intentionally not added root certificate verification for reasons not known to at-least me.

Sure lets keep this issue open. If required I can make a pull request to merge changes into this repository for Certificate verification with WifiClientSecure latest library.

@fabltd
Copy link

fabltd commented Apr 12, 2021

Hi

I have had a device working perfectly. I made an update and reuploaded and now run into the issue described. I guess the wifisecure lib changed with a core update.

The fix above however doesn't work.I get an error with this line:

netClient->setCACert(root_cert);

What am I missing?

Thanks

@raphael-bmec-co
Copy link

Please double check all the steps discussed above especially the need to cast the client.

If you don't come right, please share more information like your code and the error you are seeing.

@fabltd
Copy link

fabltd commented Apr 12, 2021

Hi

I know i am missing the casting step but I am confused where is ? Does all of the steps to enable inscure need to be done.

I don't know why this broke with the latest core update.

Is there a set of updated files with this fix?

@raphael-bmec-co
Copy link

It's broken on the latest update, because WifiClientSecure actually checks if the CA is being set and throws and exception if it isn't. You can disable this exception by setting the client to insecure or you can set the CA. Either of these options will require you to do the cast. The example code has a CA variable but it is never used.

I'm not sure I can help you with where to do the casting. @fabltd is better positioned to provide guidance here as we have actually stripped out layers of this library due to issues with the JWT generation.

@fabltd
Copy link

fabltd commented Apr 12, 2021

@raphael-bmec-co Thanks, Do you know what ESP core version they updated this in?

I have also replaced the JWT with a hardware element.

is this the cast?

WiFiClientSecure& wifiClientSecure= static_cast<WiFiClientSecure&>(netClient);

Does the get initalised in side the funtion or at declaration?

@raphael-bmec-co
Copy link

If you roll back to 1.0.4 the issue should go away.

That is the cast.

I don't know the answer to your last question offhand. Sorry.

@fabltd
Copy link

fabltd commented Apr 12, 2021

I just run into this issue:

error: invalid static_cast from type 'Client*' to type 'WiFiClientSecure&'
WiFiClientSecure &wifiClientSecure = static_cast<WiFiClientSecure &>(netClient);

Here is my code:

void setupCloudIoT(){
device = new CloudIoTCoreDevice(
project_id, location, registry_id, device_id,
private_key_str);

setupWifi();
netClient = new WiFiClientSecure();
WiFiClientSecure &wifiClientSecure = static_cast<WiFiClientSecure &>(netClient);
netClient->setCACert(root_cert);
mqttClient = new MQTTClient(512);
mqttClient->setOptions(180, true, 1000); // keepAlive, cleanSession, timeout
mqtt = new CloudIoTCoreMqtt(mqttClient, netClient, device);
mqtt->setUseLts(true);
mqtt->startMQTT();
}

@raphael-bmec-co
Copy link

You already declared netClient as type WiFiClientSecure so why are you trying to cast it?

@jmancillavalenzuela
Copy link

jmancillavalenzuela commented Apr 12, 2021

Hello sir, I have the same error!

Here is the debugger log (verbose)

18:56:02.194 -> Starting wifi
18:56:02.284 -> [D][WiFiGeneric.cpp:374] _eventCallback(): Event: 0 - WIFI_READY
18:56:02.284 -> [D][WiFiGeneric.cpp:374] _eventCallback(): Event: 2 - STA_START
18:56:02.284 -> Connecting to WiFi
18:56:04.464 -> [D][WiFiGeneric.cpp:374] _eventCallback(): Event: 4 - STA_CONNECTED
18:56:04.742 -> [D][WiFiGeneric.cpp:374] _eventCallback(): Event: 7 - STA_GOT_IP
18:56:04.787 -> [D][WiFiGeneric.cpp:419] _eventCallback(): STA IP: 192.168.1.145, MASK: 255.255.255.0, GW: 192.168.1.1
18:56:04.787 -> Waiting on time sync...
18:56:04.878 -> checking wifi...Connecting...
18:56:04.878 -> Refreshing JWT
18:56:05.387 -> [V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 274752
18:56:05.387 -> [V][ssl_client.cpp:65] start_ssl_client(): Starting socket
18:56:05.664 -> [V][ssl_client.cpp:104] start_ssl_client(): Seeding the random number generator
18:56:05.709 -> [V][ssl_client.cpp:113] start_ssl_client(): Setting up the SSL/TLS structure...
18:56:05.709 -> [V][ssl_client.cpp:129] start_ssl_client(): Loading CA cert
18:56:05.709 -> [V][ssl_client.cpp:197] start_ssl_client(): Setting hostname for TLS session...
18:56:05.709 -> [V][ssl_client.cpp:212] start_ssl_client(): Performing the SSL/TLS handshake...
18:56:09.368 -> [V][ssl_client.cpp:242] start_ssl_client(): Verifying peer X.509 certificate...
18:56:09.368 -> [V][ssl_client.cpp:251] start_ssl_client(): Certificate verified.
18:56:09.414 -> [V][ssl_client.cpp:266] start_ssl_client(): Free internal heap after TLS 235628
18:56:09.414 -> [V][ssl_client.cpp:304] send_ssl_data(): Writing HTTP request with 333 bytes...
18:56:09.691 -> [V][ssl_client.cpp:274] stop_ssl_socket(): Cleaning SSL connection.
18:56:09.691 -> not connected
18:56:09.691 -> Settings incorrect or missing a cyper for SSL
18:56:09.691 -> Connect with mqtt.2030.ltsapis.goog:8883
18:56:09.738 -> ClientId: projects/feisty-mechanic-310318/locations/us-central1/registries/iotcore-registry/devices/hs-1
18:56:09.738 -> Waiting 60 seconds, retry will likely fail

My code on esp32-mqtt.h:

void setupCloudIoT(){
device = new CloudIoTCoreDevice(
project_id, location, registry_id, device_id,
private_key_str);
setupWifi();
netClient = new WiFiClientSecure();
netClient->setCACert(root_cert);
mqttClient = new MQTTClient(512);
mqttClient->setOptions(180, true, 1000); // keepAlive, cleanSession, timeout
mqtt = new CloudIoTCoreMqtt(mqttClient, netClient, device);
mqtt->setUseLts(true);
mqtt->startMQTT();
}

the main.ino

void setup() {
// put your setup code here, to run once:
Serial.begin(115200);
pinMode(LED_BUILTIN, OUTPUT);
setupCloudIoT();
}

unsigned long lastMillis = 0;
void loop() {
mqtt->loop();
delay(10); // <- fixes some issues with WiFi stability

if (!mqttClient->connected()) {
connect();
}

// publish a message roughly every second.
if (millis() - lastMillis > 1000) {
lastMillis = millis();
publishTelemetry("Hello123");
Serial.println("datos enviados");
}
}

Im trying to connect to my config, I create the cert, my Pub/Sub is working (tested in node), but ESP32 can't connect it.

@Yaaximus
Copy link
Author

@jmancillavalenzuela I can see that your certificate is verified so you probably have no issue on certificate side. Please make sure your private key is correct and please double check your other credentials.

@Yaaximus
Copy link
Author

Yaaximus commented Apr 14, 2021

I just run into this issue:

error: invalid static_cast from type 'Client*' to type 'WiFiClientSecure&'
WiFiClientSecure &wifiClientSecure = static_cast<WiFiClientSecure &>(netClient);

Here is my code:

void setupCloudIoT(){
device = new CloudIoTCoreDevice(
project_id, location, registry_id, device_id,
private_key_str);

setupWifi();
netClient = new WiFiClientSecure();
WiFiClientSecure &wifiClientSecure = static_cast<WiFiClientSecure &>(netClient);
netClient->setCACert(root_cert);
mqttClient = new MQTTClient(512);
mqttClient->setOptions(180, true, 1000); // keepAlive, cleanSession, timeout
mqtt = new CloudIoTCoreMqtt(mqttClient, netClient, device);
mqtt->setUseLts(true);
mqtt->startMQTT();
}

Please read all the steps mentioned above you have to make changes in other files as well like CloudIoTCoreMqtt.h and CloudIoTCoreMqtt.cpp for the cast to work, because the Client class type is declared there as well. Please read this
#221 (comment)

@Alpha018
Copy link

Alpha018 commented Apr 14, 2021

@Yaaximus Sorry but I have a question, if I use the function setInsecure the connection is insecure? and if I want to use a secure connection? how can I do?

@Yaaximus
Copy link
Author

@Alpha018 If you use "setInsecure" the you are ignoring Certificate verification but you are still using public and private key to establish a connection to Google Cloud IOT Core so there is a level of security.

Incase you want to use Certificate the you will have to follow the steps mentioned above

Initially you will have to Cast netClient as WiFiClientSecure
#221 (comment)

Then you will have to get cetificate and add the certificate in variable root_cert in the file ciot_config.h
#221 (comment)

@jmancillavalenzuela
Copy link

Hi, Could you share a code example for this please? at least the conection

@warshaa
Copy link

warshaa commented Apr 29, 2021

I'm getting this error

checking wifi...Connecting...
Refreshing JWT
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 274920
[V][ssl_client.cpp:65] start_ssl_client(): Starting socket
[V][ssl_client.cpp:104] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:113] start_ssl_client(): Setting up the SSL/TLS structure...
[V][ssl_client.cpp:129] start_ssl_client(): Loading CA cert
[V][ssl_client.cpp:197] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:212] start_ssl_client(): Performing the SSL/TLS handshake...
[V][ssl_client.cpp:233] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:242] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:257] start_ssl_client(): Free internal heap after TLS 235796
[V][ssl_client.cpp:295] send_ssl_data(): Writing HTTP request with 305 bytes...
[V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.
not connected
Settings incorrect or missing a cyper for SSL
Connect with mqtt.2030.ltsapis.goog:8883
ClientId: projects/esp8266-298816/locations/europe-west1/registries/Libya/devices/LBY002
Waiting 60 seconds, retry will likely fail

@Yaaximus
Copy link
Author

Yaaximus commented Apr 29, 2021

Please make sure your private key is correct and please double check your other credentials in ciot_config.h file.

Sometimes this can also happen if the internet connections is poor.

And if you are using ESP8266 then please try the example ESP8266-lwmqtt

@Alpha018
Copy link

@warshaa check your jwt token before send the request.

@warshaa
Copy link

warshaa commented Apr 29, 2021

@warshaa check your jwt token before send the request.

already checked, nothing wrong with the JWT
after letting the device plugged in for long time I noticed it makes successful connection randomly out of many unsuccessful attempts.

@lanekatris
Copy link

Fix worked great for me: #221 (comment)
Only change was the cast folks have been mentioning: ((WiFiClientSecure*)netClient)->setCACert(root_cert);

@CHeffernan087
Copy link

Your very much Welcome, Thanks for you guidance as well.
I was also thinking the same thing that if these changes should be made into this repository or have the developers of the repository intentionally not added root certificate verification for reasons not known to at-least me.

Sure lets keep this issue open. If required I can make a pull request to merge changes into this repository for Certificate verification with WifiClientSecure latest library.

Fantastic. I had been stuck on this for a couple of days. Fix works great for me. Thanks guys ❤️

@mark-antal-csizmadia
Copy link

Fix worked great for me: #221 (comment)
Only change was the cast folks have been mentioning: ((WiFiClientSecure*)netClient)->setCACert(root_cert);

This works! Thanks a lot for figuring it out! Fyi, at the time of solving it, I was using Google Cloud IoT Core JWT library version 1.1.11 and MQTT (by Joel Gaehwiler) library version 2.4.7

@netskink
Copy link

Hmm. I came here because i have the same error. I notice I'm missing the line of code where SetCACert() is missing. I added the code mention in above. I can not compile the line for SetCACert(). I look in the WiFi101 library and notice that method is missing. I backroll to version 2.4.7 for MQTT. The Google Cloud IoT core JWT lib I am using is 1.1.11 as mentioned above. I realize MQTT is not the library defining setcacert but no worries. I'm just trying to match lib versions.

I also tried to typecast as shown but I don't have a WiFiClientSecure class. Not sure where to get it.

@nicosandller
Copy link

@netskink in case you didn't solve this yet the solution that worked for me is to add the second line in this following code:

netClient = new WiFiClientSecure();
((WiFiClientSecure *)netClient)->setCACert(root_cert);

on the esp32-mqtt.h file (lone 114 aprox.

Instead of netClient->setCACert(root_cert); as suggested early on this issue.

Hope that works for you!

@netskink
Copy link

netskink commented Aug 9, 2021

Hello @nicosandller I have not tried this yet. I will. FWIW, I opened a new issue and did a follow-up here. I'm not a esp32/esp8266 user but it seems like a similar issue. I did post this to show what I've noticed via wireshark up to this point https://github.com/netskink/ssl_publy

I'm using two different code samples. One is this one which seems to use the code to hard-code the private key as a series of bytes. The other sample uses the crypto chip for key management. They both do not connect.

I do not currently understand how either works. I'm still working on it. Hence I am so happy to have your assistance. Many thanks!

@netskink
Copy link

Hmm, @nicosandller comparing the esp32 example and MKR1000 sample (i'm using this one), the source is different. I don't have the code you reference. The code i am using uses WiFiSSLClient.h.

Hmm, I'm looking for a routine similar to setCaRoot() in this class hierarchy. It looks like the examples use different stacks. Different processors imply different stacks. So no big deal.

Starting from example code.

esp32 example has:

#include <WiFiClientSecure.h>

mkr1000 example has:

#include <WiFiSSLClient.h>

#include <WiFiSSLClient.h>
This class is in WiFi101 library and inherits from WiFiClient. Looking at the class it has a constructor and two different parameter signature connect() routines. Nothing like setCARoot(). Both of the connect routines use WiFiClient::connectSSL(). So they are just wrappers around an existing class with connectSSL() routines? LOL, why? Perhaps they were thinking some future mods could be here? As it is now though, its a wrapper.

Anyway, looking at the WiFiClient.cpp which is the base class and also in WiFi101, is the connectSSL() implementation and its a connect call from the WiFiClient base class Client connect() with SSL socket flag set.

I did not find the Client base class in any of the Arduino/Libraries folders, so I assume its comming from arduino-1.8.3, which I believe is in arduino-1.8.3/hardware/arduino/avr/cores/arduino/Client.h. However, that file defines Client::connect() with just two parameters, host and port. It does not have the option of a socket flags. I'm still learning arduino so I don't where the Client base class its using is defined.

$ find . -name "*.h" | xargs grep "class WiFiClient" 
./WiFiClient.h:class WiFiClient : public Client {
./WiFiServer.h:class WiFiClient;

Doing the same but looking for 'class Client, has no results. Its funny because it literraly says Include <Client.h>so I simply did a search forClient.h` as a file name and the only hit I had was the one in avr directory. I also don't think that would match the mkr1000 since its a SAMD.

The search continues. Perhaps this needs to be moved to WiFi101.

@Hiroya-W
Copy link
Contributor

Hi. I encountered the same problem with ESP32, and #221 (comment) helped me solve it. Thanks for sharing.

@matteocordray
Copy link

I was able to connect to Cloud IOT Core while using root certificate by following these steps:

Download primary and secondary crt files from the following link: (https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#downloading_mqtt_server_certificates)

Convert these '.crt' files in '.pem' file using the following command

openssl x509 -inform DER -in gtsltsr.crt -out primary.pem -text
openssl x509 -inform DER -in GSR4.crt -out secondary.pem -text

And then copying the content from both these file which end up looking something like this in ciotc_config.h file

const char *root_cert =

    "-----BEGIN CERTIFICATE-----\n"
    "MIIBxTCCAWugAwIBAgINAfD3nVndblD3QnNxUDAKBggqhkjOPQQDAjBEMQswCQYD\n"
    "VQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzERMA8G\n"
    "A1UEAxMIR1RTIExUU1IwHhcNMTgxMTAxMDAwMDQyWhcNNDIxMTAxMDAwMDQyWjBE\n"
    "MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM\n"
    "QzERMA8GA1UEAxMIR1RTIExUU1IwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATN\n"
    "8YyO2u+yCQoZdwAkUNv5c3dokfULfrA6QJgFV2XMuENtQZIG5HUOS6jFn8f0ySlV\n"
    "eORCxqFyjDJyRn86d+Iko0IwQDAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw\n"
    "AwEB/zAdBgNVHQ4EFgQUPv7/zFLrvzQ+PfNA0OQlsV+4u1IwCgYIKoZIzj0EAwID\n"
    "SAAwRQIhAPKuf/VtBHqGw3TUwUIq7TfaExp3bH7bjCBmVXJupT9FAiBr0SmCtsuk\n"
    "miGgpajjf/gFigGM34F9021bCWs1MbL0SA==\n"
    "-----END CERTIFICATE-----\n"
    "-----BEGIN CERTIFICATE-----\n"
    "MIIB4TCCAYegAwIBAgIRKjikHJYKBN5CsiilC+g0mAIwCgYIKoZIzj0EAwIwUDEk\n"
    "MCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBDQSAtIFI0MRMwEQYDVQQKEwpH\n"
    "bG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTEyMTExMzAwMDAwMFoX\n"
    "DTM4MDExOTAzMTQwN1owUDEkMCIGA1UECxMbR2xvYmFsU2lnbiBFQ0MgUm9vdCBD\n"
    "QSAtIFI0MRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWdu\n"
    "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEuMZ5049sJQ6fLjkZHAOkrprlOQcJ\n"
    "FspjsbmG+IpXwVfOQvpzofdlQv8ewQCybnMO/8ch5RikqtlxP6jUuc6MHaNCMEAw\n"
    "DgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFFSwe61F\n"
    "uOJAf/sKbvu+M8k8o4TVMAoGCCqGSM49BAMCA0gAMEUCIQDckqGgE6bPA7DmxCGX\n"
    "kPoUVy0D7O48027KqGx2vKLeuwIgJ6iFJzWbVsaj8kfSt24bAgAXqmemFZHe+pTs\n"
    "ewv4n4Q=\n"
    "-----END CERTIFICATE-----\n";

And obviously setting root_cert in esp32-mqtt.h in line 116

void setupCloudIoT(){

  device = new CloudIoTCoreDevice(
      project_id, location, registry_id, device_id,
      private_key_str);

  setupWifi();
  
  netClient = new WiFiClientSecure();
  netClient->setCACert(root_cert);
  mqttClient = new MQTTClient(512);
  mqttClient->setOptions(180, true, 1000); // keepAlive, cleanSession, timeout
  mqtt = new CloudIoTCoreMqtt(mqttClient, netClient, device);
  mqtt->setUseLts(true);
  mqtt->startMQTT();
}

Serial Monitor output with core debug level set to verbose

Waiting on time sync...
checking wifi...Connecting...
Refreshing JWT
[V][ssl_client.cpp:60] start_ssl_client(): Free internal heap before TLS 270088
[V][ssl_client.cpp:66] start_ssl_client(): Starting socket
[V][ssl_client.cpp:105] start_ssl_client(): Seeding the random number generator
[V][ssl_client.cpp:114] start_ssl_client(): Setting up the SSL/TLS structure...
[V][ssl_client.cpp:130] start_ssl_client(): Loading CA cert
[V][ssl_client.cpp:198] start_ssl_client(): Setting hostname for TLS session...
[V][ssl_client.cpp:213] start_ssl_client(): Performing the SSL/TLS handshake...
[V][ssl_client.cpp:234] start_ssl_client(): Verifying peer X.509 certificate...
[V][ssl_client.cpp:243] start_ssl_client(): Certificate verified.
[V][ssl_client.cpp:258] start_ssl_client(): Free internal heap after TLS 230832
[V][ssl_client.cpp:296] send_ssl_data(): Writing HTTP request with 313 bytes...
connected

Library connected!

This was able to connect to the library using an ESP32-C3. Thank you so much!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests