letsencrypt tls server cert error: unable to verify the first certificate

[jbanyer]

Hi @patrickdemers6 . I've followed your guide and generated a cert for mTLS using certbot / Let's Encrypt. The certificate has been delivered and loaded by the fleet-telemetry server however it fails the first step of the check_server_cert.sh tool.

When running this step manually I get the following output:

jay@Jays-MacBook-Pro fleet-telemetry % openssl s_client -connect "tesla.chqtest.net:443" -servername "tesla.chqtest.net" -showcerts 2>/dev/null            
CONNECTED(00000005)
---
Certificate chain
 0 s:CN=tesla.chqtest.net
   i:C=US, O=Let's Encrypt, CN=R3
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar  2 00:41:56 2024 GMT; NotAfter: May 31 00:41:55 2024 GMT
-----BEGIN CERTIFICATE-----
MIIEJDCCAwygAwIBAgISAzTarMvBl9d/8+FWEIMiGfb8MA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yNDAzMDIwMDQxNTZaFw0yNDA1MzEwMDQxNTVaMBwxGjAYBgNVBAMT
EXRlc2xhLmNocXRlc3QubmV0MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeFFA
gnqtpE3oXAS0pZsJdw3xIpneD0ISiwOT1KnpMxeha2nQTlmuUrS2YAqmxJ4H+qfm
ExUlTyKjyuq1NZIIEKOCAhMwggIPMA4GA1UdDwEB/wQEAwIHgDAdBgNVHSUEFjAU
BggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQU/0kK
h0lZN1Q3PI6zMwgD97zw2NAwHwYDVR0jBBgwFoAUFC6zF7dYVsuuUAlA5h+vnYsU
wsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRwOi8vcjMuby5sZW5j
ci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNyLm9yZy8wHAYDVR0R
BBUwE4IRdGVzbGEuY2hxdGVzdC5uZXQwEwYDVR0gBAwwCjAIBgZngQwBAgEwggEE
BgorBgEEAdZ5AgQCBIH1BIHyAPAAdQA7U3d1Pi25gE6LMFsG/kA7Z9hPw/THvQAN
LXJv4frUFwAAAY380zQEAAAEAwBGMEQCIHiQSBd4Uh3om8ODlB8EVD2L378nihTI
ZDdpNtVPhSwWAiBAiiRcU5m7aMHygQ6HIYJXGw/0MrEGGu76OF3u7ba0WQB3AHb/
iD8KtvuVUcJhzPWHujS0pM27KdxoQgqf5mdMWjp0AAABjfzTNAoAAAQDAEgwRgIh
APD34ni4V8lkkUlD7COnM/tR6GIIb7M+1/74EylzChMoAiEArda8ZwkKtavOOfWf
3oCy8Iam3TzbWTjix8u6tbFRFxQwDQYJKoZIhvcNAQELBQADggEBAEJywcL+IQES
geQLSNK2MV7Ib+HVgv13Yoarg+JckpPJ7wgn+JDmZGgeJnwBg5BJkb8XOKPJF9q0
ryyGTOemXceeskiKiweV8QNYOBnJiwU2kNH6D+O8ZBSG+0f+XmvZTJ0z/NcwzOTN
WvyJjO+I6xw9II0750HcoUQVJWvSHQVotZUlOCumr2+eqLsXk1Ki4XsVX5AcG1IN
fK4MuZGk6uJijGK881Dt22jCVP7N0ZUuUDFFXJ5zzB54+Dzt2wj/AQS1sd9ZBXz8
eKvCYKJIW8KF7HE2E7C9rlvxtYpqh2z6DcZaumOMMckktyvJIbWkTgAYqk7RAuJT
hwsonBfIV1A=
-----END CERTIFICATE-----
---
Server certificate
subject=CN=tesla.chqtest.net
issuer=C=US, O=Let's Encrypt, CN=R3
---
Acceptable client certificate CA names
CN=Tesla Issuing CA, O=Tesla Motors, L=Palo Alto, ST=California, C=US
CN=Tesla Motors GF Austin Product Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors GF Berlin Product Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors GF0 Product Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors GF3 Product Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors GF3 Product RSA Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors Product Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors Product RSA Issuing CA, OU=Motors, OU=PKI, O=Tesla Inc., C=US
CN=Tesla Motors Products CA
CN=Tesla Motors Root CA
CN=Tesla Policy CA, O=Tesla Motors, L=Palo Alto, ST=California, C=US
CN=Tesla Product RSA Root CA, OU=PKI, O=Tesla, C=US
CN=Tesla Product Root CA, OU=PKI, O=Tesla, C=US
CN=Tesla Root CA, O=Tesla Motors, L=Palo Alto, ST=California, C=US
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2865 bytes and written 419 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---

The exact same output is produced on a Linux server.

The fleet telemetry server is running at tesla.chqtest.net:443 so you can confirm for yourself. Server is dockerhub image tesla/fleet-telemetry:v0.1.11

Any help would be appreciated! Cheers.

[jbanyer]

Here is the cert and CA chains, and a dump using openssl:
cert_ca_bundle.zip

$ openssl x509 -in keys/0000_cert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:34:da:ac:cb:c1:97:d7:7f:f3:e1:56:10:83:22:19:f6:fc
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Mar  2 00:41:56 2024 GMT
            Not After : May 31 00:41:55 2024 GMT
        Subject: CN = tesla.chqtest.net
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:78:51:40:82:7a:ad:a4:4d:e8:5c:04:b4:a5:9b:
                    09:77:0d:f1:22:99:de:0f:42:12:8b:03:93:d4:a9:
                    e9:33:17:a1:6b:69:d0:4e:59:ae:52:b4:b6:60:0a:
                    a6:c4:9e:07:fa:a7:e6:13:15:25:4f:22:a3:ca:ea:
                    b5:35:92:08:10
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                FF:49:0A:87:49:59:37:54:37:3C:8E:B3:33:08:03:F7:BC:F0:D8:D0
            X509v3 Authority Key Identifier: 
                14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
            Authority Information Access: 
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/
            X509v3 Subject Alternative Name: 
                DNS:tesla.chqtest.net
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
                                67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
                    Timestamp : Mar  2 01:41:56.356 2024 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:78:90:48:17:78:52:1D:E8:9B:C3:83:94:
                                1F:04:54:3D:8B:DF:BF:27:8A:14:C8:64:37:69:36:D5:
                                4F:85:2C:16:02:20:40:8A:24:5C:53:99:BB:68:C1:F2:
                                81:0E:87:21:82:57:1B:0F:F4:32:B1:06:1A:EE:FA:38:
                                5D:EE:ED:B6:B4:59
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 76:FF:88:3F:0A:B6:FB:95:51:C2:61:CC:F5:87:BA:34:
                                B4:A4:CD:BB:29:DC:68:42:0A:9F:E6:67:4C:5A:3A:74
                    Timestamp : Mar  2 01:41:56.362 2024 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:F0:F7:E2:78:B8:57:C9:64:91:49:43:
                                EC:23:A7:33:FB:51:E8:62:08:6F:B3:3E:D7:FE:F8:13:
                                29:73:0A:13:28:02:21:00:AD:D6:BC:67:09:0A:B5:AB:
                                CE:39:F5:9F:DE:80:B2:F0:86:A6:DD:3C:DB:59:38:E2:
                                C7:CB:BA:B5:B1:51:17:14
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        42:72:c1:c2:fe:21:01:12:81:e4:0b:48:d2:b6:31:5e:c8:6f:
        e1:d5:82:fd:77:62:86:ab:83:e2:5c:92:93:c9:ef:08:27:f8:
        90:e6:64:68:1e:26:7c:01:83:90:49:91:bf:17:38:a3:c9:17:
        da:b4:af:2c:86:4c:e7:a6:5d:c7:9e:b2:48:8a:8b:07:95:f1:
        03:58:38:19:c9:8b:05:36:90:d1:fa:0f:e3:bc:64:14:86:fb:
        47:fe:5e:6b:d9:4c:9d:33:fc:d7:30:cc:e4:cd:5a:fc:89:8c:
        ef:88:eb:1c:3d:20:8d:3b:e7:41:dc:a1:44:15:25:6b:d2:1d:
        05:68:b5:95:25:38:2b:a6:af:6f:9e:a8:bb:17:93:52:a2:e1:
        7b:15:5f:90:1c:1b:52:0d:7c:ae:0c:b9:91:a4:ea:e2:62:8c:
        62:bc:f3:50:ed:db:68:c2:54:fe:cd:d1:95:2e:50:31:45:5c:
        9e:73:cc:1e:78:f8:3c:ed:db:08:ff:01:04:b5:b1:df:59:05:
        7c:fc:78:ab:c2:60:a2:48:5b:c2:85:ec:71:36:13:b0:bd:ae:
        5b:f1:b5:8a:6a:87:6c:fa:0d:c6:5a:ba:63:8c:31:c9:24:b7:
        2b:c9:21:b5:a4:4e:00:18:aa:4e:d1:02:e2:53:87:0b:28:9c:
        17:c8:57:50

[jbanyer]

I've tried multiple certificates, and also tried using a subdomain telemetry.tesla.chqtest.net and a new matching CSR. The result is the same error with the cert.

[patrickdemers6]

Hmm, can you also share output from certbot?

[patrickdemers6]

Also have you tried sending the configuration to a vehicle? Just curious if it will still work or not.

[jbanyer]

Hmm, can you also share output from certbot?

Here's the certbot output. This example is when I used a subdomain telemetry.tesla.chqtest.net, with its own CSR.
telemetry.tesla.chqtest.net.csr.zip

$ sudo certbot certonly -d telemetry.tesla.chqtest.net --csr telemetry.tesla.chqtest.net.csr
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin (apache)
2: Runs an HTTP server locally which serves the necessary validation files under
the /.well-known/acme-challenge/ request path. Suitable if there is no HTTP
server already running. HTTP challenge only (wildcards not supported).
(standalone)
3: Saves the necessary validation files to a .well-known/acme-challenge/
directory within the nominated webroot path. A seperate HTTP server must be
running and serving files from the webroot path. HTTP challenge only (wildcards
not supported). (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1
Requesting a certificate for telemetry.tesla.chqtest.net

Successfully received certificate.
Certificate is saved at:            /home/ubuntu/tesla-fleet-telemetry/0000_cert.pem
Intermediate CA chain is saved at:  /home/ubuntu/tesla-fleet-telemetry/0000_chain.pem
Full certificate chain is saved at: /home/ubuntu/tesla-fleet-telemetry/0001_chain.pem
This certificate expires on 2024-05-30.

NEXT STEPS:
- Certificates created using --csr will not be renewed automatically by Certbot. You will need to renew the certificate before it expires, by running the same Certbot command again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - --

[jbanyer]

In case it matters: my fleet telemetry server is running on AWS behind a Network Load Balancer which forwards port 443 as TCP to the fleet telemetry server. AFAIK that should support mTLS.

[jbanyer]

Also have you tried sending the configuration to a vehicle? Just curious if it will still work or not.

@patrickdemers6 actually I just tried, and it's working! The fleet-telemetry server is receiving telemetry from my vehicle, despite this error message when checking the cert.

[PrriyaR]

@patrickdemers6 I did not get the certificate from LetsEncrypt and I already have a domain and cert is issued by AWS. I used the certificate chain and domain cert.

When i try to start the server using docker-compose up i get the following error

$ docker-compose up
[+] Running 1/0
 ✔ Container fleet-telemetry-app-1  Created                                                                                                                                                0.0s
Attaching to app-1
app-1  | 2024/03/12 21:13:06 maxprocs: Leaving GOMAXPROCS=1: CPU quota undefined
app-1  | time="2024-03-12T21:13:06Z" level=info msg=config_skipping_empty_metrics_provider
app-1  | time="2024-03-12T21:13:06Z" level=info msg=starting
app-1  | panic: open /home/ec2-user/teslatelemetry/fleet-telemetry/tools/certs/server.crt: no such file or directory
app-1  |
app-1  | goroutine 1 [running]:
app-1  | main.main()
app-1  |        /go/src/fleet-telemetry/cmd/main.go:36 +0x73
app-1 exited with code 2

But i do have the cert files in that location. Any idea what could be the issue?

Here is my config file:

{
  "host": "0.0.0.0",
  "hostname": "<domainName>,
  "port": 443,
  "log_level": "debug",
  "json_log_enable": true,
  "namespace": "telemetry",
  "reliable_ack": false,
  "rate_limit": {
    "enabled": false,
    "message_limit": 100
  },
  "records": {
    "alerts": [
        "logger"
    ],
    "errors": [
        "logger"
    ],
    "V": [
          "logger"
      ]
  },
  "tls": {
    "server_cert": "/home/ec2-user/teslatelemetry/fleet-telemetry/tools/certs/server.crt",
    "server_key": "/home/ec2-user/teslatelemetry/fleet-telemetry/tools/private_key.pem"
  },
  "ca": "-----BEGIN CERTIFICATE-----\n"
}

[patrickdemers6]

Can you include the docker-compose file you're using? My hunch is you don't have a volume mounted at the proper path in the container.

[PrriyaR]

Here is the docker-compose.yml file:

`version: '3.8'

services:
  app:
    build:
      context: ./repo
    ports:
      - 0.0.0.0:443:443
    volumes:
      - /home/ec2-user/teslatelemetry/fleetfiles/certs:/config
      - /home/ec2-user/teslatelemetry/fleetfiles/config.json:/etc/fleet-telemetry/config.json
`

[jbanyer]

@PrriyaR may I humbly suggest that you use another ticket or method to request assistance so that this ticket can be used to track the original issue, which is that the check_server_cert.sh tool is throwing an error on a Let's Encrypt certificate, despite the certificate working. Thanks!

[PrriyaR]

Sure, I will move my comments out.

[amirhmk]

@jbanyer Did you ever figure out the issue with the check_server_cert.sh tool? I have a very similar setup to you, and I'm getting a similar error as well.

Will try issuing commands too now, but wasn't sure if my setup was correct so far.