SSL/TLS certificate is invalid on 1 server (50%)

[andrewsf]

❯ dig +short tldp.org A

152.19.134.151
152.19.134.152

These 2 IP addresses are not serving the same TLS certificate. One of them is invalid for the domain name tldp.org.
HSTS is on, so users are not allowed to work around this problem using http://.

❯ for ip in {151..152}; do echo; echo $ip; echo | openssl s_client -connect 152.19.134.$ip:443 -servername tldp.org 2>/dev/null | openssl x509 -noout -text | grep 'DNS:'; done

151
                DNS:jazz1.tldp.org, DNS:*.jazz1.tldp.org

152
                DNS:drone.tldp.org, DNS:en.tldp.org, DNS:git.tldp.org, DNS:infra1.tldp.org, DNS:infra2.tldp.org, DNS:jazz1.tldp.org, DNS:jazz2.tldp.org, DNS:lists.tldp.org, DNS:tldp.net, DNS:tldp.org, DNS:wiki.tldp.org, DNS:www.tldp.net, DNS:www.tldp.org

❯ diff -y \
    <(echo | openssl s_client -connect 152.19.134.151:443 -servername tldp.org 2>/dev/null) \
    <(echo | openssl s_client -connect 152.19.134.152:443 -servername tldp.org 2>/dev/null)
    
CONNECTED(00000003)						CONNECTED(00000003)
---								---
Certificate chain						Certificate chain
 0 s:/CN=jazz1.tldp.org						 0 s:/CN=jazz1.tldp.org
   i:/O=Random/OU=Domain CA				      |	   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/O=Random/OU=Domain CA				      |	 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/O=Random Certificate Authority			      |	   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---								---
Server certificate						Server certificate
-----BEGIN CERTIFICATE-----					-----BEGIN CERTIFICATE-----
MIIFVTCCAz2gAwIBAgIRAOJ87B7qsDU/WZE/DKeTQCowDQYJKoZIhvcNAQELB |	MIIF0DCCBLigAwIBAgISBA7yYfVlJBVnf57u4nYyH3/DMA0GCSqGSIb3DQEBC
JTEPMA0GA1UECgwGUmFuZG9tMRIwEAYDVQQLDAlEb21haW4gQ0EwHhcNMjEwM |	MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDV
MTEwNzEyWhcNMjQwMjA5MTEwNzEyWjAZMRcwFQYDVQQDEw5qYXp6MS50bGRwL |	EwJSMzAeFw0yMTAyMDkxMDA3MDRaFw0yMTA1MTAxMDA3MDRaMBkxFzAVBgNVB
ZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJm0LSWSFT+1IYgUz |	DmphenoxLnRsZHAub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCA
iX7SVPsFM0IetLvb7K5yW9BVtx3uTButmACgbiAnOt1pIwAZgOgBuTYAg8DMo |	mbQtJZIVP7UhiBTOArKJftJU+wUzQh60u9vsrnJb0FW3He5MG62YAKBuICc63
3SW4o/W6knZr6tR70VDey6DsNpVhArYm4a/cn+Ox6LyGb01C7Uo200z0pI69B |	ABmA6AG5NgCDwMyi+hvdJbij9bqSdmvq1HvRUN7LoOw2lWECtibhr9yf47Hov
NSbJuJY4DtPzkawvGwArZRlYrDL2zAT0uM/Mufi8eKyfzO3o5ndidIlYZRevH |	TULtSjbTTPSkjr0GpJc1Jsm4ljgO0/ORrC8bACtlGVisMvbMBPS4z8y5+Lx4r
wce0ZcN7MyVMSsuT8jh8TbeRMfe/erk7bdUB/636sGtqxTeZPw8DQ1uvDTYt5 |	7ejmd2J0iVhlF68e+tXBx7Rlw3szJUxKy5PyOHxNt5Ex9796uTtt1QH/rfqwa
YsHjDo1LznEGP7ws7XfGSFq70HHxNaqwS+QHobRgXZujeARRHSgvUDrSPdwxR |	N5k/DwNDW68NNi3lvJ1iweMOjUvOcQY/vCztd8ZIWrvQcfE1qrBL5AehtGBdm
o5kCAwEAAaOCAYowggGGMGkGCCsGAQUFBwEBBF0wWzArBggrBgEFBQcwAoYfa |	BFEdKC9QOtI93DFF5JGjmQIDAQABo4IC9zCCAvMwDgYDVR0PAQH/BAQDAgWgM
cDovL2RvbWFpbi1jYS5yYW5kb20ucmUvY3J0LzAsBggrBgEFBQcwAYYgaHR0c |	A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA
L2RvbWFpbi1jYS5yYW5kb20ucmUvb2NzcC8wXgYDVR0jBFcwVYAUlLwm21QcQ |	DgQWBBRQqgd8nSNoIex6UYIi+bpX0jpauzAfBgNVHSMEGDAWgBQULrMXt1hWy
8iQA25uZdP9HsOGhK6QpMCcxJTAjBgNVBAoMHFJhbmRvbSBDZXJ0aWZpY2F0Z |	CUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6L
dXRob3JpdHmCECKXIhGA1QPmsRfpuMcBGfAwDAYDVR0TAQH/BAIwADAwBgNVH |	My5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iub
KTAnMCWgI6Ahhh9odHRwOi8vZG9tYWluLWNhLnJhbmRvbS5yZS9jcmwvMB0GA |	LzCBxwYDVR0RBIG/MIG8gg5kcm9uZS50bGRwLm9yZ4ILZW4udGxkcC5vcmeCD
JQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAOBgNVHQ8BAf8EBAMCBaAwHQYDV |	dC50bGRwLm9yZ4IPaW5mcmExLnRsZHAub3Jngg9pbmZyYTIudGxkcC5vcmeCD
BBYEFFCqB3ydI2gh7HpRgiL5ulfSOlq7MCsGA1UdEQQkMCKCDmphenoxLnRsZ |	enoxLnRsZHAub3Jngg5qYXp6Mi50bGRwLm9yZ4IObGlzdHMudGxkcC5vcmeCC
b3JnghAqLmphenoxLnRsZHAub3JnMA0GCSqGSIb3DQEBCwUAA4ICAQDPiTHpv |	ZHAubmV0ggh0bGRwLm9yZ4INd2lraS50bGRwLm9yZ4IMd3d3LnRsZHAubmV0g
V0TMTu1uinvdiFT/0DRA5AB/D0tnhao9Vky5fi7AMNZdYjtcaoWosR7HvC23M |	d3cudGxkcC5vcmcwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBA
GAaUJKVtEpxkmz25pggjqSEVL0JBGWj0QWBZsHZcD1+VkwjMnVl6N91hvAGJX |	KDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEDB
XTDqNBzmqiyLTlJefr5E/jiIqxxvBA/qLxvXzMhUThqLcOf6iwqbcYFs0CDMX |	BgEEAdZ5AgQCBIH0BIHxAO8AdQBc3EOS/uarRUSxXprUVuYQN/vV+kfcoXOUs
O25BsqkJwCKrX68WmAKSh4pXg1aGQGyd4ksz2eS8cT7AHNmbEAGAgXqhlWmfs |	9scOygAAAXeGeSxRAAAEAwBGMEQCIEMFsNKzixIFoJeQbiVf1dG2A8Z2Ek8af
EOsvk8pNcTXBoOXq+O1g4mA3FaLGFwXk1sVpCUB9DGE47uzpVH93LSKRhQyD/ |	fhx8WqdRAiA8R+jcJyHUtW4pBxaKYqUwHZjd41aGxxGOp/yoWZ5mdAB2AH0+8
EmkBFfLkV8PPQyhzLcmtEyjSmMbpR7iFd4H6MQnbevKx996MHL6r4mHWPVmm2 |	/4hVaCTCwMqeUol5K8UOeAl/LmqXaJl+IvDXAAABd4Z5LLIAAAQDAEcwRQIhA
CbiiStBVQFH5iQcwuH9IXW7AaS1AmthPcEeneKcmK6uQyTOlEw+uXH8ssIkN/ |	Bn1xFQUs51Q33DEI3F3xs5WqPqvt/lGtMYFHJms3AiBaovhQJrLCHj7VVpgQw
Bjg3y9/8+xITcwkHJsVKSHYoEF4ubDVRTCAE5DwLNmr8M+dMCZ1zah2kRmA9S |	gn9jEgNU1ZJtbCEAWT5iEjANBgkqhkiG9w0BAQsFAAOCAQEAYBF6qjDiOY6lp
Nex7LycBcEMlJSyyVknBYy0fzNJGGxjJchRd1aqW9694rRl0vOqWLoBvT4k1L |	/59cxrkOElKznYXzpAdkEiO1qO6t0Z4fjkr7SGKCCwDHHRf7Tdw5FDgwvzp1r
zEMsVUcvmLK/Xfq3YtAzCO19dyX8VlxtTatH7dx5Fc6YZ4naWUITDiGbxcJbv |	Cah+hfU2stzj0+tbtd6Hyiq7LyDtLmenztglLmRSpsJUSJ08zamVfL9VLM5lH
JHftcAOMNygL/jDX2R3XWgiAohn2bgss7g==			      |	rfPKjSDsNAqSvl14GSkXF4jfp+CXrr4BM1ZPtCoSLstYOv1DzWPwQwvcNaA9y
							      >	TNokmwf4L3TYkMpDzujeg7yMaX0+9aohoP84mfBL8QuwYoAPYOLzmWCGxIIyR
							      >	FE09pXfqAqv6QxCccWhkQzi9t4jUECgRqUF1K07rmJpsH2v5YO8hgjtahDmu7
							      >	WtpFkw==
-----END CERTIFICATE-----					-----END CERTIFICATE-----
subject=/CN=jazz1.tldp.org					subject=/CN=jazz1.tldp.org
issuer=/O=Random/OU=Domain CA				      |	issuer=/C=US/O=Let's Encrypt/CN=R3
---								---
No client certificate CA names sent				No client certificate CA names sent
Server Temp Key: ECDH, P-384, 384 bits				Server Temp Key: ECDH, P-384, 384 bits
---								---
SSL handshake has read 3591 bytes and written 371 bytes	      |	SSL handshake has read 3346 bytes and written 371 bytes
---								---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256		New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit					Server public key is 2048 bit
Secure Renegotiation IS supported				Secure Renegotiation IS supported
Compression: NONE						Compression: NONE
Expansion: NONE							Expansion: NONE
No ALPN negotiated						No ALPN negotiated
SSL-Session:							SSL-Session:
    Protocol  : TLSv1.2						    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256			    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 61786AE094B33002F072A820FB7DA6B331B8C1F920F3B |	    Session-ID: C978569F2C2CBF439E197995C98ED7C320C27066B46A0
    Session-ID-ctx: 						    Session-ID-ctx: 
    Master-Key: 49536B5CC265DAC58B5B969F6C6129E5C2949CA08991C |	    Master-Key: DB00379CCE6569CB43B48BCC87F181F00C4A498FA3429
    TLS session ticket lifetime hint: 300 (seconds)		    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:						    TLS session ticket:
    0000 - 01 3c 8d f5 53 d0 99 1a-8f cf 9b 2e b9 cb d5 cd    |	    0000 - c4 e5 ec 3e ee e5 f9 58-25 9d 1d 0a c2 1d f6 98   
    0010 - bb 77 9e 81 09 3b 78 5d-83 07 07 2e 1b 4d 92 a9    |	    0010 - 0e ac c0 ca 28 54 7b 5e-fd 53 4b ce f9 9a 80 c9   
    0020 - d2 29 a1 32 78 d6 9c 95-26 57 f5 86 76 cb 8a 51    |	    0020 - dc 25 c9 7d 9e f5 ee e6-9f a1 61 c5 59 61 77 5c   
    0030 - 03 60 9a a8 f9 24 04 85-78 9b 83 d4 70 7c 08 4e    |	    0030 - ba b9 38 61 2d e7 7c e6-70 ab 35 18 39 1b 45 03   
    0040 - 34 8a 5f 5e 10 e6 f2 6b-5b 29 ae 29 fd ea 65 0a    |	    0040 - b8 a1 30 12 44 3a f1 c6-54 16 46 2c c5 d2 cb 26   
    0050 - 57 94 91 dd 48 d5 d7 8a-bb 89 e9 95 c4 12 db 56    |	    0050 - 00 56 74 9e d1 74 8c a5-15 c7 06 27 b8 57 7c ef   
    0060 - 0c 7f b3 bf 92 42 b8 f6-0c d6 e2 22 61 5c 50 57    |	    0060 - 6b 69 bb 06 e1 35 b0 d8-1f c7 48 48 3f 42 f6 3b   
    0070 - 8f 03 22 f0 39 f2 75 d2-80 74 72 6c 14 db 0b 4e    |	    0070 - 34 81 01 21 ac 69 15 60-f6 a5 fb 77 fe 16 db 85   
    0080 - 13 93 25 6b 01 53 b9 12-a6 03 66 c7 c5 53 a2 46    |	    0080 - 70 d2 3b 29 44 17 cd ee-39 ba 5a 52 a5 3b ea d3   
    0090 - a6 3f 46 0e bd e3 24 d9-cb ce 1e e4 50 79 37 28    |	    0090 - 64 65 39 90 68 1c 26 b1-55 10 00 8f 4f 9e 20 96   
    00a0 - 8e 9f 41 50 e8 6e 6c 9c-79 a2 4e d7 66 54 88 0c    |	    00a0 - d2 eb f4 37 37 b4 7d 04-b9 33 fb 14 2c 8e 83 37   
    00b0 - 6e 73 f0 4f 6b ae eb ae-67 6b a0 fa 54 3c db 68    |	    00b0 - 84 46 55 71 60 5e 29 4a-2c c7 93 e0 4f 9e 29 85   

    Start Time: 1620081352					    Start Time: 1620081352
    Timeout   : 7200 (sec)					    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certif |	    Verify return code: 0 (ok)
---								---

[andrewsf]

Confirmable from a third party: Qualys SSL Labs report

• 152.19.134.151 (infra.tldp.ibiblio.org): red "T" for trust issues, "B" if trust issues are ignored
• 152.19.134.152 (infra2.tldp.ibiblio.org): green "A+"

[ser]

Problems relate to the automation of script refreshing letsencrypt certs, I've just run it from hand and all is good now.

I'll try to understand why it fails when unattended.

Thanks :)