Certificate expired for https://updates.coreos.fedoraproject.org

[curantes]

Describe the bug
The Let's Encrypt cert for https://updates.coreos.fedoraproject.org have not been renewed in time before expire

Reproduction steps
Steps to reproduce the behavior:

1. zincati can't communicate to update servers

Expected behavior
Auto updates should work in Fedora CoreOS

Actual behavior
Se these errors in zincati logs:

[ERROR zincati::cincinnati] failed to check Cincinnati for updates: client-side error: error sending request for url (https://updates.coreos.fedoraproject.org/v1/graph?group=default&os_checksum=30c82ee684674b9a552ffee709501f981f35f36408085f089686e43b09aeca1b&stream=stable&node_uuid=70c950d996c341f39813155e33b04b51&platform=vmware&os_version=35.20211215.3.0&basearch=x86_64): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1914: (certificate has expired)

System details

• Vmware VM
• 35.20211215.3.0 (CoreOS)

Ignition config
N/A

Additional information

$ openssl s_client -connect updates.coreos.fedoraproject.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = updates.coreos.fedoraproject.org
verify error:num=10:certificate has expired
notAfter=Jan 18 21:12:12 2022 GMT
verify return:1
depth=0 CN = updates.coreos.fedoraproject.org
notAfter=Jan 18 21:12:12 2022 GMT
verify return:1
---
Certificate chain
 0 s:CN = updates.coreos.fedoraproject.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFRzCCBC+gAwIBAgISBCqoUPTgC70yEQeQQdm+4FrOMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEwMjAyMTEyMTNaFw0yMjAxMTgyMTEyMTJaMCsxKTAnBgNVBAMT
IHVwZGF0ZXMuY29yZW9zLmZlZG9yYXByb2plY3Qub3JnMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA08bQu9SNsMFE4xB7mSKvQwo2SBoVj1zOgep2M/U0
Ep40Tznvd1u9A0b4051AaEbQD1aDqNcEOj6kZ3qByD3MIb4GgLNS+8uTrFW1YINH
6nAzY3oJd6VtTgkb6R0l8QFwrXwwhCOtzRkWSEooytFGihEI2VRQop1c4IY4IXRw
0+dGIuNqzO/6d3GQOkW6qSQSAkaRNjY+7TLrHbUiQunyafARfJ9/ig98TtME+bAx
/nlRc5HYCy1fB3tNqJW5LQzw8IcKChXyhS9tAIVQsfVDOHeV/f/P6LYZc9pTDTVH
IMf+TLouove1BQn8MDGbXh4vy3JeEioMbnI93N4O4Ve+YwIDAQABo4ICXDCCAlgw
DgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAM
BgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRiQMITGsLRYE9F1ip1bpmhKADIBzAfBgNV
HSMEGDAWgBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYI
KwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0
cDovL3IzLmkubGVuY3Iub3JnLzArBgNVHREEJDAigiB1cGRhdGVzLmNvcmVvcy5m
ZWRvcmFwcm9qZWN0Lm9yZzBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLf
EwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCC
AQUGCisGAQQB1nkCBAIEgfYEgfMA8QB2AEHIyrHfIkZKEMahOglCh15OMYsbA+vr
S8do8JBilgb2AAABfJ/DD/cAAAQDAEcwRQIgehN8hUKPsLLreFjS+cjuW94RH/Ex
U6lzDTA5Yj5X5oUCIQDtSwEqMtlQHsZ2khnzRQ214rIdID3gNA5t6ZHx0ystUAB3
AEalVet1+pEgMLWiiWn0830RLEF0vv1JuIWr8vxw/m1HAAABfJ/DEAwAAAQDAEgw
RgIhAIY+U2FoZ7XGGugLj+klTrctFMlkERW4etTtoW3Fw0bcAiEAq9I4vuppF8GY
+6Evq3a7/PqIkKDop6jm2ni8LAKtkPgwDQYJKoZIhvcNAQELBQADggEBAB36m9Bj
StjPND4ZIWLoP8W7sKp4lW9KBQbdD0vDpiJ/AzncwhceWQ9bW1g2MSXHgOz8rsmd
DIKIGUupzkc3WrHcjrBk6xokydcLJBDx7vk0SS/DCkZRbyURiKL5lkx5ytjQQdVy
Ot91ye7qjEMoUOS/GfBXk3pL6Azh8RK2jhfb3mmN56nbsYQqKyQOA3eJJXtUnJIj
Dqx3/Hb54ihGnNR0xgxmthO9FORhxfMQboY4YWf55tqbwwkkaG9lG7Jrmfz/WAvB
V+ukwCWdrmfaezjUNi3PL32E7CpnKFBYKt74YP3BNTjxkBZZhDpw94A+YydhagL5
/FQSm3cg3mNajKI=
-----END CERTIFICATE-----
subject=CN = updates.coreos.fedoraproject.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4611 bytes and written 406 bytes
Verification error: certificate has expired
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 04C4BCFFA4EF76F2D1E8E0B4271E7DE57A0FC4E1851A4CFC79BCB0F2A342B7B3
    Session-ID-ctx: 
    Resumption PSK: 25E13F86BF4C2450C0066545467129BBDE8D6426B8FA7AB5359728E8BBF703146C5D8239684A2B11827494FDA1497BDC
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 66 70 70 72 6f 64 32 30-32 32 30 31 31 39 30 36   fpprod2022011906
    0010 - 7b 3b 05 cb c8 dd 88 20-af 0f d1 ed 1b a5 fd 09   {;..... ........
    0020 - 92 08 87 7e 28 56 01 80-28 ec c5 57 ec 86 1e 7b   ...~(V..(..W...{
    0030 - 42 78 ac 04 20 12 66 d8-ea 41 20 2c 40 0e ce 2c   Bx.. .f..A ,@..,
    0040 - 1b cd dc 24 18 f7 99 d2-c4 c9 1c 39 96 e3 89 98   ...$.......9....
    0050 - f0 d2 b8 9c a2 01 33 0a-3d 77 7f 4f bb 70 36 62   ......3.=w.O.p6b
    0060 - 67 78 3b 14 9a d3 d7 80-01 d4 ee 14 0d 65 a4 92   gx;..........e..
    0070 - 92 f0 a0 d1 55 1d 84 e3-20 e4 22 4d 91 d3 d1 23   ....U... ."M...#
    0080 - 55 09 10 7e 8c ad 1e 91-a4 53 6e 6a a2 a6 1c e8   U..~.....Snj....
    0090 - 38 8c 80 05 ad d4 11 e7-ef f6 7c 9c 8c ad 3c 17   8.........|...<.
    00a0 - c0 04 4b e9 aa 82 7d f4-8e 72 78 23 64 9d cf a7   ..K...}..rx#d...
    00b0 - 29 7a 9b f6 48 6b 09 10-1a 9e a7 be 9b e3 eb 82   )z..Hk..........
    00c0 - cc e3 d7 3c 9c cc 86 e9-8e 4d f0 27 9f a3 0a df   ...<.....M.'....
    00d0 - 28 6e 41 eb 26 ae 1b 14-74 f9 f6 4e 4c 15 e4 68   (nA.&...t..NL..h
    00e0 - ba 74 36 f6 d4 6a c9 da-64 4a 6a 04 f2 47 c2 85   .t6..j..dJj..G..
    00f0 - 84 86 b9 2a 3f de ae b6-6a 94 42 af 6d e1 0c e6   ...*?...j.B.m...
    0100 - 28 e8 6f c7 b3 92 10 1a-3c 7b ed e4 5c ba 48 ee   (.o.....<{..\.H.

    Start Time: 1642575245
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 83632774CB2603B621236EEAF1CDA3AF51B846957F298441329CD95EA989CDC0
    Session-ID-ctx: 
    Resumption PSK: 403BF49FD0081DC35FBD6F58699398A9E847B44D22BE26B4DC606EA95EACD3140FD7812A90F6DD863BD5FFB65A220C41
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 66 70 70 72 6f 64 32 30-32 32 30 31 31 39 30 36   fpprod2022011906
    0010 - 7b 5e 83 83 8d e8 a9 4b-df 82 27 55 fb 7a ef fc   {^.....K..'U.z..
    0020 - 74 18 fc a9 6e 87 88 5d-b1 3f 82 43 d7 81 78 cf   t...n..].?.C..x.
    0030 - aa 30 33 43 39 da a9 c3-fb e9 7d a2 d1 30 d5 1b   .03C9.....}..0..
    0040 - 43 84 f7 ad 0e 7b a8 8d-9a 64 1e 8a 6e 8c 5a 6c   C....{...d..n.Zl
    0050 - 38 e8 c9 70 22 45 e2 fb-46 4e 69 2d cf 54 39 b6   8..p"E..FNi-.T9.
    0060 - 45 80 75 e7 6b db 94 2f-d2 05 e8 c6 f3 4b b2 8d   E.u.k../.....K..
    0070 - 39 05 a4 cf bd 03 3d 37-97 e4 a3 e6 66 49 fc b1   9.....=7....fI..
    0080 - 47 3e 49 0f 69 85 9f cb-7c 1c 94 3d c9 6c a2 c1   G>I.i...|..=.l..
    0090 - 29 55 7b e6 51 b2 11 79-e5 a9 79 79 a6 0b f0 7b   )U{.Q..y..yy...{
    00a0 - e0 2a 2c 92 ed 6d f3 a8-8e 5c 52 c6 37 65 c7 54   .*,..m...\R.7e.T
    00b0 - a6 59 78 6d 2d d5 b7 32-67 97 b8 45 f1 18 ad e9   .Yxm-..2g..E....
    00c0 - 1d ae 30 96 6e ba 29 7e-d4 a2 e4 72 ac a6 cb 36   ..0.n.)~...r...6
    00d0 - a9 0b 6c 10 3b 12 4a c9-e1 0a 58 38 a3 f9 e2 47   ..l.;.J...X8...G
    00e0 - b2 16 4f 8e be 72 39 cd-23 4b f6 4d c4 45 1b 2f   ..O..r9.#K.M.E./
    00f0 - 2e f7 68 6a bc a8 6c 50-9a 32 c5 84 38 a6 58 7e   ..hj..lP.2..8.X~
    0100 - 12 a9 31 fc 53 cc 83 88-3d 98 2b b6 f2 3b 68 b0   ..1.S...=.+..;h.

    Start Time: 1642575245
    Timeout   : 7200 (sec)
    Verify return code: 10 (certificate has expired)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

[lucab]

Thanks for the report.
Certificate renewals are handled through fedora-infra, so we don't directly manage that and we'll need somebody to fix this for us.
I've already highlighted this in the relevant Matrix channel, but not much movement so far.
Until it get resolved, this will be tracked at https://pagure.io/fedora-infrastructure/issue/10485.

[dustymabe]

cert should be good now:

$ export SITE_URL="updates.coreos.fedoraproject.org"
$ export SITE_SSL_PORT="443"
$ openssl s_client -connect ${SITE_URL}:${SITE_SSL_PORT} \
  -servername ${SITE_URL} 2> /dev/null |  openssl x509 -noout  -dates
notBefore=Jan 19 15:05:10 2022 GMT
notAfter=Apr 19 15:05:09 2022 GMT

[lucab]

That specific endpoint seems fine now, but overall TLS ingress does not seem to be fully healthy yet.
Graph-Builder (and all metrics endpoints too) gives this:

depth=0 CN = raw-updates.coreos.fedoraproject.org
verify error:num=10:certificate has expired
notAfter=Jan 18 21:13:28 2022 GMT

[bgilbert]

From this comment it sounds like this might happen again in future. Should we either push for better monitoring or add some ourselves?

[dustymabe]

Probably.. ideally we could hook into the monitoring they already have and add ourselves to the list of people to alert (that way we at least don't manage something new).

[lucab]

I'm going ahead and closing this ticket as the incident got resolved. I'm keeping following up on the pagure one.
If the sysadmins already have some certs-expiration in place, I agree we should add our known endpoints there so that they can react before the deadline.
Taking a step back, it'd be great to have those infra certificates auto-renewing.