smt-azure.susecloud.net certificate verify failed

[jaldinger]

I have been having this certificate error on I believe all of my SLES machines on Azure since a couple of months at least. Running the update script provided doesn't seem to fix the issue, but returns the same certificate error:

2024-01-08 13:50:20,902 INFO: ~~ sc-repocheck 1.2.9 ~~
2024-01-08 13:50:20,904 INFO: Checking package versions.
2024-01-08 13:50:20,910 INFO: Package versions OK.
2024-01-08 13:50:20,910 INFO: Checking baseproduct.
2024-01-08 13:50:20,910 INFO: SLES baseproduct OK.
2024-01-08 13:50:20,910 INFO: Checking /etc/hosts for multiple records.
2024-01-08 13:50:20,910 INFO: /etc/hosts OK.
2024-01-08 13:50:20,910 INFO: Checking metadata access.
2024-01-08 13:50:20,923 INFO: Metadata OK.
2024-01-08 13:50:20,923 INFO: Checking regionserver access.
2024-01-08 13:50:22,254 INFO: Region server access OK.
2024-01-08 13:50:22,254 INFO: Checking RMT server entry is for correct region.
2024-01-08 13:50:22,254 INFO: RMT server entry OK.
2024-01-08 13:50:22,254 INFO: Checking http port access to RMT servers.
2024-01-08 13:50:22,254 INFO: http check unnecessary.
2024-01-08 13:50:22,255 INFO: Checking https port access to RMT servers.
2024-01-08 13:50:22,319 INFO: https access OK.
2024-01-08 13:50:22,320 INFO: Checking https access using RMT certs.
2024-01-08 13:50:22,320 INFO: An exception of type IndexError occurred. Disregarding.
2024-01-08 13:50:22,320 INFO: EVERYTHING OK.
2024-01-08 13:50:22,320 INFO: Collecting debug data. Please wait 1-2 minutes maybe longer, depending on machine type.
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect
    ssl_context=context,
  File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='smt-azure.susecloud.net', port=443): Max retries exceeded with url: /connect/systems (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/sbin/registercloudguest", line 231, in <module>
    cleanup()
  File "/usr/sbin/registercloudguest", line 101, in cleanup
    utils.remove_registration_data()
  File "/usr/lib/python3.6/site-packages/cloudregister/registerutils.py", line 1162, in remove_registration_data
    'https://%s/connect/systems' % server_name, auth=auth_creds
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 161, in delete
    return request('delete', url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='smt-azure.susecloud.net', port=443): Max retries exceeded with url: /connect/systems (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
2024-01-08 13:50:24,070 INFO: Check repositories. An attempt was made to fix.
2024-01-08 13:50:24,071 INFO: Debug data location: /var/log/sc-repocheck_240108_135022.tar.xz
2024-01-08 13:50:24,072 INFO: Report bugs to https://github.com/rfparedes/susecloud-repocheck/issues

The problem seems to be that the provided certificate is not accepted, and I believe that is due to "certificate pinning" as described in this blog: Accessing the Public Cloud Update Infrastructure via a Proxy.

When testing connectivity using OpenSSL, this is the result:

openssl s_client -showcerts -servername smt-azure.susecloud.net -connect smt-azure.susecloud.net:443  </dev/null  CONNECTED(00000003)
depth=0 C = DE, ST = Bavaria, L = Nuremberg, O = SUSE, OU = Public Cloud, CN = Update server certificate (smt-azure.susecloud.net), emailAddress = suse-public-cloud@susecloud.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = DE, ST = Bavaria, L = Nuremberg, O = SUSE, OU = Public Cloud, CN = Update server certificate (smt-azure.susecloud.net), emailAddress = suse-public-cloud@susecloud.net
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:C = DE, ST = Bavaria, L = Nuremberg, O = SUSE, OU = Public Cloud, CN = Update server certificate (smt-azure.susecloud.net), emailAddress = suse-public-cloud@susecloud.net
   i:C = DE, ST = Bavaria, L = Nuremberg, O = SUSE, OU = Public Cloud, CN = SUSE, emailAddress = suse-public-cloud@susecloud.net
-----BEGIN CERTIFICATE-----
MIIGhjCCBG6gAwIBAgIUVDGA6GdlHJbqrLsDdgspcM9pqnUwDQYJKoZIhvcNAQEL
BQAwgZgxCzAJBgNVBAYTAkRFMRAwDgYDVQQIDAdCYXZhcmlhMRIwEAYDVQQHDAlO
dXJlbWJlcmcxDTALBgNVBAoMBFNVU0UxFTATBgNVBAsMDFB1YmxpYyBDbG91ZDEN
MAsGA1UEAwwEU1VTRTEuMCwGCSqGSIb3DQEJARYfc3VzZS1wdWJsaWMtY2xvdWRA
c3VzZWNsb3VkLm5ldDAeFw0yMzAzMzAxNTQzNDBaFw0zMzAzMjcxNTQzNDBaMIHH
MQswCQYDVQQGEwJERTEQMA4GA1UECAwHQmF2YXJpYTESMBAGA1UEBwwJTnVyZW1i
ZXJnMQ0wCwYDVQQKDARTVVNFMRUwEwYDVQQLDAxQdWJsaWMgQ2xvdWQxPDA6BgNV
BAMMM1VwZGF0ZSBzZXJ2ZXIgY2VydGlmaWNhdGUgKHNtdC1henVyZS5zdXNlY2xv
dWQubmV0KTEuMCwGCSqGSIb3DQEJARYfc3VzZS1wdWJsaWMtY2xvdWRAc3VzZWNs
b3VkLm5ldDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMyijJMJWh7x
w1uCJIf1EM7U5bW5dgjDv6JNVUYoFBgzeMzdwLIc4WPza4xvzVqvW7PnBeavSV+t
KvxMzRdy6K6bk/Ln1nht00v61JzHYCX8qabw5Grq30aWRoPKW8ItRyUA8NNpZrq9
HTt0QFK//KBcR+S0CHQjwkWx7Nx3fxtpJ8PfrCmq1Xgip5ocChSIKxUaRIOme8ey
vv6mR3JHDNjOanEn6ILj5GBpIAObooh/q202aSib/GsrJ3u7TStwzDoGDjkMgMzh
SAnnBxtfWyeKlOeJBBfwbqkjWehwXEENo/x5hTUqcxPecYRoYdFOX5ZnYL3NO8sS
JOVTw+TRzLWsvzxvJlr1hUDZbFRLWVq/YUYY6YOoDxM9g5AYbulyiPPNbZIKVEPm
lI6d8JWX5MdDt0BHf2kpbDYbrxLJZjfT8IOXwCmbETz9Q/0A/clyaR9BffBBsxMK
vtDSWXa0z+/j3MmjNoeGCjBbvxgi2HUzTTwXXIf0tYxWV0CsDbXtb76hweFO4Z5x
TLqtlR14chyrRynZDtL3aZXFXrmYcPPWQkxXhDU8JgmT0IQdXvD0Ms+j+Ud0UvLp
DdVLNX6d8a3SBBm4qgIzeCInstlbnCFBYeKCt93JRJNlgELmT69JUwc8wpoeatp0
3khRrkYY02wX13jpAIRqKkrrBeNyxUKHAgMBAAGjgZYwgZMwgZAGA1UdEQSBiDCB
hYIXc210LWF6dXJlLnN1c2VjbG91ZC5uZXSCGXJtdC1henVyZS0xLnN1c2VjbG91
ZC5uZXSCGXJtdC1henVyZS0yLnN1c2VjbG91ZC5uZXSCGXJtdC1henVyZS0zLnN1
c2VjbG91ZC5uZXSCGXJtdC1henVyZS00LnN1c2VjbG91ZC5uZXQwDQYJKoZIhvcN
AQELBQADggIBAKPr3GsT8sACEZqTVd+oy515kwEBT20KjI4VkylEApRl204wP8kb
V7z1mb8uvRWG5kNkQoYN0nVufYOVBkd9PjAGn1eK4CNVmsZpBPrvduAheUzZlKFe
UuUI/MoO8rs1M7vP+L/wdaMZ3d7B2clAjnsW+KyI8F10HbS5QM1Kh1mQZAWbvo/r
NE9CFh9gAQY9BvUHf0gYG0qRRei5KvxmMO7itMVsEeE9NhAif3ebrsGaAAztujZo
p19yKH9RcYDT+pyCEtQzFqbZ3FTnZLbbNGMrj/DmLo4w+K9OJfUc6+m4LLtug6De
U/UZ1d1RW8A7t5WPrraMb0fr01W9ZM7wD2c+vAI/TvoVyiE9/ZSILzRlfjDnyzOR
8dU2l3EwUoHuBNTI01N7lwrEB9hJwXFYWAvCWNGO2TlGWjjHP6L0KADHsutvFD3u
fDIx25Cq5q95Xr7wuFT/jN7C9VCDkX8wlRMxYCtRLw8kUNSalYwUFU3gPpGFTNt6
tMWPeAGoAB6n1Ftg+DnFDugeXWq/lppA/MAxvl365aX/9KqRqaShOsTwgORRJ5Ak
rpkt6LcRe7/K9T4donMarMhyrANFHcqeDrDkB2yng5qpHTlcE01ClsJ3aQ6xjOTh
17wTT/uiwQdHMpONLNwthBmpqZEqt0eW3zrO3gkfZVKLPzTCITrra4Yx
-----END CERTIFICATE-----
---
Server certificate
subject=C = DE, ST = Bavaria, L = Nuremberg, O = SUSE, OU = Public Cloud, CN = Update server certificate (smt-azure.susecloud.net), emailAddress = suse-public-cloud@susecloud.net

issuer=C = DE, ST = Bavaria, L = Nuremberg, O = SUSE, OU = Public Cloud, CN = SUSE, emailAddress = suse-public-cloud@susecloud.net

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2490 bytes and written 405 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE

I compared the certificate returned by this command to the one stored in /usr/share/pki/trust/anchors/registration_server_52_188_224_179.pem (that matches the IP address stored in /etc/hosts) and in effect it was different. Just for testing the issue, I updated this file to reflect the current certificate and ran update-ca-certificates and then tried again. However, the issue still persists.

Just in case: I'm not using any proxies or doing SSL inspection, so the original certificate should be untampered with.

I'm not sure what else I can try, but I find it strange that this is happening on all of my SLES VMs, in different Azure tenants and in versions ranging from SLES15-SP1 to SLES-15-SP3. A newly installed SLES15-SP4 is working fine.

Any help would be greatly appreciated.

Thanks,

Joerg.

[jaldinger]

OK, so I finally found a solution for this issue, but I'm thinking that this should probably make its way into this project, so I'm sharing:

# get current certificate from server
openssl s_client -showcerts -servername smt-azure.susecloud.net -connect smt-azure.susecloud.net:443 </dev/null 2>/dev/null | openssl x509 -outform PEM >~/SUSE.pem

# make backup of old certificate
cp /var/lib/ca-certificates/pem/SUSE.pem /var/lib/ca-certificates/pem/SUSE.pem.bak

# overwrite old certificate with new
cp ~/SUSE.pem /var/lib/ca-certificates/pem/SUSE.pem

# Create symbolic links to files named by the hash values
c_rehash

After doing that, zypper ref connected successfully.

I can see that the current certificate is valid since March 30, 2023, so maybe that's when it stopped working? How should that certificate have been updated (if at all?). Is my solution correct at all?

Thanks,

Joerg