Certificate renewal fails with ERROR: No CA found for server certificate

[wizonet]

Problem description

I use a wildcard certificate from Let's Encrypt, which has to be renewed every three months.

This is actually a documented standard procedure that I always do in the same way - but there is now a problem with the command in the current renewal:

uyuni:~ # mgr-ssl-cert-setup --root-ca-file=/opt/cert/isrgrootx1.pem --intermediate-ca-file=/opt/cert/lets-encrypt-r3.pem --server-cert-file=/opt/cert/fullchain.pem --server-key-file=/opt/cert/privkey.pem

ERROR: No CA found for server certificate

Only the fullchain.pem and the privkey.pem are new. However, these are valid as I also use them in many other places (web server, firewall etc.).

The root and intermediate certificates have not changed either - they are from here:
https://letsencrypt.org/certificates/

The renewal three months ago still worked without any problems with the same command

Steps to reproduce

1. Generate a new wildcard certificate at Let's Encrypt
2. Copy to uyuni server
3. use mgr-ssl-cert-setup
   ...

Uyuni version

Repository     : uyuni-server-stable
Name           : Uyuni-Server-release
Version        : 2024.05-230900.217.1.uyuni3
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Server-release-2024.05-230900.217.1.uyuni3.src
Summary        : Uyuni Server
Description    : 
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system
    provisioning.

Uyuni proxy version (if used)

No response

Useful logs

No response

Additional information

No response

[mcalmer]

You give the fullchain.pem as server certificate. The command expect only the server cert here. The fullchain.pem can be provided with --intermediate-ca-file . This option can be used multiple times in this command.

[wizonet]

Ok, has this changed approximately 3 months ago? Because I use the fullchain.pem als server certificate in uyuni since years with exactly this command.

Also all apache vhosts and appliances accecpt the fullchain.pem as server certificate without any problem.

[wizonet]

Tried this - but it will also fail:

mgr-ssl-cert-setup --root-ca-file=/opt/cert/isrgrootx1.pem --intermediate-ca-file=/opt/cert/lets-encrypt-r3.pem --intermediate-ca-file=/opt/cert/fullchain.pem --server-cert-file=/opt/cert/cert.pem --server-key-file=/home/wzowim00/cert/privkey.pem
After changing the server certificate please execute:
$> spacewalk-service stop
$> systemctl restart postgresql.service
$> spacewalk-service start

As the CA certificate has been changed, please deploy the CA to all registered clients.
On salt-managed clients, you can do this by applying the highstate.
Failed to upload CA Certificate to DB: ERROR: can't find CA certificate at this location: -

ERROR: Failed to upload CA Certificate to DB

[wizonet]

I found this logfile, which contains the last successful attempt as well as the currently failed one.

/var/log/rhn/mgr-ssl-cert-setup.log:
2024/04/06 18:58:04 +02:00: /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py.processCommandline(103) - ['/usr/bin/mgr-ssl-cert-setup', '--root-ca-file=/opt/cert/isrgrootx1.pem', '--intermediate-ca-file=/opt/cert/lets-encrypt-r3.pem', '--server-cert-file=/opt/cert/fullchain.pem', '--server-key-file=/opt/cert/privkey.pem']
2024/04/06 18:58:04 +02:00: /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py.deployApache(460) - After changing the server certificate please execute:
$> spacewalk-service stop
2024/04/06 18:58:04 +02:00: /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py.deployPg(475) - $> systemctl restart postgresql.service
2024/04/06 18:58:08 +02:00: /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py.deployCAUyuni(518) - $> spacewalk-service start

As the CA certificate has been changed, please deploy the CA to all registered clients.
On salt-managed clients, you can do this by applying the highstate.

2024/06/28 18:56:40 +02:00: /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py.processCommandline(117) - ['/usr/bin/mgr-ssl-cert-setup', '--root-ca-file=/opt/cert/isrgrootx1.pem', '--intermediate-ca-file=/opt/cert/lets-encrypt-r3.pem', '--server-cert-file=/opt/cert/fullchain.pem', '--server-key-file=/opt/cert/privkey.pem']
2024/06/28 18:56:41 +02:00: /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py.writeError(713) -
ERROR: No CA found for server certificate

[mcalmer]

For easier debugging it would be helpful when you could provide all the certificates. NOT the private key please :-)

[wizonet]

lets-encrypt-r3.zip

The ZIP include:
lets-encrypt-r3.pem
isrgroot1.pem
fullchain.pem

I just download the lets-encrypt-r3.pem and isrgroot1.pem from https://letsencrypt.org/certificates/ and made a diff to my files, but there were no differences, so I think this root and intermediate certificate are still valid.

[mcalmer]

Using fullchain.pem as --server-cert-file is the root of the problem. We just take the first certificate in this file into account.
The rest of the chain is ignored.
I have made a patch to change this.

The command:

$> mgr-ssl-cert-setup --root-ca-file=/opt/cert/isrgrootx1.pem --intermediate-ca-file=/opt/cert/lets-encrypt-r3.pem --intermediate-ca-file=/opt/cert/fullchain.pem --server-cert-file=/opt/cert/cert.pem --server-key-file=/home/wzowim00/cert/privkey.pem

should have worked. The error you reported is a different one as the one before.
I will check now what happened here.

[mcalmer]

The error happens when importing the CA into the database.
In my setup it works with your certificates.
Maybe you can take the new file from my patch (https://raw.githubusercontent.com/uyuni-project/uyuni/1948d4f3381c89cc01307b48559a3f17ac27a578/spacewalk/certs-tools/mgr_ssl_cert_setup.py) and copy it in your system under /usr/lib/python3.6/site-packages/certs/mgr_ssl_cert_setup.py (I think in uyuni we still use python3.6)
and run the command again and check if it works now.

If you see the error Failed to upload CA Certificate to DB: ERROR: can't find CA certificate at this location: -
we need to investigate why this happens.