OpenSSL errors in Sidekiq

[lgbtio]

Sidekiq is throwing a lot of the OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) on and occasionally Stoplight::Error::RedLight: errors. Federated timeline is affected as it's unable to connect to almost all other Mastodon instances.

I did an SSLLabs test and the only noticeable thing is the DST Root CA X3 - expired on 30th Sep 2021. I followed this guide below, to remove the DST Root CA X3, and SSLLabs test confirms it no longer exist.

sudo apt-get remove -y certbot python3-certbot-apache
sudo snap install certbot --classic
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot renew --force-renewal --preferred-chain "ISRG Root X1"

However, Sidekiq continues to throw the OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) on and occasional Stoplight::Error::RedLight: errors.

Can anyone kindly shed some lights regarding this?

Here are some info if helpful:

Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-193-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

14 packages can be updated.
8 updates are security updates.

New release '18.04.6 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

root@host ~ # openssl x509 -enddate -noout -in /etc/ssl/certs/ISRG_Root_X1.pem
notAfter=Jun  4 11:04:38 2035 GMT
root@host ~ # openssl x509 -enddate -noout -in /etc/ssl/certs/DST_Root_CA_X3.pem                                                                                                             Can't open /etc/ssl/certs/DST_Root_CA_X3.pem for reading, No such file or direct                                                                                                             ory
139682917435136:error:02001002:system library:fopen:No such file or directory:cr                                                                                                             ypto/bio/bss_file.c:69:fopen('/etc/ssl/certs/DST_Root_CA_X3.pem','r')
139682917435136:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio                                                                                                             /bss_file.c:76:
unable to load certificate
root@host ~ # apt-cache show openssl
Package: openssl
Architecture: amd64
Version: 1.0.2g-1ubuntu4.20
Priority: standard
Section: utils
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.                                                                                                             org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 934
Depends: libc6 (>= 2.15), libssl1.0.0 (>= 1.0.2g)
Suggests: ca-certificates
Filename: pool/main/o/openssl/openssl_1.0.2g-1ubuntu4.20_amd64.deb
Size: 491618
MD5sum: 00736ee1be6c36e9d5fc5f6df781b39c
SHA1: 507e405b505dfcb16706f8fbbd5c2657de08f53f
SHA256: 9159c5bd3767cbd85f8573e5791dda3d95875d31f5d7152ea464557175b4ee09
SHA512: d84f1f7feb41878a5420dfe0422cdfc18676e6236756dee4a9d6ed938a79f989c29f0523                                                                                                             8cf9049c3ef1c30d7e8f6a6b90af32c8d4fbea55826da082754f06b2
Description-en: Secure Sockets Layer toolkit - cryptographic utility
 This package is part of the OpenSSL project's implementation of the SSL
 and TLS cryptographic protocols for secure communication over the
 Internet.
 .
 It contains the general-purpose command line binary /usr/bin/openssl,
 useful for cryptographic operations such as:
  * creating RSA, DH, and DSA key parameters;
  * creating X.509 certificates, CSRs, and CRLs;
  * calculating message digests;
  * encrypting and decrypting with ciphers;
  * testing SSL/TLS clients and servers;
  * handling S/MIME signed or encrypted mail.
Description-md5: 9b6de2bb6e1d9016aeb0f00bcf6617bd
Task: standard, ubuntu-core, ubuntu-core, mythbuntu-frontend, mythbuntu-backend-                                                                                                             slave, mythbuntu-backend-master, ubuntu-touch-core, ubuntu-touch, ubuntu-sdk-lib                                                                                                             s-tools, ubuntu-sdk
Supported: 5y

Package: openssl
Priority: standard
Section: utils
Installed-Size: 934
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.                                                                                                             org>
Architecture: amd64
Version: 1.0.2g-1ubuntu4
Depends: libc6 (>= 2.15), libssl1.0.0 (>= 1.0.2g)
Suggests: ca-certificates
Filename: pool/main/o/openssl/openssl_1.0.2g-1ubuntu4_amd64.deb
Size: 492190
MD5sum: 8280148dc2991da94be5810ad4d91552
SHA1: b5326f27aae83c303ff934121dede47d9fce7c76
SHA256: e897ffc8d84b0d436baca5dbd684a85146ffa78d3f2d15093779d3f5a8189690
Description-en: Secure Sockets Layer toolkit - cryptographic utility
 This package is part of the OpenSSL project's implementation of the SSL
 and TLS cryptographic protocols for secure communication over the
 Internet.
 .
 It contains the general-purpose command line binary /usr/bin/openssl,
 useful for cryptographic operations such as:
  * creating RSA, DH, and DSA key parameters;
  * creating X.509 certificates, CSRs, and CRLs;
  * calculating message digests;
  * encrypting and decrypting with ciphers;
  * testing SSL/TLS clients and servers;
  * handling S/MIME signed or encrypted mail.
Description-md5: 9b6de2bb6e1d9016aeb0f00bcf6617bd
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Task: standard, ubuntu-core, ubuntu-core, mythbuntu-frontend, mythbuntu-backend-                                                                                                             slave, mythbuntu-backend-master, ubuntu-touch-core, ubuntu-touch, ubuntu-sdk-lib                                                                                                             s-tools, ubuntu-sdk

You have new mail in /var/mail/root
root@host ~ # openssl version
OpenSSL 1.1.1g  21 Apr 2020
You have new mail in /var/mail/root
root@host ~ # apt search ca-certificates
Sorting... Done
Full Text Search... Done
ca-certificates/xenial-updates,xenial-updates,xenial-security,xenial-security,xenial-updates,xenial-updates,xenial-security,xenial-security,now 20210119~16.04.1 all [installed]
  Common CA certificates

ca-certificates-java/xenial-updates,xenial-updates,xenial-updates,xenial-updates,now 20160321ubuntu1 all [installed,automatic]
  Common CA certificates (JKS keystore)

ca-certificates-mono/xenial,xenial,xenial,xenial 4.2.1.102+dfsg2-7ubuntu4 all
  Common CA certificates (Mono keystore)

liblwp-protocol-https-perl/xenial,xenial,xenial,xenial 6.06-2 all
  HTTPS driver for LWP::UserAgent

root@host ~ # dpkg -l | grep libssl
ii  libssl-dev:amd64                     1.1.0g-2ubuntu4                            amd64        Secure Sockets Layer toolkit - development files
ii  libssl1.0.0:amd64                    1.0.2g-1ubuntu4.20                         amd64        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.1:amd64                      1.1.0g-2ubuntu4                            amd64        Secure Sockets Layer toolkit - shared libraries
You have new mail in /var/mail/root
root@host ~ #

[ClearlyClaire]

The error is not with your certificate, but with checking remote instances' certificates. I don't know the exact reason for the failure, but since you've installed libssl1.1 from another distribution, I imagine that libssl1.1 and some certificates in ca-certificates could be incompatible.

I'd suggest you update to a more recent ubuntu LTS, such as Ubuntu 20.04, which ships with libssl 1.1.

[ClearlyClaire]

This is surprising, I can't see why redis-server would require libssl1.0.0. What is the output of apt policy redis redis-server redis-tools?

[lgbtio]

This is surprising, I can't see why redis-server would require libssl1.0.0. What is the output of apt policy redis redis-server redis-tools?

Hi @ClearlyClaire, here's the output:

root@host ~ # apt policy redis redis-server redis-tools
redis:
  Installed: (none)
  Candidate: 5:5.0.7-2ubuntu0.1
  Version table:
     5:5.0.7-2ubuntu0.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages
        500 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages
     5:5.0.7-2 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        500 http://archive.ubuntu.com/ubuntu focal/universe i386 Packages
redis-server:
  Installed: 5:6.0.6-3chl1~xenial1
  Candidate: 5:6.0.6-3chl1~xenial1
  Version table:
 *** 5:6.0.6-3chl1~xenial1 100
        100 /var/lib/dpkg/status
     5:5.0.7-2ubuntu0.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages
     5:5.0.7-2 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
redis-tools:
  Installed: 5:6.0.6-3chl1~xenial1
  Candidate: 5:6.0.6-3chl1~xenial1
  Version table:
 *** 5:6.0.6-3chl1~xenial1 100
        100 /var/lib/dpkg/status
     5:5.0.7-2ubuntu0.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages
     5:5.0.7-2 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
root@host ~ #

[ClearlyClaire]

ok, it seems you have installed redis from a PPA that you have now disabled, and that it's still using that version of redis… if I understand https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12451 correctly, it should be safe to downgrade from 6.0.6 to 5.0.7.

To do so: apt install redis/focal redis-tools/focal redis-server/focal

[lgbtio]

Hi @ClearlyClaire redis has been downgraded to 5.0.7. Next is to uninstall libssl1.0.0, did I understand it correctly?

[ClearlyClaire]

Yes!