Test suite fails on platforms where SHA-1 signatures are unsupported

[ktdreyer]

CentOS 9 Stream and Red Hat Enterprise Linux 9 have SHA-1 signatures disabled by default.

Building asyncssh on these platforms results in many test suite failures due to lack of SHA-1 signature support, like this one:

======================================================================
ERROR: setUpClass (tests.test_forward._TestTCPForwarding)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/asyncssh-2.9.0/tests/util.py", line 368, in setUpClass
    cls.loop.run_until_complete(cls.asyncSetUpClass())
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/builddir/build/BUILD/asyncssh-2.9.0/tests/server.py", line 121, in asyncSetUpClass
    ckey_cert = ckey.generate_user_certificate(ckey, 'name',
  File "/builddir/build/BUILD/asyncssh-2.9.0/asyncssh/public_key.py", line 746, in generate_user_certificate
    return self._generate_certificate(user_key, version, serial,
  File "/builddir/build/BUILD/asyncssh-2.9.0/asyncssh/public_key.py", line 347, in _generate_certificate
    return cert_handler.generate(self, algorithm, key, serial, cert_type,
  File "/builddir/build/BUILD/asyncssh-2.9.0/asyncssh/public_key.py", line 1593, in generate
    data += String(signing_key.sign(data, signing_key.algorithm))
  File "/builddir/build/BUILD/asyncssh-2.9.0/asyncssh/public_key.py", line 557, in sign
    self.sign_ssh(data, sig_algorithm)))
  File "/builddir/build/BUILD/asyncssh-2.9.0/asyncssh/rsa.py", line 242, in sign_ssh
    return String(self._key.sign(data, _hash_algs[sig_algorithm]))
  File "/builddir/build/BUILD/asyncssh-2.9.0/asyncssh/crypto/rsa.py", line 134, in sign
    return priv_key.sign(data, PKCS1v15(), hashes[hash_name]())
  File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 501, in sign
    return _rsa_sig_sign(self._backend, padding, algorithm, self, data)
  File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 244, in _rsa_sig_sign
    pkey_ctx = _rsa_sig_setup(
  File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 213, in _rsa_sig_setup
    raise UnsupportedAlgorithm(
cryptography.exceptions.UnsupportedAlgorithm: sha1 is not supported by this backend for RSA signing.

Paramiko added a marker to tests that support sha1 so they're easy to skip (paramiko/paramiko#2011). Maybe asyncssh could do the same.

[ktdreyer]

(cc @gsauthof since he's maintaining the asyncssh package in Fedora)

[ronf]

Thanks for the report!

I haven't run into this here yet. Is this change specific to CentOS Stream/RHEL 9, or did it come in as part of a particular version of OpenSSL and/or cryptography? Assuming it is CentOS/RHEL specific, do you know how this configuration was done, so I can attempt to reproduce the issue here and see which specific tests are failing and need to be made conditional?

Also, I'm wondering if there might be better way to query whether SHA-1 is enabled or not. For instance, support for Ed25519 can be tested by calling backend.ed25519_supported. It looks like something similar might work for SHA1 using backend.signature_hash_supported. In your environment, can you try the following?

from cryptography.hazmat.backends.openssl import backend
from cryptography.hazmat.primitives.hashes import SHA1
backend.signature_hash_supported(SHA1)

This returns True in my current setup, but I'm expecting it will return False in yours.

[ronf]

Looking more closely, the OpenSSL backend has multiple checks related to SHA1:

• hash_supported(): returns False if non-FIPS hash is used in FIPS mode (which would apply to SHA1)
• signature_hash_supported(): Does an explicit check for SHA1 in FIPS mode, but this seems redundant
• hmac_supported(): Explicitly allows SHA1 in FIPS mode for HMACs
• rsa_padding_supported() allows SHA1 with MGF1 for padding in FIPS mode, even when signatures are not allowed

From what I can see, cryptography's restrictions currently only apply when FIPS mode is enabled, which means the check I mentioned above may not work here. The traceback suggests that the unsupported hash may be something enforced in OpenSSL's EVP_PKEY_CTX_set_signature_md and not in cryptography. Can you confirm that?

[gsauthof]

AFAICS, RHEL patches openssl such that sha1 can be disabled, by a system wide policy (see also and similar patches). It seems that RHEL9 has this policy enabled by default while Fedora hasn't (see also Removing SHA-1 for signatures in Fedora, LWN 2022-03-15).

It looks like Fedora 36 supports the same RHEL9 option for disabling in order to get the same experience (this comment contains some test commands).

[ronf]

Thanks for the additional links. If I understand right, support for disabling this option seems related to a new openssl.cnf setting called "rh-allow-sha1-signatures" under "[evp_properties]". However, it looks to me like this might be a change only in CentOS/RHEL, as I don't see any mention of this yet in https://github.com/openssl/openssl.

I'm wondering what the best way to test this is, ideally without having to spin up a new VM. Any suggestions?

[ktdreyer]

Any suggestions?

You can try running it in Docker or Podman:

podman run -it --rm quay.io/centos/centos:stream9

In your environment, can you try the following?

Here's my result after yum install python3-cryptography:

python3 test.py
Traceback (most recent call last):
  File "test.py", line 3, in <module>                                    
    backend.signature_hash_supported(SHA1)                                     
AttributeError: 'Backend' object has no attribute 'signature_hash_supported' 

I've been wondering if I can toggle all the sha1 values to False in tests/test_public_key.py ... I haven't tested this yet

[ronf]

You can try running it in Docker or Podman:

podman run -it --rm quay.io/centos/centos:stream9

I'm on an M1 Mac here, but after a bit of fiddling I was able to get this to work, for both arm64 and x86_64 containers. I tried running the AsyncSSH unit tests in a centos:stream9 container, but I didn't run into any test failures related to SHA-1 being disabled. Is there some other configuration I need to do within the container to disable SHA-1 support?

Here's my result after yum install python3-cryptography:

python3 test.py
Traceback (most recent call last):
File "test.py", line 3, in
backend.signature_hash_supported(SHA1)
AttributeError: 'Backend' object has no attribute 'signature_hash_supported'

Strange - which version of cryptography did you have installed there? I would expect that attribute to exist on the OpenSSL backend if you're using the latest, but perhaps the yum install is installing an earlier version. Also, just to confirm, did you run this on the machine where you were seeing the unit test failures?

I've been wondering if I can toggle all the sha1 values to False in tests/test_public_key.py ... I haven't tested this yet

I don't think that will work. The boolean there controls whether AsyncSSH will do the testing against OpenSSL (to better check interoperability) or against itself. You could completely delete all the 'sha1' entries, but even then there might be other places in the system where it picks SHA-1 based algorithms that would probably also fail.

[gsauthof]

I can reproduce it in a centos:stream9 container.

However, when I run it like this, rootless podman spams my host system's journal with all console output from the container, e.g.:

Jul 12 11:26:38 example.org interesting_banach[203217]:   File "/home/juser/asyncssh/asyncssh/public_key.py", line 347, in _generate_certificate
Jul 12 11:26:38 example.org interesting_banach[203217]:     return cert_handler.generate(self, algorithm, key, serial, cert_type,
Jul 12 11:26:38 example.org interesting_banach[203217]:   File "/home/juser/asyncssh/asyncssh/public_key.py", line 1593, in generate
Jul 12 11:26:49 example.org interesting_banach[203217]: [93B blob data]
Jul 12 11:26:49 example.org interesting_banach[203217]: [116B blob data

@ktdreyer Any idea how to fix this?

---

However, I did the following to test sha1 support:

podman run -it --rm quay.io/centos/centos:stream9

dnf -y install openssl
openssl req -new -nodes -newkey rsa:2048 -keyout localhost.key -out localhost.csr -subj "/C=US/ST=YourState/L=YourCity/O=Example-Certificates/CN=localhost.local"
openssl dgst -sha1 -binary -out original-ks.cfg.sha1 original-ks.cfg
openssl pkeyutl -inkey localhost.key -sign -in original-ks.cfg.sha1 -out original-ks.cfg.sha1.sig -pkeyopt digest:sha1

=> The last command fails as expected.

update-crypto-policies --show
update-crypto-policies --set LEGACY
openssl pkeyutl -inkey localhost.key -sign -in original-ks.cfg.sha1 -out original-ks.cfg.sha1.sig -pkeyopt digest:sha1
update-crypto-policies --set DEFAULT

=> the security policy is set to DEFAULT, by default - after changing it to LEGACY the openssl command succeeds, as expected (switching it back for the tests)

I installed some dependencies for asyncssh:

dnf -y install 'dnf-command(config-manager)'
dnf config-manager --set-enabled crb
dnf -y install epel-release epel-next-release
dnf -y install git which python3-pip python3-devel python3-setuptools nmap-ncat openssh-clients openssl python3-bcrypt python3-gssapi  python3-pyOpenSSL python3-cryptography python3-typing-extensions

NB: python3-libnacl isn't available on centosstream9 - but it's optional for asyncssh anyways

This installs:

rpm -q python3-cryptography
python3-cryptography-36.0.1-2.el9.x86_64

For the actual tests:

useradd -m juser
su - juser
git clone https://github.com/ronf/asyncssh.git
cd asyncssh
python3 setup.py test 2>&1 | tee t.log

Yields:

grep exception t.log  | sort | uniq -c
    150 cryptography.exceptions.UnsupportedAlgorithm: sha1 is not supported by this backend for RSA signing.

Also there are some permission issues:

grep 'ssh-keygen\|too' t.log  | sort | uniq -c
      2     run('ssh-keygen -e -f %s -m rfc4716 > cert' % cert)
      2     run('ssh-keygen -e -f certout -m rfc4716 > cert')
      2     run('ssh-keygen -i -f certout -m rfc4716 > cert')
      1 Check keys and certificates ... Error running command: ssh-keygen -e -f certout -m rfc4716 > cert
      1 Error running command: ssh-keygen -e -f certout -m rfc4716 > cert
      1 Error running command: ssh-keygen -e -f hostcert -m rfc4716 > cert
      1 Error running command: ssh-keygen -e -f usercert -m rfc4716 > cert
      2 Error running command: ssh-keygen -i -f certout -m rfc4716 > cert
      2 Permissions 0644 for 'certout' are too open.
      1 Permissions 0644 for 'hostcert' are too open.
      1 Permissions 0644 for 'usercert' are too open.
      2 subprocess.CalledProcessError: Command 'ssh-keygen -e -f certout -m rfc4716 > cert' returned non-zero exit status 255.
      1 subprocess.CalledProcessError: Command 'ssh-keygen -e -f hostcert -m rfc4716 > cert' returned non-zero exit status 255.
      1 subprocess.CalledProcessError: Command 'ssh-keygen -e -f usercert -m rfc4716 > cert' returned non-zero exit status 255.
      2 subprocess.CalledProcessError: Command 'ssh-keygen -i -f certout -m rfc4716 > cert' returned non-zero exit status 255.

[ronf]

Thanks very much, @gsauthof ! Following your instructions, I was able to reproduce this here.

It looks like many of the errors are actually happening when trying to run OpenSSH or OpenSSL during the test, rather failing inside AsyncSSH. I'm hoping that as long as I skip tests entirely where SHA-1 is involved, that'll address the issue for both AsyncSSH failures and these external commands getting errors, but it's going to take me a bit of time to find all the places where that's required.

There may also be some tests which just happen to choose SHA-1 (just like many tests arbitrarily choose RSA), where I'll instead want to switch to something like SHA-256. This would allow the test to run whether or not this security policy is set, so I'm only skipping tests that try to test with multiple hash algorithms. Even in those cases I'd still run the test, just removing SHA-1 from the list of hashes tested. That should preserve the coverage since all existing tests will still be run with SOME supported signature algorithm.

I'll try to take a closer look at this over the weekend.

[ronf]

Still working on this. Here are some updates:

It looks like many of the errors are related to using command-line OpenSSH/OpenSSL tools to check for interoperability. Performing these same operations in AsyncSSH doesn't seem to trigger the crypto policy errors.

Most of the errors seem to have to do with certificate handling, going away if I avoid creating an RSA OpenSSH cert which uses SHA-1 as its signature algorithm.

The permissions errors all appear to be related to the fact that DSA certificates can no longer be either created or loaded by libcrypto with the DEFAULT crypto policy. Even though the file is a certificate generated by AsyncSSH, which still works fine, ssh-keygen fails to recognize it as a certificate file and assumes it is a private key file, which then triggers the permission check. Normally, "ssh-keygen -e" doesn't require the strict permission on public key or certificate files.

The failure to both generate and load certificates signed by SHA-1 in ssh-keygen applies to both DSA & RSA certs, but since DSA supports only SHA-1, the only way to test that will be to have AsyncSSH test against itself and not involve ssh-keygen. The AsyncSSH code to do that still works fine, not triggering the check down in OpenSSL even though AsyncSSH is using OpenSSL to actually do the signing. I'm not sure why that is. With RSA keys, ssh-keygen can still work as long as the hash algorithm chosen to do the cert signing is SHA-2.

I'm seeing other errors in test_agent.py and test_connection_auth.py that I'm still tracking down, but most of those appear to be failures when asking the OpenSSH ssh-agent to sign using SHA-1.

[ronf]

I was able to confirm that all of the test failures in test_connection_auth.py were in fact related to attempting to use the SSH agent to do signing with a SHA-1 hash.

I've checked a set of updates into the "develop" branch that seems to fix all the failures in my CentOS 9 container. Along the way, I made a couple of improvements in AsyncSSH itself. Specifically, the changes include:

• A new sig_alg argument is now available to request which signature algorithm to use when creating OpenSSH certificates
• The SSH agent client has been updated to properly request SHA-2 signing for x509v3-rsa2048-sha256 keys paired with an X.509 certificate.
• Agent and connection auth unit tests were updated to avoid keys which request SHA-1 signing in the SSH agent
• DSA unit tests no longer attempt to test interoperability with OpenSSH, to avoid issues with its use of SHA-1 signatures
• An internal make_certificate function was updated to default to using SHA-2 signing for RSA-based OpenSSH certificates

@ktdreyer @gsauthof All of these changes are now in the "develop" branch if you'd like to re-run your tests. Thanks again to both of you for helping me to reproduce the issues here!

[ronf]

These changes are now available in AsyncSSH 2.12.0.