RSA key creation fails intermittently when enabling blinding (primarily on Ubuntu 20.04)

[adiroiban]

Some tests randomly fail on Twisted with the following error

cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=50901100, lib=3, func=139, reason=108, reason_text=b'error:0308B06C:bignum routines:BN_mod_inverse_no_branch:no inverse'), _OpenSSLErrorWithText(code=67665923, lib=4, func=136, reason=3, reason_text=b'error:04088003:rsa routines:RSA_setup_blinding:BN lib')])

I am running the tests on Twisted latest trunk branch.

I have updated cryptography with the latest dev version and I still get random failures

I am running on Ubuntu 20.04 with

$ python --version
Python 3.6.9 (7.3.1+dfsg-4, Apr 22 2020, 05:15:29)
[PyPy 7.3.1 with GCC 9.3.0]


$ openssl version
OpenSSL 1.1.1f  31 Mar 2020

$ pip show cffi pip setuptools
Name: cffi
Version: 1.14.0
---
Name: pip
Version: 20.2.4
---
Name: setuptools
Version: 50.3.2

To reproduce this I did

$ virtualenv -p pypy3 test
$ cd test
$ . ./bin/activate
$ git clone https://github.com/twisted/twisted.git
$ cd twisted
$ pip install -e .[dev,tls,conch]
$ pip install git+https://github.com/pyca/cryptography.git
$ trial -x twisted.conch.test.test_keys.KeyTests

The last trial step needs to be executed multiple times...but it fails 1 out of 3 runs on my system.

The same error is observed on GitHub actions VMs See for example https://github.com/twisted/twisted/runs/1311722807?check_suite_focus=true#step:8:16044

Thanks

[alex]

Full stack trace:

2020-10-26T23:15:19.8531572Z Traceback (most recent call last):
2020-10-26T23:15:19.8532845Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/conch/test/test_keys.py", line 668, in test_equal
2020-10-26T23:15:19.8533946Z     rsa3 = keys.Key(keys.Key._fromRSAComponents(n=5, e=3)._keyObject)
2020-10-26T23:15:19.8535458Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/twisted/conch/ssh/keys.py", line 795, in _fromRSAComponents
2020-10-26T23:15:19.8536618Z     keyObject = publicNumbers.public_key(default_backend())
2020-10-26T23:15:19.8538241Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/cryptography/hazmat/primitives/asymmetric/rsa.py", line 359, in public_key
2020-10-26T23:15:19.8539348Z     return backend.load_rsa_public_numbers(self)
2020-10-26T23:15:19.8540920Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 641, in load_rsa_public_numbers
2020-10-26T23:15:19.8542066Z     return _RSAPublicKey(self, rsa_cdata, evp_pkey)
2020-10-26T23:15:19.8543444Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 416, in __init__
2020-10-26T23:15:19.8544448Z     backend.openssl_assert(res == 1)
2020-10-26T23:15:19.8545851Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 244, in openssl_assert
2020-10-26T23:15:19.8547002Z     return binding._openssl_assert(self._lib, ok, errors=errors)
2020-10-26T23:15:19.8548478Z   File "/home/runner/work/twisted/twisted/.tox/alldeps-withcov-posix/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 90, in _openssl_assert
2020-10-26T23:15:19.8549464Z     errors_with_text,
2020-10-26T23:15:19.8553417Z cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=50901100, lib=3, func=139, reason=108, reason_text=b'error:0308B06C:bignum routines:BN_mod_inverse_no_branch:no inverse'), _OpenSSLErrorWithText(code=67665923, lib=4, func=136, reason=3, reason_text=b'error:04088003:rsa routines:RSA_setup_blinding:BN lib')])

[alex]

Are you able to get which version of OpenSSL is in use here?

[adiroiban]

@alex thanks for the update...

$ openssl version
OpenSSL 1.1.1f  31 Mar 2020

is there other way to find out which openssl is used.

$ python
Python 3.6.9 (7.3.1+dfsg-4, Apr 22 2020, 05:15:29)
[PyPy 7.3.1 with GCC 9.3.0] on linux
>>>> from OpenSSL import SSL
>>>> SSL.SSLeay_version(SSL.SSLEAY_VERSION), SSL.OPENSSL_VERSION_NUMBER
(b'OpenSSL 1.1.1f  31 Mar 2020', 269488239)

---

Is there a way to track down which tests / call is not leaving the stack clean.

There are random tests failure...so is not a single test that fails.

Also, the tests that fails is always green when executed alone.

The error is due to a side effect.

I tried disabling all the tests that are triggering an error...but it still fails.

I am continuing to track down what call is having he side effects

[alex]

The call that's failing matches up with the OpenSSL error stack, so I think it's a legitimate failure. Fair warning: I think this might be an OpenSSL bug!

If it only happens on PyPy, that contributes to my OpenSSL theory: all other Pythons get a wheel which statically links a newer OpenSSL, PyPy gets a distro OpenSSL. I'm about to compare the versions to see if maybe they fixed a bug in this area.

[adiroiban]

Thanks for the info.

On Python 3.8 with OpenSSL 1.1.1h the tests are green.

$ python
Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from OpenSSL import SSL
>>> SSL.SSLeay_version(SSL.SSLEAY_VERSION), SSL.OPENSSL_VERSION_NUMBER
(b'OpenSSL 1.1.1h  22 Sep 2020', 269488271)
>>> 

[alex]

Hokay. I claim openssl/openssl@35bb0e4 fixes this bug. Specifically, if you look at how pnoinv is now handled correctly in constant time BNs, and int_bn_mod_inverse no longer puts errors on the stack when they'll be handled in the caller.

Unfortunately, this really does mean it's an OpenSSL bug, and we're somewhat limited in our ability to fix it. Next step is maybe to see if the Ubuntu folks would be interested in backporting it I suppose.

[adiroiban]

Great work. Thanks for your help.

For Ubuntu, I guess that we only need to wait for an update.

Latest Ubuntu still has 1.1.1f https://packages.ubuntu.com/groovy/openssl

I see 1.1.1h was accepted to Debian Unstable https://alioth-lists.debian.net/pipermail/pkg-openssl-devel/2020-October/007393.html ...so I guess at some point it will also reach Ubuntu

[alex]

I think it'd be appropriate to ask Ubuntu/Debian to backport this fix to their stable releases.

…

On Tue, Oct 27, 2020 at 10:42 AM Adi Roiban ***@***.***> wrote: Great work. Thanks for your help. For Ubuntu, I guess that we only need to wait for an update. Latest Ubuntu still has 1.1.1f https://packages.ubuntu.com/groovy/openssl I see 1.1.1h was accepted to Debian Unstable https://alioth-lists.debian.net/pipermail/pkg-openssl-devel/2020-October/007393.html ...so I guess at some point it will also reach Ubuntu — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5521 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBBKJA2BVF65B7DVMRDSM3L3ZANCNFSM4TA2KXAA> .

-- All that is necessary for evil to succeed is for good people to do nothing.

[xnox]

I've opened https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1901723 to track this.

[alex]

@xnox thank you.

[gpotter2]

Thanks a lot for looking into this. If I may contribute a small & reproducible example (instead of running the entire twisted test suite 😄)

from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.backends import default_backend
rsa.RSAPublicNumbers(n=10, e=3).public_key(default_backend())

[alex]

This appears to also produce the following error, which is pretty cool:

[
  _OpenSSLErrorWithText(code=50843714, lib=3, func=125, reason=66, reason_text=b'error:0307D042:bignum routines:BN_mod_exp_recp:called a function you should not call'),
  _OpenSSLErrorWithText(code=67665923, lib=4, func=136, reason=3, reason_text=b'error:04088003:rsa routines:RSA_setup_blinding:BN lib')
]

[alex]

After discussion with @reaperhulk, I think we're going to disable blinding for public keys as a short term mitigation. Blinding will remain enabled for private keys, so hopefully ubuntu will still backport the fix.