Memory corruption in OpenSSL v3.0.2 on duplicate provider declaration

[baentsch]

Only in v3.0.2 the following happens:

# OPENSSL_MODULES=_build/lib OPENSSL_CONF=scripts/openssl-ca.cnf ./.local/bin/openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
double free or corruption (!prev)
Aborted (core dumped)

Prerequisites: _build/lib contains a softlink to a duplicate provider .so:

# ls -l _build/lib/
total 6472
lrwxrwxrwx 1 root root      16 May 17 06:54 oqsprovider.so -> oqsprovider.so.1
-rwxr-xr-x 1 root root 6625640 May 17 06:54 oqsprovider.so.0.5.0-dev
lrwxrwxrwx 1 root root      24 May 17 06:54 oqsprovider.so.1 -> oqsprovider.so.0.5.0-dev
lrwxrwxrwx 1 root root      14 May 17 07:07 oqsprovider2.so -> oqsprovider.so

that is referenced in the scripts/openssl-ca.cnf:

oqsprovider = oqsprovider_sect
oqsprovider2 = oqsprovider2_sect

[default_sect]
activate = 1

[oqsprovider_sect]
activate = 1
# This second provider instance can be activated (for testing) for example
# by creating a softlink with suitable name "oqsprovider2" to the originally
# created oqsprovider.{so|dylib|dll}
[oqsprovider2_sect]
activate = 1

This bug is reproducible in Ubuntu 22 (jammy) with the system-provided OpenSSL v3.0.2 as well as a built-from source version on tag "openssl-3.0.2". Here a link to a CI testrun demonstrating this behaviour and the full scripting required to reproduce (Open "Test" item as crash isn't treated as a bug in the script :-(

This bug does not occur when building openssl from source on branches/tags "master" or "openssl-3.1.0". FWIW, oqsprovider contains a build script that can facilitate such alternate builds easily, e.g., running MAKE_PARAMS=-j OPENSSL_BRANCH=openssl-3.1.0 ./scripts/fullbuild.sh.

This is an extreme test corner case not happening in normal operation (with just one oqsprovider) and as OpenSSLv3.0.2 anyway does not feature full provider signature support, this bug is no problem for that application -- but it might be indicative of some memory handling bug fixed in 3.1.0 that may be worth while backporting.

[t8m]

Are you able to reproduce it with the tip of 3.0 branch? 3.0.2 is quite an old one and there were numerous bug fixes.

[baentsch]

Are you able to reproduce it with the tip of 3.0 branch? 3.0.2 is quite an old and there were numerous bug fixes.

Good question. Answer: No: 3.0.9-dev does not show the problem. So let's hope Ubuntu catches up...

[mattcaswell]

You might want to open a bug with Ubuntu and tell them that it does not impact upstream:

https://launchpad.net/ubuntu/+source/openssl/+filebug

[baentsch]

You might want to open a bug with Ubuntu and tell them that it does not impact upstream:

https://launchpad.net/ubuntu/+source/openssl/+filebug

Done: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/2019970

[levitte]

"Only v3.0.2"... does that mean you've tried with v3.0.3 and v3.0.1? I'm a bit curious when this stopped being a bug, or rather, what, exactly, fixed it. It could also be that Ubuntu is interested in that fix...

[baentsch]

"Only v3.0.2"... does that mean you've tried with v3.0.3 and v3.0.1? I'm a bit curious when this stopped being a bug, or rather, what, exactly, fixed it. It could also be that Ubuntu is interested in that fix...

I used v3.0.2 as that version was built-in to our CI image (Ubuntu 22). Crash was confirmed by a build-from-source of the tag "openssl-3.0.2".

I now also built from source:
3.0.1: Same crash
3.0.3: Same crash
3.0.4: OK - no crash
3.0.9-dev (3.0): OK - no crash

Hope this helps.

[baentsch]

Closing as a fix is in all 3.x branch tips -- just not Ubuntu.

[baentsch]

FYI, just read a comment by the ubuntu team that made me very unhappy (from a security perspective):

[...] practice of not updating through minor openssl versions

I counted 24 OpenSSL CVE's (several of which with High Severity) that are not fixed due to this practice (since 3.0.2).

Would anyone reading here want to chime in on the discussion there (or create a more visible one)? Would it be possible to help alleviate the concerns the Ubuntu team has wrt updating OpenSSL (minor) releases?

[mattcaswell]

IIUC, Ubuntu do actually backport the CVE fixes, just not necessarily all the other bug fixes.

[t8m]

I think this is mostly artifact of history of OpenSSL "stable" releases not being stable enough. So large distributions just backport security fixes and other critical bug fixes but do not update to new patch releases. IMO we are now much better in that regard than in days of 1.0.x.

[baentsch]

IIUC, Ubuntu do actually backport the CVE fixes, just not necessarily all the other bug fixes.

Thanks for this information & Good to know: I'd never have guessed that they're going to these lengths (instead of simply upgrading to the next release). Edit/add: Also Thanks for your input at the Ubuntu forum!