OpenSSL 3.0 and 3.1: md4 support is broken

[jtru]

After upgrading a host of mine to Debian 12 (amd64, packaging OpenSSL 3.0.9), I noticed that a python application cannot authenticate towards Active Directory using NTLM any more. The underlying reason is that NTLM auth requires md4 hashing, and python3.11's hashlib on Debian 12 does not expose support for md4 hashing any more. Upon further investigation, it seems that OpenSSL itself is not aware of that lack of support, as it advertises md4 in the dgst command:

$ openssl dgst -list | grep md4
-blake2b512                -blake2s256                -md4

The same is true on archlinux (x86_64, packaging OpenSSL 3.1.1) as of today. I also built OpenSSL 3.0.9 (with the default static build config, by invoking ./Configure -static) from source and verified that behavior to match my Debian 12 and archlinux observations.

All three OpenSSL builds mentioned fail in the same peculiar way when openssl md4 or openssl speed md4 is executed, namely so:

/tmp/openssl-3.0.9/apps/openssl md4; echo $?
Error setting digest
4047A23C547F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:341:Global default library context, Algorithm (MD4 : 99), Properties ()
4047A23C547F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:272:
1

$ /tmp/openssl-3.0.9/apps/openssl speed md4; echo $?
Doing md4 for 3s on 16 size blocks: md4 error!
00000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (MD4 : 91), Properties ()
00000000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:254:
version: 3.0.9
built on: Tue Jun 20 10:05:47 2023 UTC
options: bn(64,64)
compiler: gcc -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_BUILDING_OPENSSL -DNDEBUG
CPUINFO: OPENSSL_ia32cap=0x7ef8320b078bffff:0x400004219c91a9
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
md4                  0.00         0.00         0.00         0.00         0.00         0.00 
0

I don't know much about the inner working of OpenSSL, but I think this hints at some required API conversion not having been applied to the md4-related code.

[beldmit]

md4 is moved to the legacy provider and is not available in case when legacy provider is not explicitly loaded.

openssl dgst -md4 -provider legacy works for me
Also
openssl speed -provider legacy md4 and openssl speed -provider legacy -evp md4 work.

Be careful adding legacy provider to the global configuration file, BTW. If you explicitly enable legacy provider, you also have to explicitly enable default or fips provider, otherwise the system may become unreachable.

[jtru]

Indeed, the steps I followed to try enabling the legacy providers on this Debian 12 machine turned out to be incomplete. For the record and hopeful benefit of others, this is the additional content that /etc/ssl/openssl.cnf (which is where $OPENSSLDIR/openssl.cnf symlinks to on Debian) requires to make md4 work again:

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

With this, openssl md4 et al. returns to working condition, with no apparent loss of functionality in other areas of the library, I hope. Also, python's hashlib regains the md4 algorithm:

>>> import hashlib
>>> hashlib.algorithms_available
{'md4', 'shake_256', 'whirlpool', 'md5-sha1', 'sha512_224', 'sha3_384', 'sha3_512', 'sm3', 'sha224', 'sha3_224', 'shake_128', 'blake2b', 'sha512_256', 'sha3_256', 'ripemd160', 'blake2s', 'sha512', 'sha1', 'sha384', 'md5', 'sha256'}

Thanks for the explanation and the warning.