NTLM-using test 91 fails on macOS with OpenSSL3

[bagder]

I did this

I reproduced this CI build error by installing brew install openssl@3 and then building curl on macOS.

$ ./runtests.pl 91
...
 91: protocol FAILED:
--- log/check-expected  2021-09-27 11:31:17.000000000 +0200
+++ log/check-generated 2021-09-27 11:31:17.000000000 +0200
@@ -11,7 +11,7 @@
 [CR][LF]
 GET /91 HTTP/1.1[CR][LF]
 Host: 127.0.0.1:50748[CR][LF]
-Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEAAAAAYABgAWAAAAAgACABwAAAABgAGAHgAAAAIAAgAfgAAAAAAAAAAAAAAhoIBAMIyJpR5mHpg2FZha5kRaFZ9436GAxPu0C5llxexSQ5QzVkiLSfkcpVyRgCXXqR+Am15ZG9tYWlubXlzZWxmY3VybGhvc3Q=[CR][LF]
+Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEAAAAAYABgAWAAAAAgACABwAAAABgAGAHgAAAAIAAgAfgAAAAAAAAAAAAAAhoIBAMIyJpR5mHpgFBoSh9ZjUs83Se7QqVMFHC5llxexSQ5QzVkiLSfkcpVyRgCXXqR+Am15ZG9tYWlubXlzZWxmY3VybGhvc3Q=[CR][LF]
 User-Agent: curl/7.80.0-DEV[CR][LF]
 Accept: */*[CR][LF]
 [CR][LF]

 - abort tests
TESTDONE: 1 tests were considered during 1 seconds.
TESTDONE: 0 tests out of 1 reported OK: 0%

TESTFAIL: These test cases failed: 91 

The diff starts at index 116 but is fine again from index 137 until the end of the line. curl is expected to send 2FZha5kRaFZ9436GAxPu0 but sends FBoSh9ZjUs83Se7QqVMFH instead.

I expected the following

Test 91 should work fine.

curl/libcurl version

curl 7.80.0-DEV (x86_64-apple-darwin20.6.0) libcurl/7.80.0-DEV (SecureTransport) OpenSSL/3.0.0 zlib/1.2.11 zstd/1.5.0 libidn2/2.3.2 nghttp2/1.45.1
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS Debug HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz MultiSSL NTLM NTLM_WB SSL TLS-SRP TrackMemory UnixSockets zstd

operating system

ProductName: macOS
ProductVersion: 11.5.2
BuildVersion: 20G95

[bagder]

The problem seems to origin in OpenSSL DES encryption calls within the Curl_ntlm_core_mk_lm_hash function:

curl/lib/curl_ntlm_core.c

Lines 398 to 432 in 06981ba

	CURLcode Curl_ntlm_core_mk_lm_hash(struct Curl_easy *data,
	const char *password,
	unsigned char *lmbuffer /* 21 bytes */)
	{
	CURLcode result;
	unsigned char pw[14];
	static const unsigned char magic[] = {
	0x4B, 0x47, 0x53, 0x21, 0x40, 0x23, 0x24, 0x25 /* i.e. KGS!@#$% */
	};
	size_t len = CURLMIN(strlen(password), 14);
	Curl_strntoupper((char *)pw, password, len);
	memset(&pw[len], 0, 14 - len);
	/*
	* The LanManager hashed password needs to be created using the
	* password in the network encoding not the host encoding.
	*/
	result = Curl_convert_to_network(data, (char *)pw, 14);
	if(result)
	return result;
	{
	/* Create LanManager hashed password. */
	#if defined(USE_OPENSSL) || defined(USE_WOLFSSL)
	DES_key_schedule ks;
	setup_des_key(pw, DESKEY(ks));
	DES_ecb_encrypt((DES_cblock *)magic, (DES_cblock *)lmbuffer,
	DESKEY(ks), DES_ENCRYPT);
	setup_des_key(pw + 7, DESKEY(ks));
	DES_ecb_encrypt((DES_cblock *)magic, (DES_cblock *)(lmbuffer + 8),
	DESKEY(ks), DES_ENCRYPT);

There are two calls to DES_ecb_encrypt in there. In the problematic case, on my mac, I can see that the second call gets the same output stored as the first call, thus the data at index 8 to 15 gets repeated identically as index 0 to 7. This does not happen when I run the same code on Linux. Both 64 bit, both using OpenSSL 3.0.0..

This is even reproducible in a smaller stand-alone example that I wrote based on the above mentioned function. Ripped out and put in a separate smaller file, built on mac like this (using my own opensslv3 build just to exclude the chances that the homebrew version played tricks with us, but the exact same output is seen with the homebrew installed version):

gcc des.c -lssl -lcrypto -L /Users/daniel/build-openssl/lib -I /Users/daniel/build-openssl/include -o des

I built the Linux version similarly:

gcc des.c -lssl -lcrypto -L /home/daniel/build-openssl/lib -I /home/daniel/build-openssl/include -o des

The test code: des.c

Test outputs

When run on mac:

 ./des 
Curl_ntlm_core_mk_lm_hash first stored
LMB  0: 55 29 02 03 1b ed e9 ef
Curl_ntlm_core_mk_lm_hash then stored
LMB  8: 55 29 02 03 1b ed e9 ef

When run on Linux:

$ ./des 
Curl_ntlm_core_mk_lm_hash first stored
LMB  0: 55 29 02 03 1b ed e9 ef
Curl_ntlm_core_mk_lm_hash then stored
LMB  8: aa d3 b4 35 b5 14 04 ee

[bagder]

Turns out it is the DES_set_key call here:

curl/lib/curl_ntlm_core.c

Line 153 in 06981ba

DES_set_key(&key, ks);

It fails in the second invoke, and returns -2.