7.9.3 fails to build with OpenSSL 1.1.

[carlocab]

Describe the current behavior
Building Nmap 7.9.3 fails to build when using OpenSSL 1.1. The error is encountered while building ncat, but other parts of the Nmap source distribution are probably susceptible to the same problem.

Expected behavior
The build should succeed, as it did with 7.9.2.

The build fails on macOS versions 10.15, 11, and 12 (both on x64 and arm64 for 11 and 12) and on Ubuntu Linux 16.04 with the following error:

  http_digest.o: In function `make_response':
  http_digest.c:(.text+0xd4): undefined reference to `EVP_MD_CTX_create'
  http_digest.o: In function `make_nonce':
  http_digest.c:(.text+0x430): undefined reference to `EVP_MD_CTX_create'
  collect2: error: ld returned 1 exit status
  Makefile:120: recipe for target 'ncat' failed

The form of the error is slightly different on macOS, but the content is the same:

  Undefined symbols for architecture arm64:
    "_EVP_MD_CTX_create", referenced from:
        _make_nonce in http_digest.o
        _make_response in http_digest.o
  ld: symbol(s) not found for architecture arm64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  make[1]: *** [ncat] Error 1
  make: *** [build-ncat] Error 2

Build logs are available at https://github.com/Homebrew/homebrew-core/actions/runs/2976571844.

The problem is that EVP_MD_CTX_create is an API that was replaced with EVP_MD_CTX_new in OpenSSL 1.1.

EVP_MD_CTX_create is used here:

nmap/ncat/http_digest.c

Lines 136 to 139 in 140dd72

	#if OPENSSL_API_LEVEL < 10100
	#define EVP_MD_CTX_new EVP_MD_CTX_create
	#define EVP_MD_CTX_free EVP_MD_CTX_destroy
	#endif

which suggests that OPENSSL_API_LEVEL is not defined correctly. Setting

-DOPENSSL_API_COMPAT=10101

works around the issue, but this should not be needed.

The errant commit is likely 140dd72, but this does not cleanly revert so I haven't been able to attempt a build after reverting it.

I encountered this problem while building Nmap 7.9.3 for Homebrew at Homebrew/homebrew-core#109458.

[oh2fih]

I had the same issue & commenting out lines 136-139 of ncat/http_digest.c solved it.

The condition if OPENSSL_API_LEVEL < 10100 should be revised.

[thesamesam]

Frustratingly, openssl/macros.h only exists in OpenSSL 3+.

[dmiller-nmap]

Thanks for reporting this. We did not find any issues building with OpenSSL 1.1 prior to release, but I will look into improving these preprocessor directives to support other configurations.

[dmiller-nmap]

I have pushed a change to fix this issue. Our testing was done with OpenSSL 3.0 and OpenSSL 1.0.2, but not 1.1.1. The change switches to using OPENSSL_VERSION_NUMBER instead of OPENSSL_API_LEVEL since that is what the OpenSSL headers define for all versions.

[ulises2k]

I have the same problem on debian 10

Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-21-cloud-amd64
Architecture: x86-64

[oh2fih]

Did you try the fix from d6bea8d? This is probably not a distinct issue.

[ulises2k]

Apply all the changes and it works correctly. I compile correctly, thank you very much