Add compatibility for OpenSSL 1.1.0 pre5 and previous versions

[ghost]

Add some checks to know which OpenSSL version is used, to be sure the code follows the syntax of the version used (including and until OpenSSL version 1.1.0-pre5).
This is an update of PR #448, aiming to fix issue #439, which is now working like a charm :)
Waiting for review!

[dmiller-nmap]

Need to change tests to pass make check:

test/test-wildcard.c: In function ‘set_dNSNames’:
test/test-wildcard.c:256:9: warning: implicit declaration of function ‘M_ASN1_IA5STRING_new’ [-Wimplicit-function-declaration]
         gen_name->d.dNSName = M_ASN1_IA5STRING_new();
         ^
test/test-wildcard.c:256:29: warning: assignment makes pointer from integer without a cast [enabled by default]
         gen_name->d.dNSName = M_ASN1_IA5STRING_new();
                             ^
gcc -o test/test-wildcard -g -O2 -Wall -L/home/miller/openssl1.1-inst/lib test/test-wildcard.o ncat_core.o ncat_ssl.o sys_wrap.o util.o ncat_posix.o ncat_lua.o ../liblua/liblua.a ../nsock/src/libnsock.a ../nbase/libnbase.a -lssl -lcrypto -lpcap ./../liblua/liblua.a -lm -ldl 
test/test-wildcard.o: In function `set_dNSNames':
/home/miller/nmap/nmap/ncat/test/test-wildcard.c:256: undefined reference to `M_ASN1_IA5STRING_new'
collect2: error: ld returned 1 exit status

[dmiller-nmap]

When openssl is configured as a static library (./config no-shared no-dso), then Nping can't find EVP_sha256 during configure:

checking for EVP_sha256... no
configure: error: Your version of OpenSSL does not support SHA-256. Please install OpenSSL 0.9.8 or later.
configure: error: ./configure failed for nping

This error does not occur with OpenSSL 1.0.2g.

OpenSSL config line used:

./config --prefix=$HOME/openssl1.1-inst no-shared no-dso enable-ssl2 enable-ssl3 enable-weak-ssl-ciphers

[dmiller-nmap]

Note also that OpenSSL 1.1 now uses libpthread by default, so you need to specify -lpthread in the OPENSSL_LIBS to avoid:

/home/miller/openssl1.1-inst/lib/libcrypto.a(threads_pthread.o): In function `CRYPTO_THREAD_lock_new':
threads_pthread.c:(.text+0x25): undefined reference to `pthread_rwlock_init'

So in places where we build static OpenSSL (RPM, OS X) we should either add the no-threads config option or add -lpthread to OPENSSL_LIBS.

[ghost]

@dmiller-nmap , I've just run a make check which results in errors... I wonder why you are not encountering them too, so here is the pastebin link. Concerning libpthread, I am going to add the -lpthread in the configure.ac files.

If I configure openssl 1.1 with the same config line, it results in a similar but not identical output:

checking for EVP_sha256... yes
configure: error: Your version of OpenSSL does not support SHA-256. Please install OpenSSL 0.9.8 or later.
configure: error: ./configure failed for nping

In nping/configure.ac, you can see line 173 the problematic block of code:

OPENSSL_LIBS=
if test "$use_openssl" = "yes"; then
  AC_DEFINE(HAVE_OPENSSL)
  OPENSSL_LIBS="-lssl -lcrypto -lpthread"
  LIBS_TMP="$LIBS"
  LIBS="$LIBS $OPENSSL_LIBS"
  # Check whether the installed OpenSSL supports SHA-256 (ver 0.9.8 or later).

AC_CHECK_FUNC(EVP_sha256,, AC_MSG_ERROR([Your version of OpenSSL does not support SHA-256. Please install OpenSSL 0.9.8 or later.]))

  LIBS="$LIBS_TMP"
fi

I am suspecting that this check does not work anymore on OpenSSL 1.1, what do you think @dmiller-nmap ?

[dmiller-nmap]

This is all good except for merge conflicts. There are two:

1. nse_ssl_cert.cc has an adjacent change from r36038 and r36039. Should be simple to manually merge.
2. I applied the changes to Nping's configure script together with related changes to Nsock's configure and tests/Makefile. Just leave those changes out, since they're already done in r36060.

Once these conflicts are solved, please merge this and make a note in the CHANGELOG that OpenSSL 1.1 is supported.

[ghost]

I just mixed them up here. I am currently making the commit to the SVN trunk, so this change and the merge issues will be corrected in it. I'll make an announcement in the CHANGELOG right after and write the revision number in the conversation of this PR.

[ghost]

This PR has been successfully sent and merged with the SVN trunk. The revision holding the modifications is revision 36063. Revision 36064 holds the CHANGELOG entry saying that OpenSLL 1.1.0-pre5 is now supported by Nmap.
Don't hesitate to contact me if there's anything wrong with it.

Cheers,
Vincent