Support using OpenSSL in FIPS mode

FIPS mode support needs to be enabled at compile time, by configuring
Erlang/OTP with --enable-fips option. In FIPS mode the non-FIPS
algorithms are disabled and raise error notsup. The rest of the
functions are rewritten to use the high level EVP interface when
compiled in FIPS mode. Non-FIPS build still use the old code path.

The supported protocols list is properly updated in FIPS-mode to
advertise only the enabled protocols.

FIPS mode is by default even if Erlang/OTP was built with FIPS
support. It needs to be turned on at runtime.

The official approach is to set the fips_mode application environment
parameter of the crypto application to true. This would turn FIPS mode
on when the NIF is loaded and would prevent loading the module on
error.

An other method is provided via the crypto:enable_fips_mode/1
function, but it is not recommended to be used in production, as it
won't prevent the use of the crypto module in case of an error, and
would risk OpenSSL crashing the emulator. It is very useful for test
suites however that need to check both validated and not validated
functionality.

Some implementation details:

Using the EVP interface raises similar issues with context copying as
in commit 651b949. However, since FIPS mode require OpenSSL 1.0.1
backward compatibility with 0.9.8 is not an issue here, so we are free
to provide a functional interface by context copying. The downside is
that cipher states are no longer compatible between FIPS and non-FIPS
builds, and that the new nif resource context values have a much more
limited use in the Erlang VM (e.g. you cannot persist them).

The digest_type_t structure used to store the NID of the hash
function, but it is now changed to the function that returns the
EVP_MD pointer. The reason is that the NID can be retrieved from the
EVP_MD structure (the type field), but EVP_get_digestbynid cannot be
used reliably to retrieve the EVP_MD from the NID.

configure.in

@@ -266,6 +266,10 @@ AC_ARG_ENABLE(dynamic-ssl-lib,
 AS_HELP_STRING([--disable-dynamic-ssl-lib],
                [disable using dynamic openssl libraries]))
 
+AC_ARG_ENABLE(fips,
+AS_HELP_STRING([--enable-fips], [enable OpenSSL FIPS mode support])
+AS_HELP_STRING([--disable-fips], [disable OpenSSL FIPS mode support (default)]))
+
 AC_ARG_ENABLE(builtin-zlib,
 AS_HELP_STRING([--enable-builtin-zlib],
                [force use of our own built-in zlib]))

erts/configure.in

@@ -3985,6 +3985,7 @@ dnl If set to --with-ssl=PATH we use that path as the prefix, i.e. we
 dnl use "PATH/include" and "PATH/lib".
 
 AC_SUBST(SSL_INCLUDE)
+AC_SUBST(SSL_DEFINE)
 AC_SUBST(SSL_ROOT)
 AC_SUBST(SSL_LIBDIR)
 AC_SUBST(SSL_CRYPTO_LIBNAME)
@@ -4627,6 +4628,31 @@ if test "$SSL_APP" != "" && test "$SSL_DYNAMIC_ONLY" = "yes" && \
     test "$rpath" != "" || AC_MSG_WARN([Cannot set run path during linking])
 fi
 
+AC_ARG_ENABLE(fips,
+AS_HELP_STRING([--enable-fips], [enable OpenSSL FIPS mode support])
+AS_HELP_STRING([--disable-fips], [disable OpenSSL FIPS mode support (default)]),
+[ case "$enableval" in
+    yes) enable_fips_support=yes ;;
+    *)   enable_fips_support=no ;;
+  esac ], enable_fips_support=no)
+
+if test "x$enable_fips_support" = "xyes" && test "$CRYPTO_APP" != ""; then
+   saveCFLAGS="$CFLAGS"
+   saveLDFLAGS="$LDFLAGS"
+   saveLIBS="$LIBS"
+   CFLAGS="$CFLAGS $SSL_INCLUDE"
+   LDFLAGS="$LDFLAGS $SSL_LD_RUNTIME_LIBRARY_PATH -L$SSL_LIBDIR"
+   LIBS="-lcrypto"
+   AC_CHECK_FUNC([FIPS_mode_set],
+   [SSL_DEFINE="-DFIPS_SUPPORT"],
+   [SSL_DEFINE=])
+   CFLAGS="$saveCFLAGS"
+   LDFLAGS="$saveLDFLAGS"
+   LIBS="$saveLIBS"
+else
+   SSL_DEFINE=
+fi
+
 #--------------------------------------------------------------------
 # Os mon stuff.
 #--------------------------------------------------------------------

lib/crypto/c_src/Makefile.in

@@ -42,9 +42,11 @@ SSL_LIBDIR = @SSL_LIBDIR@
 SSL_INCLUDE = @SSL_INCLUDE@
 SSL_CRYPTO_LIBNAME = @SSL_CRYPTO_LIBNAME@
 SSL_SSL_LIBNAME = @SSL_SSL_LIBNAME@
+SSL_DEFINE = @SSL_DEFINE@
 
 
 INCLUDES = $(SSL_INCLUDE) $(DED_INCLUDES)
+CFLAGS += $(SSL_DEFINE)
 
 ifeq ($(TYPE),debug)
 TYPEMARKER = .debug