fix: SSLv3 handshake with openssl-1.0.2-fips fails

[jouho]

Resolved issues:

Resolves #4476

Description of changes:

We cannot perform SSLv3 handshake when built with OpenSSL-1.0.2-fips. The reason is documented below.
This change will allow the use of MD5 algorithm in FIPS mode, which fixes handshake failures we were observing in #4476, as well as allowing SHA1 to be used for HMAC in SSLv3.

Call-outs:

Root cause

MD5 is generally not FIPS compliant and the client tries to initialize hash with MD5

s2n-tls/crypto/s2n_hash.c

Lines 111 to 117 in a99299a

	bool s2n_hash_is_available(s2n_hash_algorithm alg)
	{
	switch (alg) {
	case S2N_HASH_MD5:
	case S2N_HASH_MD5_SHA1:
	/* return false if in FIPS mode, as MD5 algs are not available in FIPS mode. */
	return !s2n_is_in_fips_mode();

However we allow MD5 for FIPS if the digest has the correct flag to indicate that it is available for FIPS.

aws-lc is able to use MD5 because in the function call to check if MD5 is allowed for FIPS, we always enable MD5 if s2n-tls is built with aws-lc:

s2n-tls/crypto/s2n_evp.c

Lines 37 to 49 in a99299a

	S2N_RESULT s2n_digest_is_md5_allowed_for_fips(struct s2n_evp_digest *evp_digest, bool *out)
	{
	RESULT_ENSURE_REF(out);
	*out = false;
	#if !defined(OPENSSL_IS_BORINGSSL) && !defined(OPENSSL_IS_AWSLC)
	if (s2n_is_in_fips_mode() && evp_digest && evp_digest->ctx && EVP_MD_CTX_test_flags(evp_digest->ctx, EVP_MD_CTX_FLAG_NON_FIPS_ALLOW)) {
	/* s2n is in FIPS mode and the EVP digest allows MD5. */
	*out = true;
	}
	#else
	if (s2n_is_in_fips_mode()) {
	/* If s2n is in FIPS mode and built with AWS-LC or BoringSSL, there are no flags to check in the EVP digest to allow MD5. */
	*out = true;

In the failing case, EVP_MD_CTX_test_flags() returns false, which results in throwing S2N_ERR_HASH_INVALID_ALGORITHM

Solution

We can fix this issue by inserting the following block of code

// this enables md5 flag
if (s2n_is_in_fips_mode() && conn->actual_protocol_version < S2N_TLS12) {
    POSIX_GUARD(s2n_hash_allow_md5_for_fips(workspace));
}

right before this line to enable MD5:

s2n-tls/tls/s2n_prf.c

Line 160 in 7a7fae2

POSIX_GUARD(s2n_hash_init(md5, S2N_HASH_MD5));

As well as allowing the use of HMAC in FIPS mode when using SHA1:

s2n-tls/crypto/s2n_hmac.c

Lines 79 to 88 in 7a7fae2

	bool s2n_hmac_is_available(s2n_hmac_algorithm hmac_alg)
	{
	switch(hmac_alg) {
	case S2N_HMAC_MD5:
	case S2N_HMAC_SSLv3_MD5:
	case S2N_HMAC_SSLv3_SHA1:
	/* Some libcryptos, such as OpenSSL, disable MD5 by default when in FIPS mode, which is
	* required in order to negotiate SSLv3. However, this is supported in AWS-LC.
	*/
	return !s2n_is_in_fips_mode() || s2n_libcrypto_is_awslc();

Testing:

Confirmed all unit tests and existing failure case is now passing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[maddeleine]

case S2N_HMAC_SSLv3_SHA1: 
/* Some libcryptos, such as OpenSSL, disable MD5 by default when in FIPS mode, which is 
* required in order to negotiate SSLv3. However, this is supported in AWS-LC. 
*/ 
return !s2n_is_in_fips_mode() || s2n_libcrypto_is_awslc();

Have you checked to see if that is true? What happens if you remove that check? Where does the error occur?

[jouho]

case S2N_HMAC_SSLv3_SHA1: 
/* Some libcryptos, such as OpenSSL, disable MD5 by default when in FIPS mode, which is 
* required in order to negotiate SSLv3. However, this is supported in AWS-LC. 
*/ 
return !s2n_is_in_fips_mode() || s2n_libcrypto_is_awslc();

Have you checked to see if that is true? What happens if you remove that check? Where does the error occur?

I tried removing return !s2n_is_in_fips_mode() || s2n_libcrypto_is_awslc(); and the handshake now works, at least when I tried locally and there is no error. This probably means there is no logic inside oepnssl-1.0.2-fips to actually prevent the use of MD5.

[lrstewart]

This doesn't seem right to me. This code is gated on "s2n_hmac_is_available(S2N_HMAC_SSLv3_MD5)". How can S2N_HMAC_SSLv3_MD5 be considered available if you have to call a method that doesn't exist outside of this test (and really shouldn't) in order to use S2N_HMAC_SSLv3_MD5? That sounds like S2N_HMAC_SSLv3_MD5 being unavailable to me.

[jouho]

s2n_hmac_is_available() doesn't really check if the algorithm is compatible with certain libcrypto and simply checks if we can recognize the given input as an HMAC algorithm (especially after my change in s2n_hmac.c). So although this test is gated by that, FIPS mode still requires a specific allowance for MD5 to be used in FIPS mode. s2n_allow_md5_for_fips() usually gets called during the handshake to enable MD5 for FIPS, but in this test we are only testing the hash, so it needs to be manually called.

[lrstewart]

If s2n_hmac_is_available() doesn't check if the algorithm is compatible with the current libcrypto, that would be different behavior from all the other s2n-tls is_available() methods. Why would it be called "is_available" instead of something like "is_valid" if the intended behavior was only to check whether the input is valid?

If s2n_hmac_is_available() reports "true", I should be able to use that hmac algorithm.

[jouho]

I believe that was partially the intended behavior originally since it was specifically blocking the use of hmac in fips mode. This logic was removed in this PR because it doesn't seem like OpenSSL fips module prevents the use of MD5 in any ways, which does not align with the comment:

/* Some libcryptos, such as OpenSSL, disable MD5 by default when in FIPS mode, which is
 * required in order to negotiate SSLv3. However, this is supported in AWS-LC.
 */

I will go ahead and change the function name from s2n_hmac_is_available to s2n_hmac_is_valid

[jouho]

The failing case was actually caused by attempting to use SHA1 for hmac. I applied the suggested change so that s2n_hmac_is_available() still gate the use of MD5 in fips, but will allow SHA1. s2n_hmac_test confirms that it is able to initialize hmac with SHA1 without manually enabling the flag. We also have s2n_sslv3_test which tests SSLv3 handshake and can confirm that openssl-1.0.2-fips is able to do SSLv3 handshake.