sha512_256: add support for GnuTLS and OpenSSL

[Karlson2k]

This is a follow-up for PR #12897.

This PR adds support for SHA-512/256 digest calculation by TLS backends.
Currently only OpenSSL and GnuTLS (actually, nettle) support SHA-512/256.

This is an alternative implementation. The initial PR is #13053.

This version has more curl branding ;), simplified code (uses size_t for size instead of unsigned int as underling libs and avoids trying to process data in smaller chunks when size_t is larger than unsigned int), added error checking and reporting for the most used function Curl_sha512_256it(), renamed some symbols to avoid potential conflicts. With this PR code detects hashing problem reported by TLS backend and returns respective error.

Sorry, I couldn't resist improving the code even when my PR was already accepted.

[Karlson2k]

Check curl (windows msys2 mingw64_libssh) failure is just a docker failure.

[Karlson2k]

Test 707 failure in AppVeyor / CMake, VS2010, Debug, x64, no SSL, Static seems to be unrelated.

[bagder]

Thanks!

[dfandrich]

Test 1615 and the other SHA-512/256 unit tests started dumping core on NetBSD after this was submitted: https://curl.se/dev/log.cgi?id=20240308034313-2594315#prob4

[Karlson2k]

NetBSD 9.3 on ARM. Interesting.
@dfandrich Any chance to see stack trace or other details about test 1615?
Looks like some problems with OpenSSL, however it is not wrong features detection as linking is OK.

[dfandrich]

You'll have to contact Michael Kaufmann, who runs that build, directly for more info.

[Karlson2k]

@mkauf

NetBSD 9.3 on ARM. Interesting. @dfandrich Any chance to see stack trace or other details about test 1615? Looks like some problems with OpenSSL, however it is not wrong features detection as linking is OK.

@mkauf Could you please share this info?

[mkauf]

@Karlson2k : The backtrace is:

Core was generated by `unit1615'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x78587c10 in memcmp () from /usr/lib/libc.so.12
#1  0x00016e78 in test (arg=0x7ffeac8c "-") at unit1615.c:126
#2  0x7db61474 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[Karlson2k]

Thanks, @mkauf

This is something weird. SIGSEGV in memcmp() where two parameters are are pointers to the arrays with fixed size and the third parameter is the size of the array.
I'd understand if this macro would be zero, but it's not. It is a fixed number.

Is it possible to make stack back trace with debug symbols and frame pointers?
Also backtraces for other failed tests would be helpful.

[Karlson2k]

Here is the place where segmentation failure happens:

curl/tests/unit/unit1615.c

Lines 118 to 126 in fb3c251

	unsigned char output_buf[SHA512_256_DIGEST_LENGTH];
	unsigned char *computed_hash; /* Just to mute compiler warning */
	/* Mute compiler warnings in 'verify_memory' macros below */
	computed_hash = output_buf;
	Curl_sha512_256it(output_buf, (const unsigned char *) test_str1,
	(sizeof(test_str1) / sizeof(char)) - 1);
	verify_memory(computed_hash, precomp_hash1, SHA512_256_DIGEST_LENGTH);

[mkauf]

@Karlson2k I have discovered that NetBSD uses a patched version of OpenSSL which has a bug. I have reported it here: https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58039

Links to NetBSD source:

• https://github.com/NetBSD/src/blob/netbsd-9/crypto/external/bsd/openssl/lib/libcrypto/sha.inc
• https://github.com/NetBSD/src/blob/netbsd-9/crypto/external/bsd/openssl/lib/libcrypto/libc-sha2xx.c

The code in this pull request is fine, it's NetBSD's fault that curl crashes.

If you want to implement a workaround:

static CURLcode
Curl_sha512_256_finish(unsigned char *digest,
                       void *context)
{
  CURLcode ret;
  Curl_sha512_256_ctx *const ctx = (Curl_sha512_256_ctx *)context;

  /* Use a buffer size of EVP_MAX_MD_SIZE to work around a bug in NetBSD:
     https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=58039 */
  unsigned char tmp_digest[EVP_MAX_MD_SIZE];

  if(EVP_DigestFinal_ex(*ctx, tmp_digest, NULL)) {
    memcpy(digest, tmp_digest, SHA512_256_DIGEST_LENGTH);
    ret = CURLE_OK;
  }
  else {
    ret = CURLE_SSL_CIPHER;
  }

  EVP_MD_CTX_destroy(*ctx);
  *ctx = NULL;

  return ret;
}

[Karlson2k]

Thanks for digging into it and finding the root cause, @mkauf!

I'll make a workaround based on your suggestion. Thanks for the suggested code changes!

[Karlson2k]

I've created PR #13133 to workaround the bug.
The PR is based on the @mkauf suggestion.