EVP_PKEY_new_raw_private_key(NID_ED25519) does not return fail on short key

[pemensik]

Detected by failures on strongswan test suite, reported on strongswan/strongswan#753.

// compile by: gcc -Wall -g test.c -o test $(pkg-config --libs --cflags openssl)
#include <openssl/evp.h>

int main(int argc, char *argv[])
{
        unsigned char data[] = { 234, };
        EVP_PKEY *p;
        p = EVP_PKEY_new_raw_private_key(NID_ED25519, NULL, data, 1 /*sizeof(data)*/);
        printf("PKEY: %p\n", p);
        return 0;
}

Above simple code raises assertion failure on openssl 3.0.0, but returned just NULL on openssl 1.1. It is deliberately passed there with too short data. I think it should not crash whole program but return error, just like previous versions.

test: malloc.c:2515: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
```.

[pemensik]

Related to MR #15643.

[mattcaswell]

Hmm. This is curious. In fact the code is attempting to fail at this point. It has detected the short key and is trying to add an error to the error stack indicating that the setup of the key has failed. The assertion failure is not coming out of OpenSSL at all, but seems to come from libc!! My first thought was that some kind of memory corruption was triggering this - but recompiling with address sanitizer switched on reveals no problems - and in fact the assertion failure goes away completely and the reproducer comes back with the NULL response as expected.

Is this a possible libc bug!!??

[mattcaswell]

Is this a possible libc bug!!??

Nope. This really is an OpenSSL problem.

[mattcaswell]

A memory corruption is occurring. I'm not sure why address sanitizer didn't pick it up...

[mattcaswell]

Fix for this in #17041

[pemensik]

Thank you for a quick solution. Requested backport in RHBZ #2023671 for Fedora.