BIO* pub = BIO_new(BIO_s_mem()); is failing in openssl 3.0 it worked fine with openssl 1.0.2zg

[aniljadaun]

Hi

BIO* pub = BIO_new(BIO_s_mem()); is not allocating any memory and ERR_print_errors_fp doesn't log any error . Can you please help me in getting this issue diagnosed ? i am using openssl 3.1 with inbuilt legacy module.

[mattcaswell]

So, by "is not allocating any memory", do you mean that the BIO_new call returns NULL? There aren't very many reasons that would occur without logging an error and none of them seem particularly likely!!

[aniljadaun]

yes that is right ,BIO_new returns null. Not sure how to debug this any way you suggest to diagnose the issue ?

[mattcaswell]

This is quite surprising. The BIO_new function is just a simple wrapper around the BIO_new_ex function which is defined here:

openssl/crypto/bio/bio_lib.c

Lines 81 to 124 in f4bc556

	BIO *BIO_new_ex(OSSL_LIB_CTX *libctx, const BIO_METHOD *method)
	{
	BIO *bio = OPENSSL_zalloc(sizeof(*bio));
	if (bio == NULL) {
	ERR_raise(ERR_LIB_BIO, ERR_R_MALLOC_FAILURE);
	return NULL;
	}
	bio->libctx = libctx;
	bio->method = method;
	bio->shutdown = 1;
	bio->references = 1;
	if (!CRYPTO_new_ex_data(CRYPTO_EX_INDEX_BIO, bio, &bio->ex_data))
	goto err;
	bio->lock = CRYPTO_THREAD_lock_new();
	if (bio->lock == NULL) {
	ERR_raise(ERR_LIB_BIO, ERR_R_MALLOC_FAILURE);
	CRYPTO_free_ex_data(CRYPTO_EX_INDEX_BIO, bio, &bio->ex_data);
	goto err;
	}
	if (method->create != NULL && !method->create(bio)) {
	ERR_raise(ERR_LIB_BIO, ERR_R_INIT_FAIL);
	CRYPTO_free_ex_data(CRYPTO_EX_INDEX_BIO, bio, &bio->ex_data);
	CRYPTO_THREAD_lock_free(bio->lock);
	goto err;
	}
	if (method->create == NULL)
	bio->init = 1;
	return bio;
	err:
	OPENSSL_free(bio);
	return NULL;
	}
	BIO *BIO_new(const BIO_METHOD *method)
	{
	return BIO_new_ex(NULL, method);
	}

As you can see most failure reasons do raise an error on failure (which should be visible via ERR_print_errors_fp). The one that doesn't is if the call to CRYPTO_new_ex_data fails. But taking the analysis further and looking at reasons why that might fail without an error only shows errors which we wouldn't expect to happen (e.g. malloc failure etc).

So, I'm at a bit of loss as to why you are seeing this, but it points to some fundamental problem.

Where did you get your copy of OpenSSL from? Did you download/build it yourself, or did it come from your OS or other 3rd party? The only way I can really suggest to further investigate this is to build a debug copy of OpenSSL and step through the BIO_new_ex function in the debugger and see where it fails.

[nhorman]

@aniljadaun additionally, can you share some reproducer code for this?