Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]No supported data to decode. Input type: PEM #16696

Open
trippleflux opened this issue Sep 28, 2021 · 24 comments
Open

[Bug]No supported data to decode. Input type: PEM #16696

trippleflux opened this issue Sep 28, 2021 · 24 comments
Labels
branch: 3.0 Merge to openssl-3.0 branch inactive triaged: question The issue contains a question

Comments

@trippleflux
Copy link

trippleflux commented Sep 28, 2021

I have a RSA private key being generated by the following :

		ctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
		if (!ctx)
		{
			goto error_rsa;
		}
		EVP_PKEY_CTX_set_dh_paramgen_prime_len(ctx, 2048);
		EVP_PKEY_CTX_set_dh_paramgen_subprime_len(ctx, 2048);
		EVP_PKEY_CTX_set_dh_paramgen_generator(ctx, 5);
		EVP_PKEY_CTX_set_cb(ctx, MyOpenSSL3_Progress_Callback);
		EVP_PKEY_CTX_set_app_data(ctx, MyOpenSSL3_Progress_Callback);

		if (EVP_PKEY_keygen_init(ctx) <= 0)
		{
			goto error_rsa;
		}
		params[0] = OSSL_PARAM_construct_uint("bits", &rsa_bits_strength);
		params[1] = OSSL_PARAM_construct_end();
		EVP_PKEY_CTX_set_params(ctx, params);

		/*
		if (EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, rsa_bits_strength) <= 0)
		{
			goto error_rsa;
		}
		*/
		BN_GENCB_set(CallBack, MyOpenSSL_Progress_Callback, lpProgress);
		if (EVP_PKEY_generate(ctx, &PKey) <= 0)
		{
			(Out)(lpProgress, _T("%s%sFailed to generate RSA key for certificate (%s bits).\r\n"), lpProgress->tszBlank, lpProgress->tszPrefix, bits);
			goto error_rsa;
		}

example generated RSA Private key :

-----BEGIN PRIVATE KEY-----
MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQC3W6Bjuul+GE65
Go/i5VJDc3mVSeKMW345aqLKIgmLJOJJrklQd43MUuGvfQStvkmZxG2ifWx0KR17
H8+LMU/1v9aWsDmXJTaobB0Oi1rK/NhZrP/29KKNBoeiGjocTw51U6/83CpyzKrU
vtz9GSP0HhFFWWpsZtkDwmiW5v2jmgyqVJK97nOsnk6ujwL/sVwKyTHFN3I/j55y
1F2Mqc8pSZuAdMKMCzvvHhf0Ef8bAYzfVWQXMsus3juCliu+5q3KzZKWiLuPzxPZ
mzD6IRrwrSmibAAV8PCgzH+PK6sn27XNMqXHfZXWCb24J+/Iu4N1tlbmRS6SIG9h
vRm29R/bv4jzzJ4oZZb5vF0ZeUqAK3ONuzxgB4Qiodm3u5b8HlAYRAzpvBQb6I5n
Xk6QhGqX47V5gLvyLrZRfSE8dO7AQphMI5vpQi1JV+KRZImLf72ED9A6KMcSHSKG
PJ4SW+IsSmQ2dywqhZhlFy0KXfUdJhVwQqMTaYGmFtFwsFIB6OiQCtX33Uk1lPp7
Xk7tDeLZT/6hskKXC1GOZ9toaGVuG+P2gVu5wLuG9sx3sFd/Ahp8O7iG98KDiBVA
LhPGJ6rzs3Uo5pk5xrZMIb4yPeaI5H8fvTR6ciUaK2BNt3JjrivEddxq2HFvT7bm
xmt6PA7DilNwgauE5V3wycP0tnNjKwIDAQABAoICAFWLHmuY5tlo3sV2Mn3xbi1A
4Q74zwSbpOqFqRgzxA8SMK6ebN7pQBL88gTtBzNF7VGbdmjSKjiRfAsW3r26TmtB
Soy3/85X0sXF1tKTylKcnxVLZInjPTkdPEcxBD5z1bOATUFjuDwh52tk9nXwdjks
k/MuUbvsSb/6wc/xSZE7zD/lQCbzai01GFFRMF15cg8oKZuwkBE+YHaDRHfdO2Sw
TieikadDkH0xJYy9nq6LTR1e2dkG3kYy0kHM7PDd72QH/6VSemsP+dtSj4mKy2ZQ
63U2/N0/sQvcDIEPpwTr+MgYooZNPYwDtGsH2tNkRGM1JBq/WQhcsIpBbo9QkkEp
UAd71juh9dZa9OYF7jd1a9wfbyzFz+DMhY2i6WFoz4RCjaK1g1HjEuY8e5oa8+HJ
XuHYfx7omt4dUc509O7Bwjq9FlYgx1rnVXeu2ujggLBCqR1RLi1+dKyQnv+ufRWA
5m317+mY7CVy5qbRk1TY5ljdqtFYi95/1hfXSDbzJN0zaulP6+X9kVZTv2oatJk/
m+WGf9z5lS2iZzKJSZwYVPB6fJrHayxhLKlJ3s90HtheSKydKYl3IUnRzhUcRiKF
6klF7brzYbLCdLn96KG2BTYbfZIERya3iYGDwQVB/tq2nHSx7jFuYZYllJUrWEyc
nKUMPgjBnNmJuFdTJkH1AoIBAQDJsGR+eAh449mTCCfmTh1FohI8nZOzJQEI9jmD
KtQKNnUcy/GIqdj3McAjO9Dlp7iW/4kqtL6D65zWPUImGFeFbJQh+zBLpq0X4ZFg
bc++WNLMLXiO1cennzRyaUl41LzK1rDq7buseSTkcgzn9ocs7fIObDco5l9o8EqX
LrWRKvfeRa5B6T1izC+9rZ7OrauJgXwdOMkiSgQUDQNZAqI15T9pMk7oUMuWB3wx
TR6tvc8M1iTLiTNHloQejOhC27+HutUOayl4IwPWA9xRFgTWLZ0U/NCJPbdDO7gT
159frFMD60FpYYg41K29BaxukVuSUnBFpOGK9QchC6KKHzA1AoIBAQDou5AFsq+m
J9fEu0M58KYREsF3lbDr7dfgVIChzSXJd2PlmZnVu4YTfr6Nz0Fb8Ez1jbGLvlsV
ghZWyhdYtbuaJMT6K9mi9TdzBRtTPPRUo7p5t1hOygEuDfrZazvw0sjjwyIir5gb
J9iwp75Zg2+2UlV902ApVkN6p4jgWXasFJ4YbSH8BPaHtrm8gUokrVT1OnPrDmwP
DHYwt7N2XFm4XQbu28VP+KnrAK8zGWLHdZGU1qmcMfT86IqCmAkmAGH0411kSOY2
UGqtVTs5LDNFP43ULDbGDQ83Os85dYqn+eJOL1fsns0O2IGLBJlJEkk0NtLWq1ig
HJ6IxmDfHXHfAoIBAAmitrpgC6+S2yMuIZHcK4PXIV3p/VNysiD2enrsweKXBXNR
vwFTc/0jUFWelvVR1PzJyojSx8NPr1eA57cgNJY4QL9626zRkrVwXlyoAmPKpI+C
oJ7aUcaNUZkhtJPkPfzG0v72SqALzZpoZHt0iLK5z3Wm79+xWd+Ok1GztpLv5ZWl
6JRXi0o8DQ/Xyp1/KJHSMNbIjrtno9g9YuqOuKcwg54H/EMXERLi2qkUUSHlzo2D
cZCTxnJmG3vGKZaAHx9h6qncrseBjL5dwtnQ3nxJA5csGN5lp2q74Kh6BbAxcGng
hGh/c12bs40zRzClGyXbcGezX98eo6+Neb91i/0CggEBANKom9ecgwNaAyNvAY3b
duXzXS1aUHdtpT7IHlrWr4Jp0AGOhEGDaPwSeHcyB44pt+94XyIHnricNI5AhNtm
PDZtW+K6eFfZsGQrVi2E+f/2556ux38ht+mw+8N7HhmBIhePlEFjQcIq0igs3c52
e1LDueCXyO4ByDioluvD7vgZ/XRMrDtLXr8/NWYcfTkvr7v9OdmlB0ij1gOI9CeJ
tih8A/+G7FXT1jICP5N4QgIbODhKCeJ5EbEu59kHtxgvtnsMpodUuRSmu7jDckTw
7kObIEzR7I5eWcx7LgFrlKMxO5fboLFsmF38uV1VP051/88EZNms5R6gaRuSvTKX
m28CggEAMmE1GmiCH2nXFlL9JrK5xrRJDowBvn9+tmks3e00CIdTOhS1Wv/VV/WA
J48ZpFsASxjOxO8ssi1b96B9HB98OlqU8cRA7ODFKWn2X2ep05rzRFqvO7Zh60qU
C9AI/HMHEGCdE45bkGJMSaMcCBDtwrQxJ3sf86hdIr6Sp8EBogBBHt0S+MXVAsHZ
YdooLS38WYH6v7GZYxvANvHzFFBwW6WjSNNfIY2DzWLJ7w8ZhNWfxW5KamPUfEOr
YUhnfzBvTC6AJiGrHGQ4kuMW/xLr+604hevgPbcKKKcQi66sbzLMLgg0e9pTsg6d
UvY99Uq/Zq7fGYRqvXL3GBvvC8DmTg==
-----END PRIVATE KEY-----

the certficate file :

-----BEGIN CERTIFICATE-----
MIIEmTCCAoECBGFSXVwwDQYJKoZIhvcNAQELBQAwETEPMA0GA1UEAwwGaW9GVFBE
MB4XDTIxMDkyNzAwMTAwNFoXDTMxMDkyNjAwMTAwNFowETEPMA0GA1UEAwwGaW9G
VFBEMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt1ugY7rpfhhOuRqP
4uVSQ3N5lUnijFt+OWqiyiIJiyTiSa5JUHeNzFLhr30Erb5JmcRton1sdCkdex/P
izFP9b/WlrA5lyU2qGwdDotayvzYWaz/9vSijQaHoho6HE8OdVOv/Nwqcsyq1L7c
/Rkj9B4RRVlqbGbZA8Jolub9o5oMqlSSve5zrJ5Oro8C/7FcCskxxTdyP4+ectRd
jKnPKUmbgHTCjAs77x4X9BH/GwGM31VkFzLLrN47gpYrvuatys2Sloi7j88T2Zsw
+iEa8K0pomwAFfDwoMx/jyurJ9u1zTKlx32V1gm9uCfvyLuDdbZW5kUukiBvYb0Z
tvUf27+I88yeKGWW+bxdGXlKgCtzjbs8YAeEIqHZt7uW/B5QGEQM6bwUG+iOZ15O
kIRql+O1eYC78i62UX0hPHTuwEKYTCOb6UItSVfikWSJi3+9hA/QOijHEh0ihjye
ElviLEpkNncsKoWYZRctCl31HSYVcEKjE2mBphbRcLBSAejokArV991JNZT6e15O
7Q3i2U/+obJClwtRjmfbaGhlbhvj9oFbucC7hvbMd7BXfwIafDu4hvfCg4gVQC4T
xieq87N1KOaZOca2TCG+Mj3miOR/H700enIlGitgTbdyY64rxHXcathxb0+25sZr
ejwOw4pTcIGrhOVd8MnD9LZzYysCAwEAATANBgkqhkiG9w0BAQsFAAOCAgEArevH
k3mRVoS4jrjmuLtbFMSXnd1QceSVSaMIJclLvrJDHZAzAoG0pSh9JAtWzri7EWRK
W1UbL9l8ng0zZ6UcO/rbulbRL5Vj11TnOjf8hiVbpuOR+37PJ8h7aocEH1vSV4zy
OlDRt3ja7SvgspphOzHJeTOCPnHBU9ylTprO+XuoUqTRTm9vwYZ741sfrxSJPT/n
VL9VRWNcrwjxYc/4EtTexOK2noSrgxiO48jmlzFGL4BNYed4NRSeR1bGOMcoXLJF
mOueFf8L0+aV8sUL12tOuios5KUchOLoqZOAGwjObEykNGKODqQMTrWuq0K715lg
BegQ0D8EoNuKTLPFXVmkdTjU8r6s87o1AezwwG7YHXfQLyfktKZkSox+dTZivfWv
9qPtOAlPTfZS+6p5kQfeLLNrQDQp0UXmBhpqBphOLByW8AKI9qzEIFP05qBlDIF5
z/yWhmVSjEgyxAiM4xJv91JUcBbVbGECouxwaKYt6q+znPhTh/5AK34cJukbEkv6
mtSuU7onD3yebDqSZLEek4gN/pDBPJ4m2Nb+9Ca+yTKtchJ9zt0lRLovlAElm8ct
YU71RyDRkmMJ4Q/fWgC8SJjmvmlPhwshks/8zR+COg66VPrQTiWu4ytvp1ZI4/BD
YMYN1SuK1R2p8SISusIQGUguMXQV4tbB1p4HY60=
-----END CERTIFICATE-----

When calling it through function below :

	if (!SSL_CTX_use_certificate_chain_file(pCtx, szFileName))
	{
		SSL_CTX_free(pCtx);
		return TRUE;
	}
	if(SSL_CTX_use_PrivateKey_file(pCtx, szFileName, SSL_FILETYPE_PEM) != 1)
	{
		FILE* fp;
		fp = fopen("errors.log", "w");
		ERR_print_errors_fp(fp);
		fclose(fp);
		SSL_CTX_free(pCtx);
		return TRUE;
	}

I am getting the following error from OpenSSL :

A0210000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
A0210000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

The Same exact function is working properly on OpenSSL 1.1.1 branch, is this a bug or wrong code?

@trippleflux trippleflux added the issue: question The issue was opened to ask a question label Sep 28, 2021
@paulidale paulidale added branch: 3.0 Merge to openssl-3.0 branch triaged: bug The issue/pr is/fixes a bug and removed issue: question The issue was opened to ask a question labels Sep 28, 2021
@slontis
Copy link
Member

slontis commented Sep 28, 2021

Calling DH setters on a RSA key is not a good idea..
(e.g. EVP_PKEY_CTX_set_dh_paramgen_prime_len)

Use the RSA setters.. You also dont check the return value from any of these calls..

@paulidale paulidale added resolved: answered The issue contained a question which has been answered triaged: question The issue contains a question and removed triaged: bug The issue/pr is/fixes a bug labels Sep 28, 2021
@trippleflux
Copy link
Author

@slontis
I have commenting in the DH setters :

		//EVP_PKEY_CTX_set_dh_paramgen_prime_len(ctx, 2048);
		//EVP_PKEY_CTX_set_dh_paramgen_subprime_len(ctx, 2048);
		//EVP_PKEY_CTX_set_dh_paramgen_generator(ctx, 5);

But still getting :

C8570000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
C8570000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

@slontis
Copy link
Member

slontis commented Sep 28, 2021

I just took your example RSA private key data and put i in a pem file..
The following code works fine for me on master..

SSL_CTX *sctx = SSL_CTX_new_ex(NULL, NULL, TLS_server_method());
SSL_CTX_use_PrivateKey_file(sctx, "rsa.pem", SSL_FILETYPE_PEM);

@slontis
Copy link
Member

slontis commented Sep 29, 2021

Did you find your issue?

@trippleflux
Copy link
Author

trippleflux commented Sep 29, 2021

@slontis
I cannot figuring it out why on mine doesn't work without restarting the app,

After the private key generation in code above, I am calling the following :

	char     szFileName[MAX_PATH+1];
	SSL_CTX *pCtx;

	if (! szCertificateName) return TRUE;

	sLen = strlen(szCertificateName);
	if (sLen+6 > sizeof(szFileName))
	{
		SetLastError(ERROR_FILENAME_EXCED_RANGE);
		return TRUE;
	}
	_snprintf_s(szFileName, sizeof(szFileName)/sizeof(*szFileName), _TRUNCATE, "%s.pem", szCertificateName);

	pCtx = SSL_CTX_new(TLS_method());
	if (!pCtx) return TRUE;

	if (!SSL_CTX_use_certificate_chain_file(pCtx, szFileName))
	{
		SSL_CTX_free(pCtx);
		return TRUE;
	}

	strcpy_s(&szFileName[sLen], sizeof(szFileName)/sizeof(*szFileName)-sLen, ".key");

	// SSL_CTX_set_default_passwd_cb(ctx, cb);

	if(!SSL_CTX_use_PrivateKey_file(pCtx, szFileName, SSL_FILETYPE_PEM))
	{
		FILE* fp;
		fp = fopen("errors.log", "w");
		ERR_print_errors_fp(fp);
		fclose(fp);
		SSL_CTX_free(pCtx);
		return TRUE;
	}

But always getting :

AC110000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
AC110000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

I have setup a breakpoint on SSL_CTX_use_PrivateKey_file, before reaches SSL_CTX_use_PrivateKey_file, the szFileName or Server.key is already exist, and I can open it in Notepad++ while when reading it through SSL_CTX_use_PrivateKey_file is failed with above error.

I don't have these problem whenever using 1.1.1 branch and below. And also I cann't reproduce my report if I am restarting my app and the RSA certificate, private, public is already generated/exist.

Can you try to generate the certificates (RSA certificate, private, public) and calling function above?, especially the SSL_CTX_use_PrivateKey_file

EDIT :
RSA Private key was written into a file (Server.key) by using this code :

		_sntprintf_s(tszFileName, sizeof(tszFileName) / sizeof(*tszFileName), _TRUNCATE, _T("%s.key"), szCertName);
		if (err = _tfopen_s(&File, tszFileName, _T("wN")))
		{
			goto error_rsa;
		}


		if (!PEM_write_PKCS8PrivateKey(File, PKey, NULL, NULL, 0, NULL, NULL))
		{
			goto error_rsa;
		}
		fclose(File);
		File = NULL;

@trippleflux trippleflux reopened this Sep 29, 2021
@slontis
Copy link
Member

slontis commented Sep 30, 2021

You could try the following code which works for me locally...

    int ret = 0;
    BIO *out = BIO_new_file("rsa.pem", "w");
    SSL_CTX *sctx = NULL;
    OSSL_PARAM params[2];
    unsigned int rsa_bits_strength = 4096;
    EVP_PKEY_CTX *pctx = NULL;
    EVP_PKEY *pkey = NULL;

    pctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
    if (pctx == NULL)
        goto err;
    if (EVP_PKEY_keygen_init(pctx) <= 0)
        goto err;

    params[0] = OSSL_PARAM_construct_uint("bits", &rsa_bits_strength);
    params[1] = OSSL_PARAM_construct_end();
    if (!EVP_PKEY_CTX_set_params(pctx, params))
        goto err;

    if (EVP_PKEY_generate(pctx, &pkey) <= 0)
        goto err;
    if (!PEM_write_bio_PKCS8PrivateKey(out, pkey, NULL, NULL, 0, NULL, NULL))
        goto err;

    BIO_free(out);
    out = NULL;

    sctx = SSL_CTX_new_ex(NULL, NULL, TLS_server_method());
    if (sctx == NULL)
        goto err;
   
    if (!SSL_CTX_use_PrivateKey_file(sctx, "rsa.pem", SSL_FILETYPE_PEM))
        goto err;
    ret = 1;
err:

It might be windoze file related...

@slontis slontis removed the resolved: answered The issue contained a question which has been answered label Sep 30, 2021
@trippleflux
Copy link
Author

In my part it created only 0 kb file, could it be because I have custom certificate things goin on, here is the complete part of the code :

		if (EVP_PKEY_generate(ctx, &PKey) <= 0)
		{
			(Out)(lpProgress, _T("%s%sFailed to generate RSA key for certificate (%s bits).\r\n"), lpProgress->tszBlank, lpProgress->tszPrefix, bits);
			goto error_rsa;
		}
		/*if (!RSA_generate_key_ex(pRsa, rsa_bits_strength, pBN_e, CallBack))
		{
			(Out)(lpProgress, _T("%s%sFailed to generate RSA key for certificate (%s bits).\r\n"), lpProgress->tszBlank, lpProgress->tszPrefix, bits);
			goto error_rsa;
		}
		*/

		if (lpProgress->lpIoSocket && lpBuffer->len)
		{
			SendQuick(lpProgress->lpIoSocket, lpBuffer->buf, lpBuffer->len);
			lpBuffer->len = 0;
		}
		/*
		if (!EVP_PKEY_assign_RSA(PKey, pRsa))
		{
			goto error_rsa;
		}
		// Rsa now owned by PKey so we don't want to free it
		pRsa = NULL;
		*/
		if (!ASN1_INTEGER_set(X509_get_serialNumber(pX509), (LONG)time(NULL)))
		{
			goto error_rsa;
		}

		// make it valid as of yesterday to handle clients in a timezone where it's yesterday locally :)
		X509_gmtime_adj(X509_get_notBefore(pX509), -60 * 60 * 24);

		X509_gmtime_adj(X509_get_notAfter(pX509), (long)60 * 60 * 24 * 365 * 10); // ten years

		if (!X509_set_pubkey(pX509, PKey))
		{
			goto error_rsa;
		}

		// get the field where we can set Country, Name, etc...
		xName = X509_get_subject_name(pX509);

		// Add name of cert to subject line
		if (!X509_NAME_add_entry_by_txt(xName, "CN", MBSTRING_ASC, szCertName, -1, -1, 0))
		{
			goto error_rsa;
		}

		// We are the issuer since self signing it
		if (!X509_set_issuer_name(pX509, xName))
		{
			goto error_rsa;
		}

		if (!X509_sign(pX509, PKey, EVP_sha256()))
		{
			goto error_rsa;
		}

		if (!X509_verify(pX509, PKey))
		{
			(Out)(lpProgress, _T("%sFailed to verify RSA key for certificate.\r\n"), lpProgress->tszPrefix);
			goto error_rsa;
		}


		if (!(pDH = DH_new()))
		{
			goto error_rsa;
		}


		(Out)(lpProgress, _T("%s%sGenerating 2048 bit Diffie-Huffman parameters:\r\n"), lpProgress->tszBlank, lpProgress->tszPrefix);
		if (lpProgress->lpIoSocket && lpBuffer->len)
		{
			SendQuick(lpProgress->lpIoSocket, lpBuffer->buf, lpBuffer->len);
			lpBuffer->len = 0;
		}

		if (!DH_generate_parameters_ex(pDH, 2048, 5, CallBack))
		{
			(Out)(lpProgress, _T("%sFailed to generate DH key for certificate.\r\n"), lpProgress->tszPrefix);
			goto error_rsa;
		}


		if (!DH_check(pDH, &i))
		{
			goto error_rsa;
		}

		if (i & DH_CHECK_P_NOT_PRIME)
		{
			(Out)(lpProgress, _T("%sFailed to generate a prime.\r\n"), lpProgress->tszPrefix);
			goto error_rsa;
		}
		if (i & DH_CHECK_P_NOT_SAFE_PRIME)
		{
			(Out)(lpProgress, _T("%sFailed to generate a safe prime.\r\n"), lpProgress->tszPrefix);
			goto error_rsa;
		}
		if (i & DH_UNABLE_TO_CHECK_GENERATOR)
		{
			(Out)(lpProgress, _T("%sFailed to validate generator.\r\n"), lpProgress->tszPrefix);
			goto error_rsa;
		}
		if (i & DH_NOT_SUITABLE_GENERATOR)
		{
			(Out)(lpProgress, _T("%sInvalid generator.\r\n"), lpProgress->tszPrefix);
			goto error_rsa;
		}
		
		Secure_Delete_Cert(szCertName);

		_sntprintf_s(tszFileName, sizeof(tszFileName) / sizeof(*tszFileName), _TRUNCATE, _T("%s.key"), szCertName);
		if (err = _tfopen_s(&File, tszFileName, _T("wN")))
		{
			goto error_rsa;
		}

		if (!PEM_write_bio_PKCS8PrivateKey(out, PKey, NULL, NULL, 0, NULL, NULL))
		{
			goto error_rsa;
		}

		BIO_free(out);
		out = NULL;
		fclose(File);
		File = NULL;

		Putlog(LOG_DEBUG, _T("Finished generating Private Key!\r\n"));
		_tcscpy_s(&tszFileName[sLen], sizeof(tszFileName) / sizeof(*tszFileName) - sLen, ".pem");
		if (err = _tfopen_s(&File, tszFileName, _T("wN")))
		{
			goto error_rsa;
		}

		if (!PEM_write_X509(File, pX509))
		{
			goto error_rsa;
		}
		fclose(File);
		File = NULL;
		Putlog(LOG_DEBUG, _T("Finished generating Certificate!\r\n"));

		_tcscpy_s(&tszFileName[sLen], sizeof(tszFileName) / sizeof(*tszFileName) - sLen, ".dhp");
		if (err = _tfopen_s(&File, tszFileName, _T("wN")))
		{
			goto error_rsa;
		}

		if (!PEM_write_DHparams(File, pDH))
		{
			goto error_rsa;
		}
		fclose(File);
		File = NULL;

		Putlog(LOG_DEBUG, _T("Finished generating Diffie-Huffman parameter!\r\n"));
		bReturn = TRUE;

@trippleflux
Copy link
Author

@slontis
The "PEM_write_PKCS8PrivateKey" doesn't work anymore on 3.0?, I am getting the following :

Generating 2048 bit Diffie-Huffman parameters:
OPENSSL_Uplink(7ACDF358,08): no OPENSSL_Applink

The code :

#include <Tchar.h>
#include <Winsock2.h>
#include <MsWsock.h>
#include <Windows.h>
#include <errno.h>
#include <strsafe.h>
#include <sys/stat.h>

#include "openssl/bio.h"
#include "openssl/err.h"
#include "openssl/ssl.h"

#include <time.h>
#include <conio.h>
#include <stdarg.h>
#include <locale.h>
#include <shlwapi.h>
#include <openssl/ssl.h>
#include <openssl/bio.h>
#include <openssl/err.h>
#include <openssl/ec.h>
#include <openssl/pem.h>
#include <openssl/conf.h>
#include <openssl/x509v3.h>
#include <openssl/engine.h>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/applink.c>
#define OPENSSL_API_COMPAT 0x10101000L
#pragma warning(disable : 4996)

int main() 
{
    int ret = 0;
    BIO* out = BIO_new_file("rsa.pem", "w");
    SSL_CTX* sctx = NULL;
    OSSL_PARAM params[2];
    unsigned int rsa_bits_strength = 4096;
    EVP_PKEY_CTX* pctx = NULL;
    EVP_PKEY* pkey = NULL;
    X509* pX509 = NULL;
    X509_NAME* xName = NULL;
    DH* pDH = NULL;
    BIGNUM* pBN_e = NULL;
	int        i=0;

    if (!(pBN_e = BN_new()) || !(BN_set_word(pBN_e, RSA_F4)))
    {
        goto err;
    }

    if (!(pX509 = X509_new()))
    {
        goto err;
    }

    pctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
    if (pctx == NULL)
        goto err;
    if (EVP_PKEY_keygen_init(pctx) <= 0)
        goto err;

    params[0] = OSSL_PARAM_construct_uint("bits", &rsa_bits_strength);
    params[1] = OSSL_PARAM_construct_end();
    if (!EVP_PKEY_CTX_set_params(pctx, params))
        goto err;

    if (EVP_PKEY_generate(pctx, &pkey) <= 0)
        goto err;

	if (!ASN1_INTEGER_set(X509_get_serialNumber(pX509), (LONG)time(NULL)))
	{
		goto err;
	}

	// make it valid as of yesterday to handle clients in a timezone where it's yesterday locally :)
	X509_gmtime_adj(X509_get_notBefore(pX509), -60 * 60 * 24);

	X509_gmtime_adj(X509_get_notAfter(pX509), (long)60 * 60 * 24 * 365 * 10); // ten years

	if (!X509_set_pubkey(pX509, pkey))
	{
		goto err;
	}

	// get the field where we can set Country, Name, etc...
	xName = X509_get_subject_name(pX509);

	// Add name of cert to subject line
	if (!X509_NAME_add_entry_by_txt(xName, "CN", MBSTRING_ASC, "Custom_One", -1, -1, 0))
	{
		goto err;
	}

	// We are the issuer since self signing it
	if (!X509_set_issuer_name(pX509, xName))
	{
		goto err;
	}

	if (!X509_sign(pX509, pkey, EVP_sha256()))
	{
		goto err;
	}

	if (!X509_verify(pX509, pkey))
	{
		printf("Failed to verify RSA key for certificate.\r\n");
		goto err;
	}


	if (!(pDH = DH_new()))
	{
		goto err;
	}


	printf("Generating 2048 bit Diffie-Huffman parameters:\r\n");

	if (!DH_generate_parameters_ex(pDH, 2048, 5, NULL))
	{
		printf("Failed to generate DH key for certificate.\r\n");
		goto err;
	}


	if (!DH_check(pDH, &i))
	{
		goto err;
	}

	if (i & DH_CHECK_P_NOT_PRIME)
	{
		printf("Failed to generate a prime.\r\n");
		goto err;
	}
	if (i & DH_CHECK_P_NOT_SAFE_PRIME)
	{
		printf("Failed to generate a safe prime.\r\n");
		goto err;
	}
	if (i & DH_UNABLE_TO_CHECK_GENERATOR)
	{
		printf("Failed to validate generator.\r\n");
		goto err;
	}
	if (i & DH_NOT_SUITABLE_GENERATOR)
	{
		printf("Invalid generator.\r\n");
		goto err;
	}

	FILE* fp;

	fp = fopen("rsa.pem", "w");
    if (!PEM_write_PKCS8PrivateKey(fp, pkey, NULL, NULL, 0, NULL, NULL))
        goto err;

    BIO_free(out);
	fclose(fp);
    out = NULL;

    sctx = SSL_CTX_new_ex(NULL, NULL, TLS_server_method());
    if (sctx == NULL)
        goto err;

	if (!SSL_CTX_use_PrivateKey_file(sctx, "rsa.pem", SSL_FILETYPE_PEM))
	{
		printf("Failed to use private key!\n");
		goto err;
	}
    ret = 1;
    printf("Success\n");
err:
    ret = 0;
}

Regarding the issue of :

AC110000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
AC110000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

Could it be regarding line ending that are read & write by OpenSSL 3.0?, in old 1.1.1 OpenSSL 1.1.1 windows is creating and reading Unix UTF-8 line ending (LF, detected by Notepad++) while on OpenSSL 3.0, it using Windows UTF-8 Line endings (CR LF).

I have changed my code into :

		if (out_cert == NULL)
			goto error_rsa;

		if (!PEM_write_bio_X509(out_cert, pX509))
		{
			goto error_rsa;
		}
		BIO_free(out_cert);
		out_cert = NULL;
		Putlog(LOG_DEBUG, _T("Finished generating Certificate!\r\n"));

		if (out_dh == NULL)
			goto error_rsa;

		if (!PEM_write_bio_DHparams(out_dh, pDH))
		{
			goto error_rsa;
		}
		BIO_free(out_dh);
		out_dh = NULL;

		Putlog(LOG_DEBUG, _T("Finished generating Diffie-Huffman parameter!\r\n"));

		if (out == NULL)
			goto error_rsa;

		if (!PEM_write_bio_PKCS8PrivateKey(out, PKey, NULL, NULL, 0, NULL, NULL))
		{
			goto error_rsa;
		}
		BIO_free(out);
		out = NULL;

		Putlog(LOG_DEBUG, _T("Finished generating Private Key!\r\n"));
		bReturn = TRUE;

The "No supported data to decode" seems doesn't happened on first example codes that is copied above, but strangely it only happened in my codebase, also I have tested changed my Windows 10 version from 21H1 into 21H2 the problem still persist with OpenSSL 3.0 only, so I doubt it's a windows problems.

@mattcaswell
Copy link
Member

The "PEM_write_PKCS8PrivateKey" doesn't work anymore on 3.0?, I am getting the following :

Generating 2048 bit Diffie-Huffman parameters:
OPENSSL_Uplink(7ACDF358,08): no OPENSSL_Applink

That's not a 3.0 specific issue - applink has been around for a long time. You need to compile and link the file applink.c into your application:

https://www.openssl.org/docs/man3.0/man3/OPENSSL_Applink.html

@trippleflux
Copy link
Author

@mattcaswell
I have included the applink.c and yes indeed it's working fine now, but I am still cannot figuring it out why :

AC110000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
AC110000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

Is happened in my codebase.

@mattcaswell
Copy link
Member

Is there actually any data in the rsa.pem file?

This looks suspicious in your code:

    BIO* out = BIO_new_file("rsa.pem", "w");
...
	fp = fopen("rsa.pem", "w");
    if (!PEM_write_PKCS8PrivateKey(fp, pkey, NULL, NULL, 0, NULL, NULL))
        goto err;

    BIO_free(out);
	fclose(fp);

You are opening a BIO for the rsa.pem file (which under the covers calls "fopen" on the file). Then entirely separately you are "fopen"ing the same file and writing data to it. Then you "BIO_free" the BIO which (which under the covers calls "fclose" on the file) and then finally you call fclose again. That looks very strange. Since you never seem to use the "out" BIO it can be removed.

All the stuff in your reproducer about creating an X509 and then signing it, as well as all the stuff about generating DH parameters all seem irrelevant to the reproducer since you never do anything with the results. I would remove all of that from your reproducer and shrink it down to the simplest possible code that demonstrates the problem.

@trippleflux
Copy link
Author

Yes, the rsa.key should not have any content/created before, because I am using code to pre-generated the .key at startup. I have posted before that I have changed the code into the following :

    BIO* out = BIO_new_file("rsa.key", "w+");
    BIO* out_cert = BIO_new_file("rsa.pem", "w+");
    BIO* out_dh = BIO_new_file("rsa.dhp", "w+");
		if (out_cert == NULL)
			goto error_rsa;

		if (!PEM_write_bio_X509(out_cert, pX509))
		{
			goto error_rsa;
		}
		BIO_free(out_cert);
		out_cert = NULL;
		Putlog(LOG_DEBUG, _T("Finished generating Certificate!\r\n"));

		if (out_dh == NULL)
			goto error_rsa;

		if (!PEM_write_bio_DHparams(out_dh, pDH))
		{
			goto error_rsa;
		}
		BIO_free(out_dh);
		out_dh = NULL;

		Putlog(LOG_DEBUG, _T("Finished generating Diffie-Huffman parameter!\r\n"));

		if (out == NULL)
			goto error_rsa;

		if (!PEM_write_bio_PKCS8PrivateKey(out, PKey, NULL, NULL, 0, NULL, NULL))
		{
			goto error_rsa;
		}
		BIO_free(out);
		out = NULL;

		Putlog(LOG_DEBUG, _T("Finished generating Private Key!\r\n"));
		bReturn = TRUE;

    sctx = SSL_CTX_new_ex(NULL, NULL, TLS_server_method());
    if (sctx == NULL)
        goto err;

	if (!SSL_CTX_use_PrivateKey_file(sctx, "rsa.key", SSL_FILETYPE_PEM))
	{
		printf("Failed to use private key!\n");
		goto err;
	}

I have tried to simplified the code but haven't been able to reproduce it isolated/simple code/case.

Could it be regarding line ending that are read & write by OpenSSL 3.0?, in old 1.1.1 OpenSSL 1.1.1 windows is creating and reading Unix UTF-8 line ending (LF, detected by Notepad++) while on OpenSSL 3.0, it using Windows UTF-8 Line endings (CR LF).

@mattcaswell
Copy link
Member

What happens if you remove all of you code except for the bit calling SSL_CTX_use_PrivateKey_file and attempt to use an rsa.key file that exists from a previous run. Do you get the same problem?

I would expect the line endings to be handled properly. But it is a plausible theory that something is going wrong there.

@trippleflux
Copy link
Author

Restarting the app is managed to load/check the private key with SSL_CTX_use_PrivateKey_file , my problem is only happening whenever after generating the key,pem,dhp and then straightly checking it out with SSL_CTX_use_PrivateKey_file , so without app restart/shutdown.

Cannot reproduce it with this code :

#include <Tchar.h>
#include <Winsock2.h>
#include <MsWsock.h>
#include <Windows.h>
#include <errno.h>
#include <strsafe.h>
#include <sys/stat.h>

#include "openssl/bio.h"
#include "openssl/err.h"
#include "openssl/ssl.h"

#include <time.h>
#include <conio.h>
#include <stdarg.h>
#include <locale.h>
#include <shlwapi.h>
#include <openssl/ssl.h>
#include <openssl/bio.h>
#include <openssl/err.h>
#include <openssl/ec.h>
#include <openssl/pem.h>
#include <openssl/conf.h>
#include <openssl/x509v3.h>
#include <openssl/engine.h>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/applink.c>
#define OPENSSL_API_COMPAT 0x10101000L
#pragma warning(disable : 4996)

int test()
{
	int ret = 0;
	SSL_CTX* sctx = NULL;
	sctx = SSL_CTX_new_ex(NULL, NULL, TLS_server_method());
	if (sctx == NULL)
		goto err;


	if (!SSL_CTX_use_certificate_chain_file(sctx, "rsa.pem"))
	{
		SSL_CTX_free(sctx);
		return TRUE;
	}

	if (!SSL_CTX_use_PrivateKey_file(sctx, "rsa.key", SSL_FILETYPE_PEM))
	{
		printf("Failed to use private key!\n");
		goto err;
	}

	if (SSL_CTX_check_private_key(sctx) != 1)
	{
		SSL_CTX_free(sctx);
		return TRUE;
	}

	printf("Success Private Key\n");
	ret = 1;
err :
	ret = 0;

}

int main() 
{
    int ret = 0;
	BIO* out = BIO_new_file("rsa.key", "w+");
	BIO* out_cert = BIO_new_file("rsa.pem", "w+");
	BIO* out_dh = BIO_new_file("rsa.dhp", "w+");
    OSSL_PARAM params[2];
    unsigned int rsa_bits_strength = 4096;
    EVP_PKEY_CTX* pctx = NULL;
    EVP_PKEY* pkey = NULL;
    X509* pX509 = NULL;
    X509_NAME* xName = NULL;
    DH* pDH = NULL;
    BIGNUM* pBN_e = NULL;
	int        i=0;

    if (!(pBN_e = BN_new()) || !(BN_set_word(pBN_e, RSA_F4)))
    {
        goto err;
    }

    if (!(pX509 = X509_new()))
    {
        goto err;
    }

    pctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
    if (pctx == NULL)
        goto err;
    if (EVP_PKEY_keygen_init(pctx) <= 0)
        goto err;

    params[0] = OSSL_PARAM_construct_uint("bits", &rsa_bits_strength);
    params[1] = OSSL_PARAM_construct_end();
    if (!EVP_PKEY_CTX_set_params(pctx, params))
        goto err;

    if (EVP_PKEY_generate(pctx, &pkey) <= 0)
        goto err;

	if (!ASN1_INTEGER_set(X509_get_serialNumber(pX509), (LONG)time(NULL)))
	{
		goto err;
	}

	// make it valid as of yesterday to handle clients in a timezone where it's yesterday locally :)
	X509_gmtime_adj(X509_get_notBefore(pX509), -60 * 60 * 24);

	X509_gmtime_adj(X509_get_notAfter(pX509), (long)60 * 60 * 24 * 365 * 10); // ten years

	if (!X509_set_pubkey(pX509, pkey))
	{
		goto err;
	}

	// get the field where we can set Country, Name, etc...
	xName = X509_get_subject_name(pX509);

	// Add name of cert to subject line
	if (!X509_NAME_add_entry_by_txt(xName, "CN", MBSTRING_ASC, "Custom_One", -1, -1, 0))
	{
		goto err;
	}

	// We are the issuer since self signing it
	if (!X509_set_issuer_name(pX509, xName))
	{
		goto err;
	}

	if (!X509_sign(pX509, pkey, EVP_sha256()))
	{
		goto err;
	}

	if (!X509_verify(pX509, pkey))
	{
		printf("Failed to verify RSA key for certificate.\r\n");
		goto err;
	}


	if (!(pDH = DH_new()))
	{
		goto err;
	}


	printf("Generating 2048 bit Diffie-Huffman parameters:\r\n");

	if (!DH_generate_parameters_ex(pDH, 2048, 5, NULL))
	{
		printf("Failed to generate DH key for certificate.\r\n");
		goto err;
	}


	if (!DH_check(pDH, &i))
	{
		goto err;
	}

	if (i & DH_CHECK_P_NOT_PRIME)
	{
		printf("Failed to generate a prime.\r\n");
		goto err;
	}
	if (i & DH_CHECK_P_NOT_SAFE_PRIME)
	{
		printf("Failed to generate a safe prime.\r\n");
		goto err;
	}
	if (i & DH_UNABLE_TO_CHECK_GENERATOR)
	{
		printf("Failed to validate generator.\r\n");
		goto err;
	}
	if (i & DH_NOT_SUITABLE_GENERATOR)
	{
		printf("Invalid generator.\r\n");
		goto err;
	}

	FILE* fp;

	if (out_cert == NULL)
		goto err;

	if (!PEM_write_bio_X509(out_cert, pX509))
	{
		goto err;
	}
	BIO_free(out_cert);
	out_cert = NULL;
	printf("Finished generating Certificate!\r\n");

	if (out_dh == NULL)
		goto err;

	if (!PEM_write_bio_DHparams(out_dh, pDH))
	{
		goto err;
	}
	BIO_free(out_dh);
	out_dh = NULL;

	printf("Finished generating Diffie-Huffman parameter!\r\n");

	if (out == NULL)
		goto err;

	if (!PEM_write_bio_PKCS8PrivateKey(out, pkey, NULL, NULL, 0, NULL, NULL))
	{
		goto err;
	}
	BIO_free(out);
	out = NULL;

	printf("Finished generating Private Key!\r\n");
    printf("Success\n");
	test();
err:
    ret = 0;
}

But from my codebase for 3.0, getting :

AC110000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
AC110000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

This issue doesn't happened with OpenSSL 1.1.1 branch :-(

@trippleflux
Copy link
Author

trippleflux commented Oct 12, 2021

I am trying to 'bypass' reading private key from the file through OpenSSL by using fopen into a buffer and convert it back into EVP_KEY, the code is below :

	if (!SSL_CTX_use_certificate_chain_file(pCtx, "rsa.pem"))
	{
		SSL_CTX_free(pCtx);
		return TRUE;
	}
	char* source = NULL;
	FILE* fp = fopen("rsa.key", "rt");
	if (fp != NULL) {
		/* Go to the end of the file. */
		if (fseek(fp, 0L, SEEK_END) == 0) {
			/* Get the size of the file. */
			long bufsize = ftell(fp);
			if (bufsize == -1) { /* Error */ }

			/* Allocate our buffer to that size. */
			source = malloc(sizeof(char) * (bufsize + 1));

			/* Go back to the start of the file. */
			if (fseek(fp, 0L, SEEK_SET) != 0) { /* Error */ }

			/* Read the entire file into memory. */
			size_t newLen = fread(source, sizeof(char), bufsize, fp);
			if (ferror(fp) != 0) {
				fputs("Error reading file", stderr);
			}
			else {
				source[newLen++] = '\0'; /* Just to be safe. */
			}
		}
		fclose(fp);
	}

	int pkeyLen2;
	unsigned char *uctempBuf2;
	pkeyLen2 = i2d_PrivateKey(pkey, NULL);
	uctempBuf2 = source;
	i2d_PrivateKey(pkey, &uctempBuf2);

	if (!SSL_CTX_use_PrivateKey(pCtx, pkey))
	{
		FILE* fp;
		fp = fopen("errors.log", "w");
		ERR_print_errors_fp(fp);
		fclose(fp);
		SSL_CTX_free(pCtx);
		return TRUE;
	}

	Putlog(LOG_DEBUG, _T("SSL_CTX_use_PrivateKey is a success.\r\n"));

	if (!SSL_CTX_check_private_key(pCtx))
	{
		SSL_CTX_free(pCtx);
		return TRUE;
	}

	Putlog(LOG_DEBUG, _T("SSL_CTX_check_private_key is a success.\r\n"));

But it failed the SSL_CTX_check_private_key!

[EDITED]
The code now are :

	char source[MAXBUFLEN + 1];
	FILE* fp = fopen(szFileName, "r");
	if (fp != NULL) {
		size_t newLen = fread(source, sizeof(char), MAXBUFLEN, fp);
		if (ferror(fp) != 0) {
			fputs("Error reading file", stderr);
		}
		else {
			source[newLen++] = '\0'; /* Just to be safe. */
		}

		fclose(fp);
	}

	SSL_CTX* CTX;
	X509* cert = NULL;
	RSA* rsa = NULL;
	BIO* cbio, * kbio;
	const char* cert_buffer = source;
	const char* key_buffer = NULL;

	cbio = BIO_new_mem_buf((void*)cert_buffer, -1);
	cert = PEM_read_bio_X509(cbio, NULL, 0, NULL);

	if (!SSL_CTX_use_certificate(pCtx, cert))
	{
		SSL_CTX_free(pCtx);
		return TRUE;
	}

	strcpy_s(&szFileName[sLen], sizeof(szFileName) / sizeof(*szFileName) - sLen, ".key");

	// SSL_CTX_set_default_passwd_cb(ctx, cb);
	char source2[MAXBUFLEN + 1];
	FILE* fp2 = fopen(szFileName, "r");
	if (fp2 != NULL) {
		size_t newLen = fread(source2, sizeof(char), MAXBUFLEN, fp2);
		if (ferror(fp2) != 0) {
			fputs("Error reading file", stderr);
		}
		else {
			source2[newLen++] = '\0'; /* Just to be safe. */
		}

		fclose(fp2);
	}

	key_buffer = source2;
	kbio = BIO_new_mem_buf((void*)key_buffer, -1);
	rsa = PEM_read_bio_RSAPrivateKey(kbio, NULL, 0, NULL);

	if (!SSL_CTX_use_PrivateKey(pCtx, rsa))
	{
		FILE* fp;
		fp = fopen("errors.log", "w");
		ERR_print_errors_fp(fp);
		fclose(fp);
		SSL_CTX_free(pCtx);
		return TRUE;
	}

	Putlog(LOG_DEBUG, _T("SSL_CTX_use_PrivateKey is a success.\r\n"));

	if (!SSL_CTX_check_private_key(pCtx))
	{
		SSL_CTX_free(pCtx);
		return TRUE;
	}

	Putlog(LOG_DEBUG, _T("SSL_CTX_check_private_key is a success.\r\n"));

But weirdly I am still getting :

80600000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
80600000:error:0A0C0102:SSL routines:SSL_CTX_use_PrivateKey:passed a null parameter:ssl\ssl_rsa.c:348:

@trippleflux
Copy link
Author

trippleflux commented Oct 12, 2021

Managed to reproduce it with minimal test code :

#include <Tchar.h>
#include <Winsock2.h>
#include <MsWsock.h>
#include <Windows.h>
#include <errno.h>
#include <strsafe.h>
#include <sys/stat.h>

#include "openssl/bio.h"
#include "openssl/err.h"
#include "openssl/ssl.h"

#include <time.h>
#include <conio.h>
#include <stdarg.h>
#include <locale.h>
#include <shlwapi.h>
#include <openssl/ssl.h>
#include <openssl/bio.h>
#include <openssl/err.h>
#include <openssl/ec.h>
#include <openssl/pem.h>
#include <openssl/conf.h>
#include <openssl/x509v3.h>
#include <openssl/engine.h>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/applink.c>
#define OPENSSL_API_COMPAT 0x10101000L
#pragma warning(disable : 4996)

int test()
{
	int ret = 0;
	SSL_CTX* sctx = NULL;
	size_t   sLen;
	EVP_PKEY* pkey = NULL;
	char     szFileName[MAX_PATH + 1];
	LPSTR szCertificateName = "rsa";
	sctx = SSL_CTX_new_ex(NULL, NULL, TLS_method());
	if (sctx == NULL)
		goto err;
	SSL_CTX_set_min_proto_version(sctx, TLS1_2_VERSION);

	sLen = strlen(szCertificateName);

	_snprintf_s(szFileName, sizeof(szFileName) / sizeof(*szFileName), _TRUNCATE, "%s.pem", szCertificateName);
	if (!SSL_CTX_use_certificate_chain_file(sctx, szFileName))
	{
		SSL_CTX_free(sctx);
		return TRUE;
	}


	char* source = NULL;
	const char* key_buffer = NULL;
	FILE* fp = fopen("rsa.key", "rt");
	if (fp != NULL) {
		/* Go to the end of the file. */
		if (fseek(fp, 0L, SEEK_END) == 0) {
			/* Get the size of the file. */
			long bufsize = ftell(fp);
			if (bufsize == -1) { /* Error */ }

			/* Allocate our buffer to that size. */
			source = malloc(sizeof(char) * (bufsize + 1));

			/* Go back to the start of the file. */
			if (fseek(fp, 0L, SEEK_SET) != 0) { /* Error */ }

			/* Read the entire file into memory. */
			size_t newLen = fread(source, sizeof(char), bufsize, fp);
			if (ferror(fp) != 0) {
				fputs("Error reading file", stderr);
			}
			else {
				source[newLen++] = '\0'; /* Just to be safe. */
			}
		}
		fclose(fp);
	}

	OSSL_DECODER_CTX* dctx;
	BIO* kbio;
	const char* format = "PEM";   /* NULL for any format */
	const char* structure = NULL; /* any structure */
	const char* keytype = NULL;  /* NULL for any key */
	const unsigned char* pass = NULL;
	key_buffer = source;
	const unsigned char* data = source;
	size_t datalen = sizeof(source);

	dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, format, structure,
		keytype,
		OSSL_KEYMGMT_SELECT_KEYPAIR
		| OSSL_KEYMGMT_SELECT_DOMAIN_PARAMETERS,
		NULL, NULL);
	if (dctx == NULL) {
		/* error: no suitable potential decoders found */
	}
	if (pass != NULL)
		OSSL_DECODER_CTX_set_passphrase(dctx, pass, strlen(pass));
	if (OSSL_DECODER_from_data(dctx, &data, &datalen)) {
		/* pkey is created with the decoded data from the buffer */
	}
	else {
		/* decoding failure */
	}
	OSSL_DECODER_CTX_free(dctx);

	if (!SSL_CTX_use_PrivateKey(sctx, pkey))
	{
		FILE* fp;
		fp = fopen("errors.log", "w");
		ERR_print_errors_fp(fp);
		fclose(fp);
		SSL_CTX_free(sctx);
		return TRUE;
	}

	if (SSL_CTX_check_private_key(sctx) != 1)
	{
		SSL_CTX_free(sctx);
		return TRUE;
	}

	printf("Success Private Key\n");
	ret = 1;
err :
	ret = 0;

}

int main() 
{
    int ret = 0;
	BIO* out = BIO_new_file("rsa.key", "w+");
	BIO* out_cert = BIO_new_file("rsa.pem", "w+");
	BIO* out_dh = BIO_new_file("rsa.dhp", "w+");
    OSSL_PARAM params[2];
    unsigned int rsa_bits_strength = 4096;
    EVP_PKEY_CTX* pctx = NULL;
    EVP_PKEY* pkey = NULL;
    X509* pX509 = NULL;
    X509_NAME* xName = NULL;
    DH* pDH = NULL;
    BIGNUM* pBN_e = NULL;
	int        i=0;

    if (!(pBN_e = BN_new()) || !(BN_set_word(pBN_e, RSA_F4)))
    {
        goto err;
    }

    if (!(pX509 = X509_new()))
    {
        goto err;
    }

    pctx = EVP_PKEY_CTX_new_from_name(NULL, "RSA", NULL);
    if (pctx == NULL)
        goto err;
    if (EVP_PKEY_keygen_init(pctx) <= 0)
        goto err;

    params[0] = OSSL_PARAM_construct_uint("bits", &rsa_bits_strength);
    params[1] = OSSL_PARAM_construct_end();
    if (!EVP_PKEY_CTX_set_params(pctx, params))
        goto err;

    if (EVP_PKEY_generate(pctx, &pkey) <= 0)
        goto err;

	if (!ASN1_INTEGER_set(X509_get_serialNumber(pX509), (LONG)time(NULL)))
	{
		goto err;
	}

	// make it valid as of yesterday to handle clients in a timezone where it's yesterday locally :)
	X509_gmtime_adj(X509_get_notBefore(pX509), -60 * 60 * 24);

	X509_gmtime_adj(X509_get_notAfter(pX509), (long)60 * 60 * 24 * 365 * 10); // ten years

	if (!X509_set_pubkey(pX509, pkey))
	{
		goto err;
	}

	// get the field where we can set Country, Name, etc...
	xName = X509_get_subject_name(pX509);

	// Add name of cert to subject line
	if (!X509_NAME_add_entry_by_txt(xName, "CN", MBSTRING_ASC, "Custom_One", -1, -1, 0))
	{
		goto err;
	}

	// We are the issuer since self signing it
	if (!X509_set_issuer_name(pX509, xName))
	{
		goto err;
	}

	if (!X509_sign(pX509, pkey, EVP_sha256()))
	{
		goto err;
	}

	if (!X509_verify(pX509, pkey))
	{
		printf("Failed to verify RSA key for certificate.\r\n");
		goto err;
	}


	if (!(pDH = DH_new()))
	{
		goto err;
	}


	printf("Generating 2048 bit Diffie-Huffman parameters:\r\n");

	if (!DH_generate_parameters_ex(pDH, 2048, 5, NULL))
	{
		printf("Failed to generate DH key for certificate.\r\n");
		goto err;
	}


	if (!DH_check(pDH, &i))
	{
		goto err;
	}

	if (i & DH_CHECK_P_NOT_PRIME)
	{
		printf("Failed to generate a prime.\r\n");
		goto err;
	}
	if (i & DH_CHECK_P_NOT_SAFE_PRIME)
	{
		printf("Failed to generate a safe prime.\r\n");
		goto err;
	}
	if (i & DH_UNABLE_TO_CHECK_GENERATOR)
	{
		printf("Failed to validate generator.\r\n");
		goto err;
	}
	if (i & DH_NOT_SUITABLE_GENERATOR)
	{
		printf("Invalid generator.\r\n");
		goto err;
	}

	FILE* fp;

	if (out_cert == NULL)
		goto err;

	if (!PEM_write_bio_X509(out_cert, pX509))
	{
		goto err;
	}
	BIO_free(out_cert);
	out_cert = NULL;
	printf("Finished generating Certificate!\r\n");

	if (out_dh == NULL)
		goto err;

	if (!PEM_write_bio_DHparams(out_dh, pDH))
	{
		goto err;
	}
	BIO_free(out_dh);
	out_dh = NULL;

	printf("Finished generating Diffie-Huffman parameter!\r\n");

	if (out == NULL)
		goto err;

	if (!PEM_write_bio_PKCS8PrivateKey(out, pkey, NULL, NULL, 0, NULL, NULL))
	{
		goto err;
	}
	BIO_free(out);
	out = NULL;

	printf("Finished generating Private Key!\r\n");
    printf("Success\n");
	test();
err:
    ret = 0;
}

It will failed at "SSL_CTX_use_PrivateKey"

6C430000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
6C430000:error:0A0C0102:SSL routines:SSL_CTX_use_PrivateKey:passed a null parameter:ssl\ssl_rsa.c:348:

Perhaps relevant with SSL_CTX_use_PrivateKey_file error above :

AC110000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM
AC110000:error:0A080009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib:ssl\ssl_rsa.c:384:

@trippleflux trippleflux changed the title No supported data to decode. Input type: PEM [Bug]No supported data to decode. Input type: PEM Oct 13, 2021
@kaduk
Copy link
Contributor

kaduk commented Oct 14, 2021

A couple things to try: print the value of pkey after the (successful) OSSL_DECODER_from_data() call, and sleep or otherwise stall execution after writing out the cert/key to file but before calling test(), and inspect that the files on disk have proper contents at that time.
It looks like the BIO_free(out_cert) should be performing an implicit fflush() as part of fclose(), but the symptoms are otherwise pretty consistent with an incompletely written file.

@trippleflux
Copy link
Author

@kaduk
Yes, It wasn't happening with OpenSSL 1.1.1, from your pointer I have added "SleepEx(2000, TRUE);" into the code before SSL_CTX_use_PrivateKey_file but doesn't make any differences.

@kaduk
Copy link
Contributor

kaduk commented Oct 14, 2021

My suggestion was not to just add the sleep. My suggestion was to go and look at the file on disk during the sleep.

@trippleflux
Copy link
Author

@kaduk
Added a file reader for the rsa.key after the 2,000 Milliseconds sleep, able to read it from beginning to end, but somehow the file is not properly closed/flushed?, like you've said, on BIO_free it should be immediately flushed the file.

@trippleflux
Copy link
Author

trippleflux commented Oct 15, 2021

My suggestion was not to just add the sleep. My suggestion was to go and look at the file on disk during the sleep.

Already tried, the file seems perfectly created and ended, but it's line endings is Windows UTF-8, not Unix UTF-8 which is default in OpenSSL 1.1.1

@trippleflux
Copy link
Author

I have posted the reproducible code in #16696 (comment) , should I upload the complete VS 2019 project files?.

So the issue is somewhere before SSL_CTX_use_PrivateKey & SSL_CTX_use_PrivateKey_file because using privatekey from either buffer "SSL_CTX_use_PrivateKey" or file read "SSL_CTX_use_PrivateKey_file" getting the same :

6C430000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto\encode_decode\decoder_lib.c:101:No supported data to decode. Input type: PEM

Although I cannot get reproducible test code with SSL_CTX_use_PrivateKey_file, but isn't it this is worth to look for, at-least it can be reproducible with buffer way "SSL_CTX_use_PrivateKey"?

@buczaks
Copy link

buczaks commented Jan 6, 2024

I had this issue with nginx on a freebsd server .. looks like it has nothing to do with the openssl3 upgrade (even though I did that at that server too) but the fix, at least for nginx and probably elsewhere, is to simply rename the cert .pem to *.crt and the key needs a *.key extension then nginx is happy. Nginx likes the certificate to end with .crt and the key as .key. It doesn't understand what a .pem file is. nginx version: nginx/1.24.0

@nhorman
Copy link
Contributor

nhorman commented Jun 10, 2024

Marking as inactive, to be closed at the end of 3.4 dev, barring further input

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch: 3.0 Merge to openssl-3.0 branch inactive triaged: question The issue contains a question
Projects
None yet
Development

No branches or pull requests

7 participants