Error loading key ".ssh/id_rsa": error in libcrypto

[tpgxyz]

Hi,
im using my ssh RSA key for many years. After update to openssl3 i noticed an issue:

[tpg@tpg-virtualbox .ssh]$ ssh-add
Error loading key "/home/tpg/.ssh/id_rsa": error in libcrypto

I've validated my keys:
[tpg@tpg-virtualbox .ssh]$ ssh-keygen -l -f id_rsa.pub
4096 SHA256:hereisthehash xxx@gmail.com (RSA)

Newly generated key with ssh-keygen works.

I tried to generate RSA key with openssl:

[tpg@tpg-virtualbox .ssh]$ openssl genrsa -des3 -out private.pem 2048
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
[tpg@tpg-virtualbox .ssh]$ openssl rsa -in private.pem -outform PEM -pubout -out public.pem
Enter pass phrase for private.pem:
writing RSA key
[tpg@tpg-virtualbox .ssh]$ openssl pkey -check -in private.pem -noout
Enter pass phrase for private.pem:
Key is valid
[tpg@tpg-virtualbox .ssh]$ openssl pkey -pubcheck -in public.pem -noout
Could not read key from public.pem
Unable to load key

My system is OpenMandriva Cooker

[tpg@tpg-virtualbox .ssh]$ rpm -qa | grep openssl
openssl-3.0.0-0.alpha8.1.x86_64

[tpg@tpg-virtualbox .ssh]$ rpm -qa | grep openssh
openssh-8.4p1-2.x86_64

[mattcaswell]

Error loading key "/home/tpg/.ssh/id_rsa": error in libcrypto

"error in libcrypto" isn't a very helpful error message!! I wonder if you get anything more useful by trying to inspect the key directly with the openssl command line, e.g. the following command will print the details of the private key in a readable text form:

openssl rsa -in /home/tpg/.ssh/id_rsa -noout -text

Do you get any errors from this command. If so what are they?

[tpg@tpg-virtualbox .ssh]$ openssl verify private.pem  
Enter pass phrase for private.pem:
Could not read certificate file from private.pem
Unable to load certificate file
[tpg@tpg-virtualbox .ssh]$ openssl verify public.pem 
Could not read certificate file from public.pem
Unable to load certificate file

This is not going to work. openssl verify is used to verify X509 certificates. You're supplying it with an RSA private/public key which is not the same thing. Hence it complains.

Perhaps what you meant was:

openssl pkey -check -in private.pem -noout
openssl pkey -pubcheck -in public.pem -noout

[tpgxyz]

@mattcaswell i've filled this issue as i tried almost everything to debug this. All i got is "error in libcrypto"

[tpg@tpg-virtualbox .ssh]$ openssl rsa -in id_rsa -noout -text
Could not read private key from id_rsa
Unable to load private key

Sorry for pasting wrong output from openssl verify.
Here is good one:

[tpg@tpg-virtualbox .ssh]$ openssl pkey -check -in private.pem -noout
Enter pass phrase for private.pem:
Key is valid
[tpg@tpg-virtualbox .ssh]$ openssl pkey -pubcheck -in public.pem -noout
Could not read key from public.pem
Unable to load key

[mattcaswell]

Could not read private key from id_rsa
Unable to load private key

That isn't much more helpful :-)

Can you take a look at what is inside the id_rsa file. I'm interested in the PEM headers. Mine look like this:

-----BEGIN RSA PRIVATE KEY-----
...snip private key data...
-----END RSA PRIVATE KEY-----

What do yours look like?

[tpgxyz]

@mattcaswell if you help me how to debug this i'll provide needed information:

cat private.pem

-----BEGIN ENCRYPTED PRIVATE KEY-----
somehash
-----END ENCRYPTED PRIVATE KEY-----

cat id_rsa

-----BEGIN OPENSSH PRIVATE KEY-----
myhash
-----END OPENSSH PRIVATE KEY-----

[mattcaswell]

Ah.

-----BEGIN OPENSSH PRIVATE KEY-----

This is OpenSSH's own private key format. OpenSSL does not support it directly, so we won't be able to use OpenSSL command line tools to examine it. That is a shame. That means the error is somewhere between ssh and the OpenSSL API. You might need to raise an issue with OpenSSH to help track this down. Almost certainly the problem is in OpenSSL but without know what OpenSSH is doing to result in that error, its going to be very difficult for us to figure out what the fix is.

[tpgxyz]

@mattcaswell Ok i understand this, but why RSA private key generated with openssl3 cannot be loaded:

[tpg@tpg-virtualbox .ssh]$ openssl genrsa -des3 -out private.pem 2048
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
[tpg@tpg-virtualbox .ssh]$ openssl rsa -in private.pem -outform PEM -pubout -out public.pem
Enter pass phrase for private.pem:
writing RSA key
[tpg@tpg-virtualbox .ssh]$ openssl pkey -check -in private.pem -noout
Enter pass phrase for private.pem:
Key is valid
[tpg@tpg-virtualbox .ssh]$ openssl pkey -pubcheck -in public.pem -noout
Could not read key from public.pem
Unable to load key

[mattcaswell]

$ openssl pkey -pubcheck -in public.pem -noout
Could not read key from public.pem
Unable to load key

Oops. My mistake. The command should be:

$ openssl pkey -pubcheck -pubin -in public.pem -noout

[tpgxyz]

@mattcaswell Looks like this is openssh issue as openssl can read RSA keys by itself. Thenks for your help. I'll contact openssh people.

[tpg@tpg-virtualbox belr]$ openssl pkey -pubcheck -pubin -in public.pem -noout
Key is valid

[mattcaswell]

Looks like this is openssh issue as openssl can read RSA keys by itself.

You might want to check if it happens with OpenSSL 1.1.1. OpenSSL 3.0 is still in dev and is buggy. Its entirely possible that this still ends up being an OpenSSL bug.

[tpgxyz]

@mattcaswell Hi again,
i added some debug output to openssh OpenMandrivaAssociation/openssh@cdac490

[tpg@tpg-virtualbox belcard]$ ssh-add
Error loading key "/home/tpg/.ssh/id_rsa": error:00000000:lib(0)::reason(0)

[mattcaswell]

Unfortunately that doesn't really tell us anything:

error:00000000:lib(0)::reason(0)

This just means there is no error on the OpenSSL error stack. That doesn't mean that an error did not occur - just that there isn't a hint as to what it is on the stack.

[tpgxyz]

I filled bug for OpenSSH https://bugzilla.mindrot.org/show_bug.cgi?id=3233

[Fubuchi]

There is something wrong with openssh in Openmandriva, this issue happened for me as well. I am using their rolling repo.

[tpgxyz]

Looks like certificates are rejected by openvpn if it is linked to openssl3

Please see this bug:
https://issues.openmandriva.org/show_bug.cgi?id=2670

[tpgxyz]

Output of a strace ssh-add -vvv :

newfstatat(AT_FDCWD, "/foo/.ssh/id_rsa", {st_mode=S_IFREG|0600, st_size=3326, ...}, 0) = 0
openat(AT_FDCWD, "/foo/.ssh/id_rsa", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...}, AT_EMPTY_PATH) = 0
getuid()                                = 0
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...}, AT_EMPTY_PATH) = 0
read(4, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 3326
read(4, "", 770)                        = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
futex(0x7f29f67cf20c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde50, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde58, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "306: Error loading key \"/foo/.s"..., 63306: Error loading key "/foo/.ssh/id_rsa": error in libcrypto
) = 63

[levitte]

We do have certain problems with reading PEM files in OpenSSL 3 for the moment. There are several angles that we look at.
Pleas stay tuned.

[tpgxyz]

@levitte Are you saying that from Nov 2020 nobody is taking care of this ? And yes i remain tuned starting from the day i filled this issue.

[t8m]

It would be great for debugging if we had some disposable RSA private key that exposes this issue.

[levitte]

@levitte Are you saying that from Nov 2020 nobody is taking care of this ? And yes i remain tuned starting from the day i filled this issue.

We currently have multiple items that need tending to. All that takes time, and sometimes multiple issue being raised before the underlying problem becomes clear. Sorry that it's trying your patience.

[tpgxyz]

Well i can try to dig for some old ssh keys. My current one looks like this:

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CB, <- here is the IV
-----END RSA PRIVATE KEY-----

@levitte Yes i know how software development works, and i my point is there maybe some people who may got hit byt this issue and could not handle re-conversion process of old certs.

[petrovr]

Interesting ... As is reported vesion is 8.4p1, and so OpenSSH defects like https://bugzilla.mindrot.org/show_bug.cgi?id=2901 and https://bugzilla.mindrot.org/show_bug.cgi?id=2913 should not apply.

OpenBSD product uses quite unexpected management to load keys. It is based on heurestic management of reasons returned by cryptograpic library(1) and specific "empty" password management(2).

Since 7.9/8.0 (1) is used only with "non-empty" passwords. Error in openssh here - out of scope. Let say that (1) is applicable after password prompt.

About (2) - it cannot produce "error in lib crypto" unless source is patched by vendor.
In brief: code try to read PEM with empty password, pem callback return -1, PEM_read fail and code return "wrong password".

I cannot see relation to #14100 - it is applicable for (1), i.e. after password prompt.

Issue could be due to some patches to original code.

Note: I assume that there is no ask-pass program in use.

[tpgxyz]

We have yet another report that FreeRDP crashes with OpenSSLv3
http://issues.openmandriva.org/show_bug.cgi?id=2672

[t8m]

We have yet another report that FreeRDP crashes with OpenSSLv3
http://issues.openmandriva.org/show_bug.cgi?id=2672

Not sure how this is related to the original issue. It should be reported as separate issue ideally with some reproducer cut out of the FreeRDP code. Otherwise it would be really hard to find the cause.

[levitte]

Note that, at least for openssh-portable, ssh-add only understands -----BEGIN OPENSSH PRIVATE KEY-----, not -----BEGIN RSA PRIVATE KEY-----. These two macros are dead giveaways:

https://github.com/openssh/openssh-portable/blob/0a4b23b11b9a4e6eec332dd5c6ab2ac6f62aa164/sshkey.c#L70-L71

(I ran ssh-add through the debugger to discover this)

[petrovr]

Hi, Richard Levitte wrote:

Thanks for the nudge, @petrovr. You're right, I stopped too soon. I've gone further now, and it turns out that there's failure to translate the libcrypto error here: https://github.com/openssh/openssh-portable/blob/0a4b23b11b9a4e6eec332dd5c6ab2ac6f62aa164/sshkey.c#L4610 It's understandable why, as it seems that no error is reported at all... it seems to me like the decoding process is pruning errors a bit too hard...

Off topic: for missing error reasons in master branch I open a separate issue. OpenSSH scenario is following: (1) try load of key with empty password (2) password prompt if (1) fail with "wrong password" (3) try load of key with entered password (4) ask again for password if (3) fail due to password error (2 or 3 prompts) Remark: in (1) error reasons translation is not used. I post in an OpenSSL issues information for this with number of related OpenSSH defects. The one on OpenSSH defect has script that allows to find RSA keys with "empty" password. Script is usable with more recent OpenSSL 1.0.2 releases, after patch X that enhance key processing. It cannot be run with 1.1* as those release does not accept "empty" passwrod. Instead to remove "translation" authors decide to implement: (a) own PEM parser callback that return error on empty password and in addition (b) if PEM parser fail and password is empty to return "wrong password" Such work-around is suitable for (1) but useless for (3). Issue with this case is that it lacks information for password prompt. And so I cannot understand how in (1) returned failure reason is 'error in crypto library'. Of course for such reason code does not prompt for password. Regards, Roumen Petrov

[levitte]

Issue with this case is that it lacks information for password prompt. And so I cannot understand how in (1) returned failure reason is 'error in crypto library'.

That seems to happen because PEM_read_bio_PrivateKey returned NULL, but the OpenSSL error queue had no error recorded, so all SSH can report is that something went wrong in libcrypto.

There is a PR that reworks the error reports specifically for PEM and DER decoding, that was approved just a few hours ago, so it may be that ssh-add linked against OpenSSL master branch will work better within 24 hrs

[petrovr]

That seems to happen because PEM_read_bio_PrivateKey returned NULL, ...
Not really. Back to #13443 (comment)

This report lack information for password prompt. Let review code:

	clear_libcrypto_errors();
	if ((pk = PEM_read_bio_PrivateKey(bio, NULL, pem_passphrase_cb,
	    (char *)passphrase)) == NULL) {
	       /*
		* libcrypto may return various ASN.1 errors when attempting
		* to parse a key with an incorrect passphrase.
		* Treat all format errors as "incorrect passphrase" if a
		* passphrase was supplied.
		*/
		if (passphrase != NULL && *passphrase != '\0')
			r = SSH_ERR_KEY_WRONG_PASSPHRASE;
		else
			r = convert_libcrypto_error();
		goto out;
	}

First call is with empty password. In case of error (PREM read return NULL) code returns "wrong pass". Next step is "password promt" and then we could see "library error".

Look like fake issue.

[tpgxyz]

@petrovr If you need me to provide here extra information, do so then. Please guide me what you want and I'll deliver it.

[petrovr]

Sorry, I misread current OpenSSH code.

I have to change #13443 (comment) as without URI prefix issues are rendered to current site.

[petrovr]

It seems to me issue is not related to encoder/decoder errors.

Working with OpenSSL store requires suitable error management.
Decoder errors is normal to be excluded unless some one define new "store item for error on a particular entry". Such item could "transfer" errors" to reader. I cannot see any usable in this model.

In PKIX-SSH when is used key from a store and "STORE" does not return items code just return error reason selected from existing - SSH_ERR_KEY_NOT_FOUND.
At application level I do not care for exact error - UI method is provided and cryptographic library is responsible to perform suitable number of prompts and decoding.

So using STORE mean different error management. In this particular case return of key is required.

For instance if store does not return key raise PEM error with reason PEM_R_BAD_DECRYPT or PEM_R_BAD_PASSWORD_READ! Stop!!!

Does a development team would like to resolve issues in bloody third party code? Obviously no.

If code return error in PEM library with a new reason bloody error management in third party code will return invalid format - it is better but not enough to make ssh-add working.
External code must be adjusted to new reality.

[h-vetinari]

#14423 mentioned...

Fixing this should also fix these issues: #13443, #13493, #14100, #14566

... and was merged recently. Perhaps it's worth to recheck if this is still an issue on master (or with the most recent alpha)

[tpgxyz]

I just verified ssh-add with openssl-3.0.0-alpha16 and now it WORKS ! Thanks!

[lhotari]

I got this error message with a private key that had been copy pasted as text. It turns out that the file must end with a new line for it to work.

[MareoRaft]

I got this error message when using ssh-keygen -t ed25519 to create my key but not when using ssh-keygen (no options).

[petrovr]

MareoRaft wrote:

…

I got this error message when using `ssh-keygen -t ed25519` to create my key but not when using `ssh-keygen` (no options).

-t is used for key generation otherwise is ignored. Error in libcrypto could be raised only if is PKIX-SSH (release >= 13.0/3 Mar 2021, OpenSSL >= 1.1.1). MareoRaft, would you confirm? OpenBSD implementation does not use ed25519 if build is with OpenSSL. Regards, Roumen

[sbassi]

I got this error message when using ssh-keygen -t ed25519 to create my key but not when using ssh-keygen (no options).

Same to me, so I made a new key for the box with this problem.

[elalbaicin]

I get the same error when running ssh-add /root/.ssh/id_rsa. The error message:

Error loading key "/root/.ssh/id_rsa": error in libcrypto

Without further details.

[songleo]

I met the same error when using ssh to connect to my server (I have uploaded my public key to my server).

Load key \"/builds/soli/SSH_PRIVATE_KEY\": error in libcrypto

I am not sure if anyone has any hints. thanks

[kebeda]

I am also getting this error on Fedora 37, OpenSSH_8.8p1, OpenSSL 3.0.8 using ED25519 keys against a remote server using OpenSSH_7.4 pat OpenSSH_7.4* compat 0x04000006. I have two ED25519 keys that are used to connect to this server, one for a non-prod environment and one for a production one. The non-prod key works flawlessly every time. The prod one also works flawlessly when using DBeaver and the SSHJ implementation, but trying to use the same key file on the same computer via ssh in a terminal yields the error in libcrypto message. But only for the prod key! Hope this helps isolate where the problem might be.

[tlh24]

I just verified ssh-add with openssl-3.0.0-alpha16 and now it WORKS ! Thanks!

Yes, I had this problem too -- made a new key with ssh-keygen, then ran ssh-add. The second step fixed the (confusing) error: Load key ".ssh/id_rsa.pub": error in libcrypto

[retifrav]

private key that had been copy pasted as text
turns out that the file must end with a new line

If you are on Windows, and you do have newline in the end of the file, but you still get the error, then try to convert it to Unix-style, for example with dos2unix:

$ dos2unix.exe ~/.ssh/your-private-key

That fixed it in my case.

[samuelnihoul]

This occurred because the private key was missing, in my case.

[peterhorsley]

This occurred for me as I had an IdentityFile directive in ~/.ssh/config which was pointing to my public key. Changing it to point to the private key fixed the "error in libcrypto". Would be great if this error message was more specific.