add support for new OpenSSH private key format #1343

jaredhobbs · 2018-11-28T00:25:09Z

This work is based off the work done in #618


        add support for new OpenSSH private key format

This work is based off the work done in #618


        blacken


        blacken


        blacken, tests


        try to fix test

ploxiln · 2018-11-29T00:29:50Z

Is it not possible for the code supporting the new format in ed25519key.py to be removed, and for Ed25519Key to use the shared code now in PKey for that instead?

jaredhobbs · 2018-11-30T16:02:44Z

Probably. I'll investigate when I get some free time but I'm pretty swamped at the moment.

thomasd3 · 2018-12-10T09:39:31Z

Is there any ETA for when the new format could be supported?


        ignore few rounds in bcrypt kdf call

carumusan · 2019-02-12T09:27:01Z

Please merge this. Looks like since OpenSSH 7.8 (released 6 months ago 2018-08-24) the new format is the default. https://www.openssh.com/txt/release-7.8

dgjustice · 2019-03-06T23:23:51Z

Bump! Fedora 29, openssh 7.9p1. Paramiko can't read my private key files without this change.

bdrydyk · 2019-03-07T23:50:33Z

Bumping this as well. This has broken my automation from OS X Mojave

mitom · 2019-03-13T11:54:16Z

Is there anything blocking this that requires help? Lacking this is causing major issues for us.


        handle exception that can occur if no regex match

kindlehl · 2019-03-31T01:33:01Z

I'm currently having issues as well. Broken on Debian 9


        add noqa comments

aedelbro · 2019-04-25T13:31:34Z

Would it not be better to make it a raw string and drop flake comment?
r"^-{5}BEGIN (RSA|DSA|EC|OPENSSH) PRIVATE KEY-{5}\s*$"
https://lintlyci.github.io/Flake8Rules/rules/W605.html


        Use raw string

jaredhobbs · 2019-04-25T13:36:58Z

Would it not be better to make it a raw string and drop flake comment?
r"^-{5}BEGIN (RSA|DSA|EC|OPENSSH) PRIVATE KEY-{5}\s*$"
https://lintlyci.github.io/Flake8Rules/rules/W605.html

Yes, fixed.

pgierz · 2019-05-06T09:56:12Z

Any idea when this will go in?

ploxiln · 2019-05-22T18:10:00Z

paramiko/ecdsakey.py

+        elif pkformat == self.PRIVATE_KEY_FORMAT_OPENSSH:
+            curve, verkey, sigkey = self._uint32_cstruct_unpack(data, "sss")
+            try:
+                key = ec.derive_private_key(sigkey, curve, default_backend())


this doesn't work: sigkey needs to be an integer, and curve needs to be translated into one of the Curve objects in _ECDSA_CURVES

noticed while working on my port and cleanup of this. here's a test to show the errors:
https://github.com/ploxiln/paramiko-ng/pull/13/files#diff-523fff3457d80985c9d2429eb3aecb6e
the fix is mixed into my large refactor, since I figured this out towards the end:
https://github.com/ploxiln/paramiko-ng/pull/13/files#diff-00344f8a080e03112e93210cdfd37302

ploxiln · 2019-05-22T18:14:16Z

paramiko/pkey.py

+
+        # Remove padding
+        padlen = byte_ord(keydata[len(keydata) - 1])
+        return keydata[: len(keydata) - padlen]


this unpad suffers from roughly the same bug I fixed for Ed25519Key in #1400

greenscar · 2019-06-17T18:37:19Z

any movement? We all know Apple is the new Microsoft, doing everything their own way. However, It kinda sucks when I can't run Ansible / Testinfra from my mac box due to
paramiko.ssh_exception.SSHException: not a valid RSA private key file"

newearthmartin · 2019-08-23T11:07:13Z

Can someone please merge this? It has been open for months. Fabric/paramiko is failing on mac.

joaquinclearmetal · 2019-08-25T13:00:36Z

It would be nice if Paramiko supported SSH. I have to recreate keys if they were created on macOS (and apparently this affects Fedora).

ChrisEdgington · 2019-10-07T19:06:41Z

I was able to fix this problem by converting my macos id_rsa to pem format via:

ssh-keygen -p -m PEM -f ~/.ssh/id_rsa

cbonitz · 2019-10-16T09:57:25Z

OpenSSH formatted private keys seem to be the default on Ubuntu 19.04, just like on MacOS and Fedora as mentioned by previous commenters. This is not an edge case anymore, this will concern almost all users of unix-like operating systems very soon.

XVilka · 2019-11-07T06:21:27Z

Would be nice to have this finally merged.

jamesharris-garmin · 2019-11-25T16:13:51Z

Could we please just run black on pkey and resolve the tests on tests/test_pkey.py?

bitprophet · 2019-12-03T19:33:05Z

Diving into reviewing/tweaking/merging this now. Thanks for everyone who's been keeping the patchset rolling, and for everyone's patience 🙏

bitprophet · 2019-12-03T20:40:05Z

Merged to latest master & blackened, taking a look:

No changelog? 😢 Added an entry and linked to most of the major related tickets (sob, there are so many) including crediting a pile of folks ❤️
This code only seems to support BEGIN OPENSSH PRIVATE KEY style heads and not "real" RFC 4716 style headers which are BEGIN SSH2 PRIVATE KEY.
- However, since ssh-keygen only appears to spit out the former header style, and given I'm not experienced with these formats yet myself, I'm going to leave this be.
  - Suspect OpenSSH departed from the RFC afterwards for whatever reason, and this problem domain (OpenSSH compatibility) is definitely one where practice trumps theory 😂
- Should be pretty easy to extend it later for the variant if folks need it.
Renamed the new keys a bit, it wasn't clear to me offhand why they were named <size>_o.key, guessing o for OpenSSH, but I like obvious naming...
Not a huge fan of the (as far as I can tell) internal-only format signifier constants (eg PKey.PRIVATE_KEY_FORMAT_ORIGINAL) but it's not a great use of my time to clean things up to that level, so I settled for:
- making them private so they're not part of the public API and can be tweaked later
- updating the exceptions raised when they go off the rails (they were not consistent or terrifically readable).
  - also I'm honestly not even sure how we can end up in that particular code branch (the code that spits out the pkformat value would raise its own exception first!) but again, going to avoid rocking the boat too much for now, in case I'm misreading this.
Renamed references to "old/new format" as "pem/openssh" to be clearer (to me, anyway)
~~Added tests for the exception behavior around invalid key types (this is, AFAICT, an original sin, but this is a great time to add new tests for it)~~ Never mind, this stuff is still too hairy to mess with outside of a #387 style overhaul
It's not clear to me why PKey._uint32_cstruct_unpack does a blanket "any exception is transmuted into an equivalent SSHException". I don't recall this being any sort of contract of Paramiko's behavior, and if anything, we are trying to dial down the abuse of "everything is this one exception class" from the old days. All this does it make it harder to tell quite what broke?
- After more digging: this is likely because a bunch of mid-level code such as SSHClient attempts to ask the question "did code that reads SSH keys have a problem with a given key? Ignore that for now and move on to the next auth method".
- I really don't like this (that code should arguably just be doing a bare except, as that is one of the few acceptable use cases for such) but it is an established pattern, just one I'd not been super aware of. Something to axe in #387 too probably. Settled for a comment for now.
Cleaned up some "foo" + str(val) + "bar" style string interpolation. Folks, it's 2019, we have .format() and (once we drop Python <3.5...in, probably, 2030...) f-strings now. (This is admittedly something around the codebase from its 2004 days, as well...got some of those too...)
Did some final real world manual tests with new keys generated on my (macOS Mojave) system's ssh-keygen.

bitprophet · 2019-12-03T22:10:13Z

This will be out in Paramiko 2.7 (only one more ticket after this one before that's cut!)

grant-humphries · 2019-12-03T22:43:05Z

@bitprophet thank you for your work on this!!

jaredhobbs · 2019-12-03T22:51:53Z

thanks so much!

johnnybubonic · 2019-12-03T23:14:13Z

yay

please don't forget about #1136 and #1261 too! they should be able to be resolved now that this format's supported.

bitprophet · 2019-12-03T23:29:13Z

@johnnybubonic I put 'em in a milestone, I can't say when they'll be gotten to but at least they're on my radar now, thank you!


        Delete duplicated OPENSSH-format parsing in Ed25519Key

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>


        Delete duplicated OPENSSH-format parsing in Ed25519Key

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>


        Delete duplicated OPENSSH-format parsing in Ed25519Key

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>


        Delete duplicated OPENSSH-format parsing in Ed25519Key

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

jaredhobbs added 5 commits Nov 28, 2018

add support for new OpenSSH private key format

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

eff204f

This work is based off the work done in #618

blacken

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

e72aeaf

blacken

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

6b374a3

blacken, tests

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

2dcdba7

try to fix test

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

79a6bbf

jaredhobbs mentioned this pull request Nov 29, 2018

Cannot parse OpenSSH 6.5 formatted private keys #602

Open

ploxiln mentioned this pull request Dec 3, 2018

"not a valid RSA private key file" on Mac #1348

Closed

ignore few rounds in bcrypt kdf call

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

242be1a

ploxiln mentioned this pull request Jan 30, 2019

not a valid RSA private key file #1930

Closed

handle exception that can occur if no regex match

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

8d57be2

add noqa comments

Verified

This commit was signed with a verified signature.

jaredhobbs Jared Hobbs

GPG key ID: 9F830F8241706AB0 Learn about signing commits

Loading status checks…

f9d8c03

Use raw string

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

25de1a7

ploxiln reviewed May 22, 2019

View changes

paramiko/pkey.py

# Remove padding

padlen = byte_ord(keydata[len(keydata) - 1])

return keydata[: len(keydata) - padlen]

This comment has been minimized.

Sign in to view

Copy link

ploxiln May 22, 2019
Contributor

this unpad suffers from roughly the same bug I fixed for Ed25519Key in #1400

rmelotte mentioned this pull request Oct 9, 2019

Revive beerocks analyzer #308

Merged

ploxiln mentioned this pull request Nov 4, 2019

Paramiko Invalid key on ed25519 host key #1517

Open

This was referenced Nov 16, 2019

ECDSA keys parsing fails due to key format changes #1548

Closed

Authentication failed with public key #1135

Closed

bitprophet added this to the p0 milestone Nov 25, 2019

ploxiln mentioned this pull request Nov 25, 2019

Add functions to auto-detect and load any private key type #1558

Open

paramiko deleted a comment from XVilka Dec 3, 2019

paramiko deleted a comment from jamesharris-garmin Dec 3, 2019

bitprophet merged commit 25de1a7 into paramiko:master Dec 3, 2019
1 check failed

1 check failed

continuous-integration/travis-ci/pr The Travis CI build failed
Details

This was referenced Dec 5, 2019

bug of parsing new format keys #1565

Closed

fix loading ECDSA key in new openssh private-key format #1566

Merged

fix new openssh private key format when no padding #1567

Merged

intgr mentioned this pull request Dec 11, 2019

Delete duplicated OPENSSH-format parsing in Ed25519Key #1571

Open

paramiko / paramiko

Sponsor paramiko/paramiko

Join GitHub today

add support for new OpenSSH private key format #1343

add support for new OpenSSH private key format #1343

Conversation

jaredhobbs commented Nov 28, 2018

This comment has been minimized.

ploxiln commented Nov 29, 2018

This comment has been minimized.

jaredhobbs commented Nov 30, 2018

This comment has been minimized.

thomasd3 commented Dec 10, 2018

This comment has been minimized.

carumusan commented Feb 12, 2019 • edited

This comment has been minimized.

dgjustice commented Mar 6, 2019

This comment has been minimized.

bdrydyk commented Mar 7, 2019

This comment has been minimized.

mitom commented Mar 13, 2019

This comment has been minimized.

kindlehl commented Mar 31, 2019

This comment has been minimized.

aedelbro commented on f9d8c03 Apr 25, 2019

This comment has been minimized.

jaredhobbs commented Apr 25, 2019

This comment has been minimized.

pgierz commented May 6, 2019

This comment has been minimized.

ploxiln May 22, 2019 • edited

This comment has been minimized.

ploxiln May 22, 2019

This comment has been minimized.

greenscar commented Jun 17, 2019

This comment has been minimized.

newearthmartin commented Aug 23, 2019 • edited

This comment has been minimized.

joaquinclearmetal commented Aug 25, 2019

This comment has been minimized.

ChrisEdgington commented Oct 7, 2019

This comment has been minimized.

cbonitz commented Oct 16, 2019

This comment has been minimized.

XVilka commented Nov 7, 2019

This comment has been minimized.

jamesharris-garmin commented Nov 25, 2019

This comment has been minimized.

bitprophet commented Dec 3, 2019

This comment has been minimized.

bitprophet commented Dec 3, 2019 • edited

This comment has been minimized.

bitprophet commented Dec 3, 2019

This comment has been minimized.

grant-humphries commented Dec 3, 2019

This comment has been minimized.

jaredhobbs commented Dec 3, 2019

This comment has been minimized.

johnnybubonic commented Dec 3, 2019 • edited

This comment has been minimized.

bitprophet commented Dec 3, 2019 • edited

carumusan commented Feb 12, 2019 •

edited

aedelbro commented on `f9d8c03` Apr 25, 2019

ploxiln May 22, 2019 •

edited

newearthmartin commented Aug 23, 2019 •

edited

bitprophet commented Dec 3, 2019 •

edited

johnnybubonic commented Dec 3, 2019 •

edited

bitprophet commented Dec 3, 2019 •

edited