add support for new OpenSSH private key format #1343

jaredhobbs · 2018-11-28T00:25:09Z

This work is based off the work done in #618

This work is based off the work done in paramiko#618

ploxiln · 2018-11-29T00:29:50Z

Is it not possible for the code supporting the new format in ed25519key.py to be removed, and for Ed25519Key to use the shared code now in PKey for that instead?

jaredhobbs · 2018-11-30T16:02:44Z

Probably. I'll investigate when I get some free time but I'm pretty swamped at the moment.

thomasd3 · 2018-12-10T09:39:31Z

Is there any ETA for when the new format could be supported?

carumusan · 2019-02-12T09:27:01Z

Please merge this. Looks like since OpenSSH 7.8 (released 6 months ago 2018-08-24) the new format is the default. https://www.openssh.com/txt/release-7.8

dgjustice · 2019-03-06T23:23:51Z

Bump! Fedora 29, openssh 7.9p1. Paramiko can't read my private key files without this change.

bdrydyk · 2019-03-07T23:50:33Z

Bumping this as well. This has broken my automation from OS X Mojave

mitom · 2019-03-13T11:54:16Z

Is there anything blocking this that requires help? Lacking this is causing major issues for us.

kindlehl · 2019-03-31T01:33:01Z

I'm currently having issues as well. Broken on Debian 9

aedelbro · 2019-04-25T13:31:34Z

Would it not be better to make it a raw string and drop flake comment?
r"^-{5}BEGIN (RSA|DSA|EC|OPENSSH) PRIVATE KEY-{5}\s*$"
https://lintlyci.github.io/Flake8Rules/rules/W605.html

jaredhobbs · 2019-04-25T13:36:58Z

Would it not be better to make it a raw string and drop flake comment?
r"^-{5}BEGIN (RSA|DSA|EC|OPENSSH) PRIVATE KEY-{5}\s*$"
https://lintlyci.github.io/Flake8Rules/rules/W605.html

Yes, fixed.

pgierz · 2019-05-06T09:56:12Z

Any idea when this will go in?

ploxiln · 2019-05-22T18:10:00Z

paramiko/ecdsakey.py

+        elif pkformat == self.PRIVATE_KEY_FORMAT_OPENSSH:
+            curve, verkey, sigkey = self._uint32_cstruct_unpack(data, "sss")
+            try:
+                key = ec.derive_private_key(sigkey, curve, default_backend())


this doesn't work: sigkey needs to be an integer, and curve needs to be translated into one of the Curve objects in _ECDSA_CURVES

noticed while working on my port and cleanup of this. here's a test to show the errors:
https://github.com/ploxiln/paramiko-ng/pull/13/files#diff-523fff3457d80985c9d2429eb3aecb6e
the fix is mixed into my large refactor, since I figured this out towards the end:
https://github.com/ploxiln/paramiko-ng/pull/13/files#diff-00344f8a080e03112e93210cdfd37302

ploxiln · 2019-05-22T18:14:16Z

paramiko/pkey.py

+
+        # Remove padding
+        padlen = byte_ord(keydata[len(keydata) - 1])
+        return keydata[: len(keydata) - padlen]


this unpad suffers from roughly the same bug I fixed for Ed25519Key in #1400

greenscar · 2019-06-17T18:37:19Z

any movement? We all know Apple is the new Microsoft, doing everything their own way. However, It kinda sucks when I can't run Ansible / Testinfra from my mac box due to
paramiko.ssh_exception.SSHException: not a valid RSA private key file"

jamesharris-garmin · 2019-11-25T16:13:51Z

Could we please just run black on pkey and resolve the tests on tests/test_pkey.py?

bitprophet · 2019-12-03T19:33:05Z

Diving into reviewing/tweaking/merging this now. Thanks for everyone who's been keeping the patchset rolling, and for everyone's patience 🙏

bitprophet · 2019-12-03T20:40:05Z

Merged to latest master & blackened, taking a look:

No changelog? 😢 Added an entry and linked to most of the major related tickets (sob, there are so many) including crediting a pile of folks ❤️
This code only seems to support BEGIN OPENSSH PRIVATE KEY style heads and not "real" RFC 4716 style headers which are BEGIN SSH2 PRIVATE KEY.
- However, since ssh-keygen only appears to spit out the former header style, and given I'm not experienced with these formats yet myself, I'm going to leave this be.
  - Suspect OpenSSH departed from the RFC afterwards for whatever reason, and this problem domain (OpenSSH compatibility) is definitely one where practice trumps theory 😂
- Should be pretty easy to extend it later for the variant if folks need it.
Renamed the new keys a bit, it wasn't clear to me offhand why they were named <size>_o.key, guessing o for OpenSSH, but I like obvious naming...
Not a huge fan of the (as far as I can tell) internal-only format signifier constants (eg PKey.PRIVATE_KEY_FORMAT_ORIGINAL) but it's not a great use of my time to clean things up to that level, so I settled for:
- making them private so they're not part of the public API and can be tweaked later
- updating the exceptions raised when they go off the rails (they were not consistent or terrifically readable).
  - also I'm honestly not even sure how we can end up in that particular code branch (the code that spits out the pkformat value would raise its own exception first!) but again, going to avoid rocking the boat too much for now, in case I'm misreading this.
Renamed references to "old/new format" as "pem/openssh" to be clearer (to me, anyway)
~~Added tests for the exception behavior around invalid key types (this is, AFAICT, an original sin, but this is a great time to add new tests for it)~~ Never mind, this stuff is still too hairy to mess with outside of a #387 style overhaul
It's not clear to me why PKey._uint32_cstruct_unpack does a blanket "any exception is transmuted into an equivalent SSHException". I don't recall this being any sort of contract of Paramiko's behavior, and if anything, we are trying to dial down the abuse of "everything is this one exception class" from the old days. All this does it make it harder to tell quite what broke?
- After more digging: this is likely because a bunch of mid-level code such as SSHClient attempts to ask the question "did code that reads SSH keys have a problem with a given key? Ignore that for now and move on to the next auth method".
- I really don't like this (that code should arguably just be doing a bare except, as that is one of the few acceptable use cases for such) but it is an established pattern, just one I'd not been super aware of. Something to axe in #387 too probably. Settled for a comment for now.
Cleaned up some "foo" + str(val) + "bar" style string interpolation. Folks, it's 2019, we have .format() and (once we drop Python <3.5...in, probably, 2030...) f-strings now. (This is admittedly something around the codebase from its 2004 days, as well...got some of those too...)
Did some final real world manual tests with new keys generated on my (macOS Mojave) system's ssh-keygen.

bitprophet · 2019-12-03T22:10:13Z

This will be out in Paramiko 2.7 (only one more ticket after this one before that's cut!)

grant-humphries · 2019-12-03T22:43:05Z

@bitprophet thank you for your work on this!!

jaredhobbs · 2019-12-03T22:51:53Z

thanks so much!

johnnybubonic · 2019-12-03T23:14:13Z

yay

please don't forget about #1136 and #1261 too! they should be able to be resolved now that this format's supported.

bitprophet · 2019-12-03T23:29:13Z

@johnnybubonic I put 'em in a milestone, I can't say when they'll be gotten to but at least they're on my radar now, thank you!

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

@ploxiln

The Ed25519Key class contained key parsing and decryption logic for OPENSSH-format keys before that code became generic. Now that paramiko#1343 is merged, this logic becomes redundant and can be removed. This cleanup was extracted from ploxiln#13 and the original author @ploxiln is credited as commit author even though I (@intgr) am submitting this pull request. Signed-off-by: Marti Raudsepp <marti@juffo.org>

jaredhobbs added 5 commits Nov 28, 2018

add support for new OpenSSH private key format

eff204f

This work is based off the work done in paramiko#618

blacken

e72aeaf

blacken

6b374a3

blacken, tests

2dcdba7

try to fix test

79a6bbf

jaredhobbs mentioned this pull request Nov 29, 2018

Cannot parse OpenSSH 6.5 formatted private keys #602

Open

ploxiln mentioned this pull request Dec 3, 2018

"not a valid RSA private key file" on Mac #1348

Closed

ignore few rounds in bcrypt kdf call

242be1a

ploxiln mentioned this pull request Jan 30, 2019

not a valid RSA private key file fabric/fabric#1930

Closed

handle exception that can occur if no regex match

8d57be2

add noqa comments

f9d8c03

Use raw string

25de1a7

ploxiln mentioned this pull request May 1, 2019

Fabric can't connect to SSH server while ssh command works fine fabric/fabric#1972

Closed

ploxiln mentioned this pull request May 1, 2019

paramiko paramiko.ssh_exception.SSHException: using keyfile #1358

Closed

ploxiln mentioned this pull request May 1, 2019

Raises 'Invalid key' on valid OPENSSH private key #1430

Closed

ploxiln mentioned this pull request May 2, 2019

add support for new OpenSSH private key format ploxiln/paramiko-ng#13

Merged

ploxiln reviewed May 22, 2019

View changes

This comment was marked as disruptive content.

Sign in to view

ploxiln mentioned this pull request Nov 16, 2019

ECDSA keys parsing fails due to key format changes #1548

Closed

ploxiln mentioned this pull request Nov 19, 2019

Authentication failed with public key #1135

Closed

bitprophet added this to the p0 milestone Nov 25, 2019

ploxiln mentioned this pull request Nov 25, 2019

Add functions to auto-detect and load any private key type #1558

Open

paramiko deleted a comment from XVilka Dec 3, 2019

paramiko deleted a comment from jamesharris-garmin Dec 3, 2019

bitprophet merged commit 25de1a7 into paramiko:master Dec 3, 2019
1 check failed

ploxiln mentioned this pull request Dec 5, 2019

bug of parsing new format keys #1565

Closed

ploxiln mentioned this pull request Dec 5, 2019

fix loading ECDSA key in new openssh private-key format #1566

Merged

ploxiln mentioned this pull request Dec 5, 2019

fix new openssh private key format when no padding #1567

Merged

intgr mentioned this pull request Dec 11, 2019

Delete duplicated OPENSSH-format parsing in Ed25519Key #1571

Open

ploxiln mentioned this pull request Mar 17, 2020

Why use paramiko-ng? ploxiln/fab-classic#28

Closed

ltalirz mentioned this pull request Jul 6, 2020

DOCS: Add computer and code setup how-tos aiidateam/aiida-core#4216

Merged

ltalirz mentioned this pull request Jul 6, 2020

update paramiko to 2.7 aiidateam/aiida-core#4222

Merged

totaam mentioned this pull request Jan 22, 2021

paramiko does not support the new openssh key format Xpra-org/xpra#2307

Closed

add support for new OpenSSH private key format #1343

add support for new OpenSSH private key format #1343

jaredhobbs commented Nov 28, 2018

ploxiln commented Nov 29, 2018

jaredhobbs commented Nov 30, 2018

thomasd3 commented Dec 10, 2018

carumusan commented Feb 12, 2019 •

edited

dgjustice commented Mar 6, 2019

bdrydyk commented Mar 7, 2019

mitom commented Mar 13, 2019

kindlehl commented Mar 31, 2019

aedelbro commented on `f9d8c03` Apr 25, 2019

jaredhobbs commented Apr 25, 2019

pgierz commented May 6, 2019

ploxiln May 22, 2019 •

edited

ploxiln May 22, 2019

greenscar commented Jun 17, 2019

This comment was marked as disruptive content.

jamesharris-garmin commented Nov 25, 2019

bitprophet commented Dec 3, 2019

bitprophet commented Dec 3, 2019 •

edited

bitprophet commented Dec 3, 2019

grant-humphries commented Dec 3, 2019

jaredhobbs commented Dec 3, 2019

johnnybubonic commented Dec 3, 2019 •

edited

bitprophet commented Dec 3, 2019 •

edited

add support for new OpenSSH private key format #1343

add support for new OpenSSH private key format #1343

Conversation

jaredhobbs commented Nov 28, 2018

ploxiln commented Nov 29, 2018

jaredhobbs commented Nov 30, 2018

thomasd3 commented Dec 10, 2018

carumusan commented Feb 12, 2019 • edited

dgjustice commented Mar 6, 2019

bdrydyk commented Mar 7, 2019

mitom commented Mar 13, 2019

kindlehl commented Mar 31, 2019

aedelbro commented on f9d8c03 Apr 25, 2019

Choose a reason for hiding this comment

jaredhobbs commented Apr 25, 2019

pgierz commented May 6, 2019

ploxiln May 22, 2019 • edited

Choose a reason for hiding this comment

ploxiln May 22, 2019

Choose a reason for hiding this comment

greenscar commented Jun 17, 2019

This comment was marked as disruptive content.

jamesharris-garmin commented Nov 25, 2019

bitprophet commented Dec 3, 2019

bitprophet commented Dec 3, 2019 • edited

bitprophet commented Dec 3, 2019

grant-humphries commented Dec 3, 2019

jaredhobbs commented Dec 3, 2019

johnnybubonic commented Dec 3, 2019 • edited

bitprophet commented Dec 3, 2019 • edited

carumusan commented Feb 12, 2019 •

edited

aedelbro commented on `f9d8c03` Apr 25, 2019

ploxiln May 22, 2019 •

edited

bitprophet commented Dec 3, 2019 •

edited

johnnybubonic commented Dec 3, 2019 •

edited

bitprophet commented Dec 3, 2019 •

edited