Support for PKCS#8 pkey format

[cadedaniel]

Related to #387

For RSA private keys, apparently the rsa pkey header -----BEGIN RSA PRIVATE KEY----- is not always used. The header -----BEGIN PRIVATE KEY----- (sans RSA ) is used in some formats, like PKCS#8. source

The current implementation of paramiko pkey parses the private key file, looking for the following header:
beginning_of_key = '-----BEGIN ' + tag + ' PRIVATE KEY-----'
If it doesn't find it, it raises an SSHException('not a valid ' + tag + ' private key file').

Thus the consideration to have when refactoring pkey loading is to support the PKCS#8 format :)

I'd be happy to do a PR but like it has been said in 387, "most of the existing PRs poking at it are too limited in scope; this sort of change has a high chance for bugs and breaking backwards compatibility (intentionally or no) and I feel it needs a broadly considered update."

[bitprophet]

Thanks for the report, and greatly appreciate you taking #387 into consideration!

Part of me says that it might be plausible to add this particular wrinkle without substantially affecting the rest of the problem, but another part notes that it'd still be best to wait since it involves changing the body of PKey._read_private_key_file. (I'm close to finding the time to take an initial stab at all of that, FWIW.)

Does make me wonder if we could/should be relying more on other libraries for these commonly used formats; for example, our crypto backend knows how to write PKCS8 style RSA (etc) keys, as per here. It looks like it cannot read them yet but I wonder if @alex / @reaperhulk and co would be open to adding the functionality on that level.

At the moment we seem to split this region of functionality across the two libs - eg Cryptography on its own can load basic RSA/DSA/EC key files, so presumably we could leverage more of that than we are; but Cryptography can't do ED25519 - even though Alex added it to Paramiko recently, using lower level Crypto primitives.

[alex]

cryptography can read PKCS#8 keys, in addition to writing them.

I think clearing out all the PEM handling code in paramiko, and using cryptography as much as possible is a good idea.

cryptography will get native Ed25519 support as soon as OpenSSL does, which looks like it'll be OpenSSL 1.1.1.

[johnnybubonic]

just commenting on this so i can find it again so i can track when i can gen ed25519 keys with paramiko.

[sanjuroj]

I recently tried to install Fabric on my Mac and ran into the exact issue posted in #1363. The (valid) private key had a first line containing BEGIN OPENSSH PRIVATE KEY. When I attempted to use a simple Fabric test script, I got this Paramiko error: paramiko.ssh_exception.SSHException: not a valid RSA private key file. I hope this is the right place to post confirmation of this issue, since the #1363 was closed.

Do you know of any workaround for this? I don't quite know how to use the workaround that @kuzeyron posted to #1363, since I need both public and private keys.

Thanks!

[qinqon]

Hello have same issue after updateing my fedora, I used the workaround here (involves new keys :-( ) net-ssh/net-ssh#638 (comment)

ssh-keygen -m PEM -t rsa

[SharkFourSix]

Hello have same issue after updateing my fedora, I used the workaround here (involves new keys :-( ) net-ssh/net-ssh#638 (comment)

ssh-keygen -m PEM -t rsa

This doesn't seem to work still. Using Fabric

[bitprophet]

Unless I'm missing something there's no open PRs for this, but I am putting it in the p1 milestone for now hoping I'm wrong or somebody (or me!) has time to write one. The notes above by Alex give me hope that it should not be a lot of work or require scary new deps.

[bitprophet]

I'm actually wondering how valid this is at this point:

• OP (@cadedaniel) mentions PKCS#8 explicitly, specifically the BEGIN PRIVATE KEY header (no extra defining word between BEGIN and PRIVATE).
  • This is certainly a thing, but I don't see any obvious references to it in OpenSSH land, besides how ssh-keygen is capable of translating to or from this format into one that the rest of SSH actually understands.
  • E.g. I see code in openssh-portable for handling many other PEM style headers (including the recently added to Paramiko 'OpenSSH format', BEGIN OPENSSH PRIVATE KEY) but nothing that would handle PKCS#8 style.
  • While I'm academically open to us adding support for things OpenSSH itself lacks, it's super unusual, not least because the vastly most common use case for Paramiko and related code is to communicate with OpenSSH sshd's. That means my offhand priority for this is pretty low.
• The rest of the comments actually seem related to Cannot parse OpenSSH 6.5 formatted private keys #602 and friends, which is about that OpenSSH format, and is NOT the same thing as "wanting PKCS#8" as far as I can tell.

I'm going to take this off the table for Paramiko 2.7; if the originator can provide some details about where they're getting these PKCS#8 style keys and what SSHD(s) they're using them with, I'd be happy to include it in a later release. Thanks!

[cjrobbertse]

So there is no work around? Even the keygen side of things. e.g. use a different tool that produces the correct SSH key headers?

[bitprophet]

I'd guess you can use ssh-keygen -i -m PKCS8 -f path-to-pkcs8.key > openssh-formatted.key to translate such keys into the OpenSSH format; ought to be just fine as workarounds go.

I was just unable to give that a test on my end as I don't have one of these less-common keys around, couldn't just make one w/ ssh-keygen itself, and didn't have time to really dig into how to create them (looks like maybe one would use openssl?).

Further, without a solid idea of how users arrive at this kind of key (is it actually some semi common tool/workflow and I've just never run into it? or is it actually pretty rare and folks just have unusual requirements in their org? etc), it's hard to judge whether internal Paramiko support for them is worth development effort or if the best solution is to add an FAQ entry with that ssh-keygen command. Open to input here!

[WurstCommander]

I tried the following with a private ppk file:

puttygen convert to "Open SSH key" file.
This file I tried in several ways to convert to RSA / PKCS8 (with your command above adding -P and -m pem and so on) etc. nothing works.

Is there anyway to get a valid RSA file for paramiko from a ppk private file? I tried everything I could find.

[johnnybubonic]

@WurstCommander You need to convert it from the ppk format first before you can use ssh-keygen transformations on it (though you shouldn't need to after using puttygen). You can do this on linux by installing the linux build of puttygen (most distros have it in a package named "putty-tools" or just "putty"):

puttygen keyfile.ppk -O private-openssh-new -o convertedkey  # DON'T use "private-openssh", use "private-openssh-new"
puttygen keyfile.ppk -O  public-openssh -o convertedkey.pub

[johnnybubonic]

Note that the above will generate a key with the new format, "BEGIN OPENSSH PRIVATE KEY". This is the format that ssh now uses by default (which IIRC @bitprophet just added support for).

If you used -O private-openssh instead, it'd generate a key with the legacy format (e.g. "BEGIN RSA PRIVATE KEY").

[cjrobbertse]

This may not be relevant to all, but, I managed to work around my problem, and now my application works as intended. I was using Fabric to perform a command on a remote server, using an OpenSSH private/public key pair.

c = Connection(
    host="server_ip",
    user="username",
    connect_kwargs={
        "key_filename": "/path/to/private_key"
    })
result = c.run('uname -s', hide=True)
print(result)

This gave the error:

ValueError: q must be exactly 160, 224, or 256 bits long

---

To get my application to work, instead of using Fabric, I decided to use Paramiko directly and it worked the first time.

k = paramiko.RSAKey.from_private_key_file("/path/to/private_key")
    # Create an SSH client
    c = paramiko.SSHClient()
    # Add the key policy to the client
    c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    app.logger.info("Connecting to server")
    # Create the connection to the hostname, using the username, with the private key defined before
    c.connect(hostname = "hostname", username = "username", pkey = k)
    app.logger.info("Connected")
    app.logger.info("Executing {}".format(inp_cmd))
    # Assing the result variables from executing the command
    c_stdin, c_stdout, c_stderr = c.exec_command(inp_cmd)
    # Assign the comand output to result.
    result = c_stdout.read()
    # Log the result
    app.logger.info(result)
    # Log errors
    app.logger.info("Errors")
    app.logger.info(c_stderr.read())
    # Close connection
    c.close()

So in summary; Using Fabric did not work. Using Paramiko did!

[WurstCommander]

@WurstCommander You need to convert it from the ppk format first before you can use ssh-keygen transformations on it (though you shouldn't need to after using puttygen). You can do this on linux by installing the linux build of puttygen (most distros have it in a package named "putty-tools" or just "putty"):

puttygen keyfile.ppk -O private-openssh-new -o convertedkey  # DON'T use "private-openssh", use "private-openssh-new"
puttygen keyfile.ppk -O  public-openssh -o convertedkey.pub

Thanks for your help, I did the stuff you mentioned.

I got now a key with

as the first line of the keyfile

import os
import paramiko 

host = 'myhost'
myusername = 'myuser'

output = "default"


key = paramiko.RSAKey.from_private_key_file('convertedkey', password='mypass')

def connect:
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(host, username=myusername, pkey=key) 
   ---other stuff
     ssh.close()

and still it get:

Traceback (most recent call last):
  File "c:\Users\xx\.vscode\extensions\ms-python.python-2020.1.57204\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\xx\.vscode\extensions\ms-python.python-2020.1.57204\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\xx\.vscode\extensions\ms-python.python-2020.1.57204\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\Program Files (x86)\python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files (x86)\python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files (x86)\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\0815_Spielwiese\pythonarchiv\SapLogAnalyser\main.py", line 10, in <module>
    key = paramiko.RSAKey.from_private_key_file('convertedkey', password='xxxx')
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\pkey.py", line 235, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\rsakey.py", line 55, in __init__
    self._from_private_key_file(filename, password)
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\rsakey.py", line 175, in _from_private_key_file
    data = self._read_private_key_file("RSA", filename, password)
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\pkey.py", line 308, in _read_private_key_file
    data = self._read_private_key(tag, f, password)
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\pkey.py", line 337, in _read_private_key
    data = self._read_private_key_openssh(lines[start:end], password)
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\pkey.py", line 481, in _read_private_key_openssh
    return _unpad_openssh(keydata)
  File "C:\Program Files (x86)\python37\lib\site-packages\paramiko\pkey.py", line 55, in _unpad_openssh
    raise SSHException("Invalid key")
paramiko.ssh_exception.SSHException: Invalid key

[johnnybubonic]

@WurstCommander what does python3 -c 'import paramiko; print(paramiko.__version__)' give?

[WurstCommander]

@johnnybubonic
python -c 'import paramiko; print(paramiko.__version__)'

2.7.1

FYI: Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:05:16)

Which seems to be the newest version:

pip install --upgrade paramiko
Requirement already up-to-date: paramiko in c:\program files (x86)\python37\lib\site-packages (2.7.1)
Requirement already satisfied, skipping upgrade: cryptography>=2.5 in c:\program files (x86)\python37\lib\site-packages (from paramiko) (2.8)
Requirement already satisfied, skipping upgrade: bcrypt>=3.1.3 in c:\program files (x86)\python37\lib\site-packages (from paramiko) (3.1.7)
Requirement already satisfied, skipping upgrade: pynacl>=1.0.1 in c:\program files (x86)\python37\lib\site-packages (from paramiko) (1.3.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in c:\program files (x86)\python37\lib\site-packages (from cryptography>=2.5->paramiko) (1.13.2)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in c:\users\a10fc30\appdata\roaming\python\python37\site-packages (from cryptography>=2.5->paramiko) (1.13.0)
Requirement already satisfied, skipping upgrade: pycparser in c:\program files (x86)\python37\lib\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.5->paramiko) (2.19)

[ploxiln]

@WurstCommander that key fails this check in _unpad_openssh() near the top of pkey.py:

    if 0x20 <= padding_length < 0x7f:
        return data  # no padding, last byte part comment (printable ascii)
    if padding_length > 15:
        raise SSHException("Invalid key")

Does your key have 16 bytes of padding? (Does changing 15 to 16 make it work?) Or does your key have an embedded comment which ends with a non-ascii non-printable character? (newline char? utf8 for some accented character?)

[WurstCommander]

@ploxiln
Sorry for my late response. I'm not knowledgeable regarding padding. How can I see the padding of the key? I searched with google and found nothing helpful.

This is the pagent info of the original key (which I converted like johnnybubonic said above)

Thank you for your help

[eighthave]

As of OpenSSL v3.0, openssl genrsa and openssl rsa both default to outputting PKCS#8, making those key files unusable with Paramiko.
https://www.openssl.org/docs/man3.0/man1/openssl-rsa.html#traditional