Cannot authenticate with a private key #363

walac · 2022-10-04T12:58:41Z

Describe the bug

When I try to connect to a host using a private key, it doesn't work. With the ssh command line client, it works fines.

To Reproduce

Steps to reproduce the behavior:

Run podman run -p 2022:22 docker.io/walac/pyktest-ssh:latest (the docker image is publicly available)
Run the following python script

from pssh.clients import SSHClient
import logging as l

l.getLogger('pssh.host_logger').setLevel(l.DEBUG)
l.getLogger('pssh').setLevel(l.DEBUG)

pkey = \
b"""-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEApq7NTeQ4JsLMTdZdiV3UMUzvj8wcmZbsMozUnNXM6LK9fBxZkbx9
/rTZlgPMBHFhJZcTMAH4/gkWJZ4yBMCHu1oh9jyzPNbdSMcCKlJxMzmnTebGDbK7acBBiC
hHOy2IzIbaM/M6bqk+OKpbCRx0rPNcwAx/YZq4E73+hZ3ZnODV0n7UAS2Z42MBFOW8/Jn7
+XyKQLozjn+iEs/BZ5SKU5d6WNpbqh/Dka5pG3HhtQDU3zq4nPn53mE2jYZly5it7dBPv9
C985FHPDXmeO+l6DNPFtpX+MdTW6Umf2ZBuSMai2p9idE/BsF69j4kuLBT8LgjT3sqX4Ph
1pA4zO1w2Uuszv6We6PlQ/BX4n+k1SPc+EiPljK9RxRB8pDCNuZI8yWlrloCCmyE7KSOkr
Z0l7aArHs4i6fqL66bwD3ZZfHqfMV/8VwIQV//YQqYfugmWqsbFPeEJkoNb0pnUYfFiRy3
4qCsRz3pfxXeWhIoQsT8iLKvYvZDaNmrDIdjXw1TAAAFiHgCmvh4Apr4AAAAB3NzaC1yc2
EAAAGBAKauzU3kOCbCzE3WXYld1DFM74/MHJmW7DKM1JzVzOiyvXwcWZG8ff602ZYDzARx
YSWXEzAB+P4JFiWeMgTAh7taIfY8szzW3UjHAipScTM5p03mxg2yu2nAQYgoRzstiMyG2j
PzOm6pPjiqWwkcdKzzXMAMf2GauBO9/oWd2Zzg1dJ+1AEtmeNjARTlvPyZ+/l8ikC6M45/
ohLPwWeUilOXeljaW6ofw5GuaRtx4bUA1N86uJz5+d5hNo2GZcuYre3QT7/QvfORRzw15n
jvpegzTxbaV/jHU1ulJn9mQbkjGotqfYnRPwbBevY+JLiwU/C4I097Kl+D4daQOMztcNlL
rM7+lnuj5UPwV+J/pNUj3PhIj5YyvUcUQfKQwjbmSPMlpa5aAgpshOykjpK2dJe2gKx7OI
un6i+um8A92WXx6nzFf/FcCEFf/2EKmH7oJlqrGxT3hCZKDW9KZ1GHxYkct+KgrEc96X8V
3loSKELE/Iiyr2L2Q2jZqwyHY18NUwAAAAMBAAEAAAGANzoy3yyPaFRh4iZ1TobuwDkIJS
KRlVg6wZME/UQfxTg37U/tY4rLSmH8uCZg6lXwxMY2PtFggTdchbFRTF7IekymdRJupulg
X3VE4+X2CO2A4CBnHfHLBAKGFAmYdGSlb3L9CHp+MV1VNuxHBpFNnWJzKFTbvejINg5dL0
N6b27Vmg2YzB0iYIOUv6pRQHb61JKtsj2fyIbEmiwJHntKi+DkUl1jQs6Me5AI1mWYzeLN
JXus4KI45rRaIXFW+CsJaFotgbRdXLcS23nehPhdzzD+vLlvb3uHIxtV4vFnjpVthH94Ov
Sr6jNXp+U9r6GqO9PrZe5FImFkA1ACd9HbkTCaptAoRorq/EG/BeDH7sZ+HXb+byxBzdXL
rbE4CTOSGVjssEX/sxj58KsOMaiA+h3+DFLA2qqbYD/+gCwK+9E+rBAU1zeafWvSxHrPzr
rvFz01u8CdqilXFYeCRDsj3xkuQPlRimHrX6w/X/qvZ7pSQTr+apIU3O+V81FTvSbNAAAA
wQCtjd+HU8pwwXlNH3gEGX+YIQEgmRwgRXQm3mONEGsY/2udPfqtuPzNpTcqcStc+hhYlA
ndFQLKIPJktNOIrc8D2d/VdvXUX8YGiCDQNWOOCmXcMK3frp8Q7NWFWGDCBUZictbpusDe
5ulROhtD2T5T4FC9inJLK1cfIqV/Gu36eoH6aBKRzTsub1rM0VBFqo7lVgKytK2Q60MnR8
TZe1au5gTV17oULelmYVj+mGiaojc9lPt5MtaUZlQFjbVxpaQAAADBALnFkNIrMbnhHk+J
J66VlKvr2Bm+ihzwgbOxYDYKqPL+rasLWuiY0xmEbmIgV89ZXRaOwYJWoMpaVuTue7+swx
viF97lOBAAPpmk8MZcBF31n8Pk8LTlOuzH2PR/t5t2hJsHazM/+nFAto2mccvnMkpUAg8X
WYltoLPf+fNqcpshx5TEMYLR58PQ4xVHJZQmY3Ysqy1si6kRuvYfArgk3VKadtDbXORk0g
Hq6Hl748Z/cd+w8C0j+v+ynPK0+7OBtQAAAMEA5bHeFUQtvRTOBCUMwV6jZ7alKj9+mZDR
eMQCkvSQTbDLSVgeL6GPg69xZfrfnea/tSKnTqibYH43FXKVaX0ePsNjmzZLAEPx2AjWNh
uHnmqem9tuYG/bldVj+GcqeAanyRPY9PvWY27jG1nq8kTMG03NZLoXKxKG5WToc6aMDdzA
R13A9j3abQ/gp+n/KCWpGGwm8845WdWNzgc4QCvnEot1H53STHXjHpkaGnLM9ERW3nHuxg
1syV87ktSaXtfnAAAADXdjb3N0YUBmZWRvcmEBAgMEBQ==
-----END OPENSSH PRIVATE KEY-----"""

ssh = SSHClient(host='localhost',
                user='root',
                pkey=pkey,
                port=2022,
                num_retries=1,
                identity_auth=False)

Expected behavior

A connection to the remote host is established.

Actual behavior

Traceback (most recent call last):
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 204, in _auth_retry
    self.auth()
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 349, in auth
    return self._pkey_auth(self.pkey)
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 389, in _pkey_auth
    return self._pkey_from_memory(_pkey)
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/native/single.py", line 255, in _pkey_from_memory
    self.session.userauth_publickey_frommemory(
  File "ssh2/session.pyx", line 293, in ssh2.session.Session.userauth_publickey_frommemory
  File "ssh2/utils.pyx", line 166, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wcosta/test.py", line 47, in <module>
    ssh = SSHClient(host='localhost',
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/native/single.py", line 138, in __init__
    super(SSHClient, self).__init__(
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 188, in __init__
    self._init()
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 196, in _init
    self._auth_retry()
  File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 210, in _auth_retry
    raise AuthenticationError(msg, self.host, self.port, ex, retries, self.num_retries)
pssh.exceptions.AuthenticationError: ('Authentication error while connecting to %s:%s - %s - retries %s/%s', 'localhost', 2022, AuthenticationError(), 1, 1)

Additional information

parallel-ssh in ./.local/lib/python3.10/site-packages (2.12.0)
ssh-python in /usr/lib64/python3.10/site-packages (from parallel-ssh) (0+unknown)
gevent>=1.3.0 in ./.local/lib/python3.10/site-packages (from parallel-ssh) (21.12.0)
ssh2-python in ./.local/lib/python3.10/site-packages (from parallel-ssh) (1.0.0)
zope.event in ./.local/lib/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (4.5.0)
zope.interface in ./.local/lib/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (5.4.0)
setuptools in /usr/lib/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (59.6.0)
greenlet<2.0,>=1.1.0 in /usr/lib64/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (1.1.2)

The text was updated successfully, but these errors were encountered:

V346Cen · 2022-11-02T14:09:45Z

@walac what is the remote SSH server you're trying to connect to, and what version? I've seen a problem that's very similar to yours. In my testing with parallel-ssh-2.12.0 I had no trouble connecting to OpenSSH_8.5p1, but I couldn't connect to OpenSSH_8.8p1 server.

Here's my example code and the result when trying to connect with OpenSSH_8.8p1: https://gist.github.com/V346Cen/367f9fc20cfa471567ac6f71f5efa51b. The key I used for testing was SSH-RSA, ~~but I got the same results with ED25519 key~~. Unfortunately the debug prints don't provide much information about the problem.

I was able to "fix" my example by changing the import statement in line 4:

-from pssh.clients import ParallelSSHClient
+from pssh.clients.ssh.parallel import ParallelSSHClient

My understanding is that this changes the library internally used by parallel-ssh from ssh2-python to ssh-python. After this change my example works with both remote OpenSSH versions.

I suppose the problem could be somewhere further down parallel-ssh dependency chain. OpenSSH has seen some breaking changes in recent versions, they dropped some cryptographic routines considered unsafe nowadays. So maybe an adaptation is required somewhere?

Edit: Please disregard the part about ED25519 key, it was an error in my testing.

walac · 2022-11-07T21:07:29Z

@V346Cen my SSH server is OpenSSH_8.8p1, OpenSSL 3.0.5 5 Jul 2022, so I guess I am having the same problem of yours.

pkittenis · 2022-11-22T10:41:56Z

Hi,

This sounds like a compatibility issue with OpenSSH 8.8 and libssh2.

Please try the clients under from pssh.clients.ssh import ParallelSSHClient, SSHClient - likely they will work given the above.

To resolve the issue will need a newer libssh2 and ssh2-python - leaving this open to track.

V346Cen · 2023-02-20T10:52:34Z

I had to investigate this further, so here's the outcome. Maybe it will be useful to someone who finds this thread before the problem is fixed.

The problem is related to ssh-rsa public key signature algorithm. It is based on SHA-1 hashing function and is considered insecure. Starting from OpenSSH 8.8 this algorithm is disabled by default in SSH server. [1]

If SSH client connecting to such server is unable to use newer public key signature algorithms based on SHA-2, then it will not be able to connect using ssh-rsa keys. Apparently ssh2-python is affected, while ssh-python is not.

There are three possible workarounds at this point:

Use ssh-python backend.
Use a different type of SSH keys instead of ssh-rsa (e.g. ed25519).
Explicitly re-enable ssh-rsa public key signature algorithm on SSH server side (not recommended).

[1] https://www.openssh.com/txt/release-8.8

rmarops · 2023-06-14T20:59:42Z

Option 3 does not work me against Amazon Linux 2023 and pssh using a SHA-256 rsa key. Same key used outside of pssh from command line works fine. For the record, I tried PubkeyAcceptedAlgorithms +ssh-rsa , PubkeyAcceptedKeyTypes +ssh-rsa, HostKeyAlgorithms +ssh-rsa while trying to get it to work for the short term.
Debug always shows: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]. Seems hard disabled.

sjackson-adaptiv · 2023-10-03T15:21:29Z

Please try the clients under from pssh.clients.ssh import ParallelSSHClient, SSHClient - likely they will work given the above.

I want to add that this worked for me, thanks for the suggestion!

ParallelSSH/parallel-ssh#363

max-l · 2024-01-24T21:06:15Z

I have the same problem, with both "rsa" and "ed25519" keys, see the code below.

Both of the keys DO work on the same server (and same account), with the ssh terminal client.

If I log in from the terminal with the verbose flag (-v), I get the following server side info:

debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: compat_banner: match: OpenSSH_7.4 pat OpenSSH_7.4* compat 0x04000006

Interestingly the same test code below DOES work with another server, that has the following ssh server side info:

debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000

import os
from pssh.clients import SSHClient


def test_pssh():

    host = " ... my host ..."

    pkey = os.path.expanduser("~/.ssh/id_ed25519")

    ssh_client = SSHClient(host, "my-username", pkey=pkey)

    cmd = "ls -al"

    host_out = ssh_client.run_command(cmd)
    stdout_txt = "\n".join(list(host_out.stdout))
    stderr_lines = list(host_out.stderr)


    if host_out.exit_code != 0:
        stderr_txt = "\n".join(stderr_lines)
        raise Exception(f"remote call failed {host_out.exit_code}, '{cmd}'\n{stderr_txt}\n")

    print(stdout_txt)

Traceback (most recent call last):
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 204, in _auth_retry
    self.auth()
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 349, in auth
    return self._pkey_auth(self.pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 389, in _pkey_auth
    return self._pkey_from_memory(_pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/native/single.py", line 255, in _pkey_from_memory
    self.session.userauth_publickey_frommemory(
  File "ssh2/session.pyx", line 293, in ssh2.session.Session.userauth_publickey_frommemory
  File "ssh2/utils.pyx", line 168, in ssh2.utils.handle_error_codes
ssh2.exceptions.PublickeyUnverifiedError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 204, in _auth_retry
    self.auth()
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 349, in auth
    return self._pkey_auth(self.pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 389, in _pkey_auth
    return self._pkey_from_memory(_pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/native/single.py", line 255, in _pkey_from_memory
    self.session.userauth_publickey_frommemory(
  File "ssh2/session.pyx", line 293, in ssh2.session.Session.userauth_publickey_frommemory
  File "ssh2/utils.pyx", line 166, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

pkittenis added pending reproduction blocked by upstream and removed pending reproduction labels Nov 22, 2022

egede added a commit to egede/spotmanager that referenced this issue Nov 27, 2023

Change parallel ssh import following

2b52a6c

ParallelSSH/parallel-ssh#363

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot authenticate with a private key #363

Cannot authenticate with a private key #363

walac commented Oct 4, 2022

V346Cen commented Nov 2, 2022 •

edited

walac commented Nov 7, 2022

pkittenis commented Nov 22, 2022 •

edited

V346Cen commented Feb 20, 2023 •

edited

rmarops commented Jun 14, 2023

sjackson-adaptiv commented Oct 3, 2023

max-l commented Jan 24, 2024

Cannot authenticate with a private key #363

Cannot authenticate with a private key #363

Comments

walac commented Oct 4, 2022

V346Cen commented Nov 2, 2022 • edited

walac commented Nov 7, 2022

pkittenis commented Nov 22, 2022 • edited

V346Cen commented Feb 20, 2023 • edited

rmarops commented Jun 14, 2023

sjackson-adaptiv commented Oct 3, 2023

max-l commented Jan 24, 2024

V346Cen commented Nov 2, 2022 •

edited

pkittenis commented Nov 22, 2022 •

edited

V346Cen commented Feb 20, 2023 •

edited