Client password authentication does not work on OpenSSH server versions < 7.0

[Rahul-RB]

Bug report:

I have a remote host (say as some IP ip1). I can connect to it:

• Via the Paramiko based Parallel Client (pssh.clients.miko.parallel.ParallelSSHClient).

    from pssh.clients.miko.parallel import ParallelSSHClient
    hostConfig = {
                            "ip1":{
                                       "user":"....",
                                       "password":"...."
                            },
                            "ip2:{
                                       "user":"....",
                                       "password":"...."}
    }
    cl = ParallelSSHClient(list(hostConfig.keys()),host_config=hostConfig)
    cl.run_command("ls")
    # I get correct output.

• Via the regular Paramiko SSHClient also.
• Via the linux ssh command.

But using the Native parallel client is giving errors

Steps to reproduce: (Same steps as for Paramiko parallel client code above, just replace the import)

    from pssh.clients.native.parallel import ParallelSSHClient
    hostConfig = {
                            "ip1":{
                                       "user":"....",
                                       "password":"...."
                            },
                            "ip2:{
                                       "user":"....",
                                       "password":"...."}
    }
    cl = ParallelSSHClient(list(hostConfig.keys()),host_config=hostConfig)
    cl.run_command("ls")
    # Waits a few minutes, then gives auth errors.

Expected behaviour: Give the outputs of commands.

Actual behaviour:

Gives a whole bunch of authentication errors:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 277, in auth
    self._identity_auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 259, in _identity_auth
    raise AuthenticationException("No authentication methods succeeded")
pssh.exceptions.AuthenticationException: No authentication methods succeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 286, in _password_auth
    self.session.userauth_password(self.user, self.password)
  File "ssh2/session.pyx", line 266, in ssh2.session.Session.userauth_password
  File "ssh2/utils.pyx", line 131, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 192, in _init
    self.auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 282, in auth
    self._password_auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 288, in _password_auth
    raise AuthenticationException("Password authentication failed")
pssh.exceptions.AuthenticationException: Password authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 277, in auth
    self._identity_auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 259, in _identity_auth
    raise AuthenticationException("No authentication methods succeeded")
pssh.exceptions.AuthenticationException: No authentication methods succeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 286, in _password_auth
    self.session.userauth_password(self.user, self.password)
  File "ssh2/session.pyx", line 266, in ssh2.session.Session.userauth_password
  File "ssh2/utils.pyx", line 131, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 192, in _init
    self.auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 282, in auth
    self._password_auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 288, in _password_auth
    raise AuthenticationException("Password authentication failed")
pssh.exceptions.AuthenticationException: Password authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 277, in auth
    self._identity_auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 259, in _identity_auth
    raise AuthenticationException("No authentication methods succeeded")
pssh.exceptions.AuthenticationException: No authentication methods succeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 286, in _password_auth
    self.session.userauth_password(self.user, self.password)
  File "ssh2/session.pyx", line 266, in ssh2.session.Session.userauth_password
  File "ssh2/utils.pyx", line 131, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 192, in _init
    self.auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 282, in auth
    self._password_auth()
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 288, in _password_auth
    raise AuthenticationException("Password authentication failed")
pssh.exceptions.AuthenticationException: Password authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/parallel.py", line 223, in run_command
    greenlet_timeout=greenlet_timeout)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/base_pssh.py", line 96, in run_command
    self.get_output(cmd, output, timeout=greenlet_timeout)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/base_pssh.py", line 141, in get_output
    (channel, host, stdout, stderr, stdin) = cmd.get(timeout=timeout)
  File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/parallel.py", line 237, in _run_command
    raise ex
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/parallel.py", line 230, in _run_command
    self._make_ssh_client(host)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/parallel.py", line 387, in _make_ssh_client
    keepalive_seconds=self.keepalive_seconds)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 132, in __init__
    THREAD_POOL.apply(self._init)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/gevent/pool.py", line 159, in apply
    return self.spawn(func, *args, **kwds).get()
  File "src/gevent/event.py", line 268, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 296, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 286, in gevent._event.AsyncResult.get
  File "src/gevent/event.py", line 266, in gevent._event.AsyncResult._raise_exception
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise
    raise value.with_traceback(tb)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/gevent/threadpool.py", line 281, in _worker
    value = func(*args, **kwargs)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 195, in _init
    return self._connect_init_retry(retries)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 174, in _connect_init_retry
    return self._init(retries=retries)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 195, in _init
    return self._connect_init_retry(retries)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 174, in _connect_init_retry
    return self._init(retries=retries)
  File "/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/pssh/clients/native/single.py", line 197, in _init
    raise AuthenticationException(msg, self.host, self.port, ex)
pssh.exceptions.AuthenticationException: ('Authentication error while connecting to %s:%s - %s', '<ip1>', 22, AuthenticationException('Password authentication failed',))

Additional info:

• pssh version 1.9.1
• ssh2_python version 0.17.0
• paramiko version 2.4.2

[pkittenis]

Hi there,

Thanks for the interest and report. Let's not mix in #141 - the OP has not responded and there is no guarantee it is same issue.

It is not uncommon for the two clients to have different behaviour - they use different libraries. If paramiko works for you as a work around then great, but it is deprecated and pending removal. I don't think this is related to password auth, or python version, hence bug label removal.

It is more likely the server is using a host key the library does not support. Should be able to confirm by looking at server logs when the native client connects. If you can also post server config to reproduce this with, that would be useful. Without being able to reproduce there is nothing to be done.

[Rahul-RB]

Hi!
Thanks for the reply.

Can you please tell me what server configs will be helpful?

[pkittenis]

Contents of /etc/ssh/sshd_config please. Also would be good to check if public key auth works.

[Rahul-RB]

• Please find attached the sshd_config of the host facing issues. (I can confirm that both the hosts (ip1 and ip2 in my issue) facing issues have same config).

• Also, public key auth works as intended. (I followed these steps to check that, please do mention if that was the right check to do).

sshd_config.txt

[pkittenis]

Unable to replicate so far - both password=<..> and host_config with password entries work as expected. Will try with the above server cfg - it does not seem to use unsupported server keys at first glance.

Might be worth seeing if there is anything logged by the SSH server on the unsuccessful attempts. Usually this is in /var/log/auth.log.

[spctm]

I can confirm the same issue when connecting to ESXi hosts. None of my Ubuntu hosts have problems connecting to when using native client. Its easy to repro if you have any VMware ESXi 6.0 hosts. While using paramiko client works fine. The problem is I would like to use the native client exclusively since the paramiko client will be removed in the future.

~# python3 test_pssh.py
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 277, in auth
self._identity_auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 259, in _identity_auth
raise AuthenticationException("No authentication methods succeeded")
pssh.exceptions.AuthenticationException: No authentication methods succeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 286, in _password_auth
self.session.userauth_password(self.user, self.password)
File "ssh2/session.pyx", line 266, in ssh2.session.Session.userauth_password
File "ssh2/utils.pyx", line 131, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 192, in _init
self.auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 282, in auth
self._password_auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 288, in _password_auth
raise AuthenticationException("Password authentication failed")
pssh.exceptions.AuthenticationException: Password authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 277, in auth
self._identity_auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 259, in _identity_auth
raise AuthenticationException("No authentication methods succeeded")
pssh.exceptions.AuthenticationException: No authentication methods succeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 286, in _password_auth
self.session.userauth_password(self.user, self.password)
File "ssh2/session.pyx", line 266, in ssh2.session.Session.userauth_password
File "ssh2/utils.pyx", line 131, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 192, in _init
self.auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 282, in auth
self._password_auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 288, in _password_auth
raise AuthenticationException("Password authentication failed")
pssh.exceptions.AuthenticationException: Password authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 277, in auth
self._identity_auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 259, in _identity_auth
raise AuthenticationException("No authentication methods succeeded")
pssh.exceptions.AuthenticationException: No authentication methods succeeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 286, in _password_auth
self.session.userauth_password(self.user, self.password)
File "ssh2/session.pyx", line 266, in ssh2.session.Session.userauth_password
File "ssh2/utils.pyx", line 131, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 192, in _init
self.auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 282, in auth
self._password_auth()
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 288, in _password_auth
raise AuthenticationException("Password authentication failed")
pssh.exceptions.AuthenticationException: Password authentication failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test_pssh.py", line 14, in
output = client.run_command('uname -a')
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/parallel.py", line 223, in run_command
greenlet_timeout=greenlet_timeout)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/base_pssh.py", line 96, in run_command
self.get_output(cmd, output, timeout=greenlet_timeout)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/base_pssh.py", line 141, in get_output
(channel, host, stdout, stderr, stdin) = cmd.get(timeout=timeout)
File "src/gevent/greenlet.py", line 709, in gevent._greenlet.Greenlet.get
File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception
File "/usr/local/lib/python3.6/dist-packages/gevent/_compat.py", line 47, in reraise
raise value.with_traceback(tb)
File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/parallel.py", line 237, in _run_command
raise ex
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/parallel.py", line 230, in _run_command
self._make_ssh_client(host)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/parallel.py", line 387, in _make_ssh_client
keepalive_seconds=self.keepalive_seconds)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 132, in init
THREAD_POOL.apply(self._init)
File "/usr/local/lib/python3.6/dist-packages/gevent/pool.py", line 159, in apply
return self.spawn(func, *args, **kwds).get()
File "src/gevent/event.py", line 268, in gevent._event.AsyncResult.get
File "src/gevent/event.py", line 296, in gevent._event.AsyncResult.get
File "src/gevent/event.py", line 286, in gevent._event.AsyncResult.get
File "src/gevent/event.py", line 266, in gevent._event.AsyncResult._raise_exception
File "/usr/local/lib/python3.6/dist-packages/gevent/_compat.py", line 47, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/gevent/threadpool.py", line 281, in _worker
value = func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 195, in _init
return self._connect_init_retry(retries)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 174, in _connect_init_retry
return self._init(retries=retries)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 195, in _init
return self._connect_init_retry(retries)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 174, in _connect_init_retry
return self._init(retries=retries)
File "/usr/local/lib/python3.6/dist-packages/pssh/clients/native/single.py", line 197, in _init
raise AuthenticationException(msg, self.host, self.port, ex)
pssh.exceptions.AuthenticationException: ('Authentication error while connecting to %s:%s - %s', '10.104.32.233', 22, AuthenticationException('Password authentication failed',))

[spctm]

I haven't tried with python2 so I cannot comment on whether this problem is exclusively with python3. I debugged a little further but I can conclude that the password is being passed correctly to native client. In spite of this I get an authentication failed error, which cannot be right.

[Rahul-RB]

Hi @pkittenis , please find attached the auth.log file which has the lines only for the connection I tried via PSSH Native client., (I did tail-f and ensured the request times matched the times in log file).

Also, can confirm with @spctm , the VMs I'm connecting to are on VMWare ESXi 6.7.0.
Also, the VMs itself are running FreeBSD 8 64-bit.

Please do let me now if there's any other information required since ESXi is a licensed product.

auth.log

[Rahul-RB]

Unable to replicate so far - both password=<..> and host_config with password entries work as expected. Will try with the above server cfg - it does not seem to use unsupported server keys at first glance.

Might be worth seeing if there is anything logged by the SSH server on the unsuccessful attempts. Usually this is in /var/log/auth.log.

@pkittenis , if it's any help, the latest Paramiko Client throws these warnings when trying to connect:

/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/paramiko/ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.ecdsa_curve.curve_class(), pointinfo
/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  m.add_string(self.Q_C.public_numbers().encode_point())
/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.curve, Q_S_bytes
/home/rahulrb/.virtualenvs/test/lib/python3.6/site-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  hm.add_string(self.Q_C.public_numbers().encode_point())

I guess the two sets of warnings are repeated since I connect to two IPs at once.