New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bcrypt TypeError "Unicode-objects must be encoded before hashing" #1039

Closed
tvannahl opened this Issue Aug 15, 2017 · 11 comments

Comments

Projects
None yet
3 participants
@tvannahl

tvannahl commented Aug 15, 2017

I am using Python 3.6.2 and paramiko 2.2.1. Since I am not using the ed25519 private key explicitly in my code I assume this is a paramiko bug.

  File "/home/tvn/projects/sowiwas/web/_venv3/lib/python3.6/site-packages/paramiko/client.py", line 394, in connect                                              
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)                                                                                                 
  File "/home/tvn/projects/sowiwas/web/_venv3/lib/python3.6/site-packages/paramiko/client.py", line 617, in _auth                                                
    key = pkey_class.from_private_key_file(filename, password)                                                                                                   
  File "/home/tvn/projects/sowiwas/web/_venv3/lib/python3.6/site-packages/paramiko/pkey.py", line 205, in from_private_key_file                                  
    key = cls(filename=filename, password=password)                                                                                                              
  File "/home/tvn/projects/sowiwas/web/_venv3/lib/python3.6/site-packages/paramiko/ed25519key.py", line 59, in __init__                                          
    signing_key = self._parse_signing_key_data(data, password)                                                                                           
  File "/home/tvn/projects/sowiwas/web/_venv3/lib/python3.6/site-packages/paramiko/ed25519key.py", line 122, in _parse_signing_key_data                                                           
    ignore_few_rounds=True,                                                                                                                                                                       
  File "/home/tvn/projects/sowiwas/web/_venv3/lib/python3.6/site-packages/bcrypt/__init__.py", line 118, in kdf                                                                                   
    raise TypeError("Unicode-objects must be encoded before hashing")                                                                                                                             
TypeError: Unicode-objects must be encoded before hashing                                                    
@ploxiln

This comment has been minimized.

Contributor

ploxiln commented Aug 15, 2017

Is your key file in the "new openssh format" (the "-o" option to ssh-keygen) by any chance?

@tvannahl

This comment has been minimized.

tvannahl commented Aug 21, 2017

I do have an ed25519 key which is according to ssh-keygen(1) always in the new format.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Aug 21, 2017

@tvannahl - you said:

I am not using the ed25519 private key explicitly

Can you clarify this? What are you explicitly using for auth in this situation, if anything? (I.e. what args are you giving to connect, are you allowing the SSH agent to operate and if so what's in it, what default-named key files are in your ~/.ssh/ and what format are those, etc etc.)

I ask because this may be an issue with how Paramiko currently tries all key types in an arbitrary order for any given key file it's been asked to load. (Normally that doesn't explode in this particular way, but that's why I'm asking, to see if we can figure out exactly what key material is triggering it.)

EDIT: if you enable debug-level stdlib logging and supply that output, that will also help as it will give at least some info about which keys are being tried when.

@tvannahl

This comment has been minimized.

tvannahl commented Sep 1, 2017

Sorry I try to clearify.

The enviroment

The client does have an rsa and ed25519 keypair and has the server registered within known_hosts. The client is only able to connect to the server using a password, so no keypair will work for authentication.

The minimal code

I have written the following minimal example where I have been able to replicate the problem:

#!/usr/bin/env python3                                                 
import paramiko                                                        
import logging                                                         
import sys                                                             
                                                                       
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)            
                                                                       
server, user, pw = "server.example.org", "tvannahl", "mypassword"      
client = paramiko.client.SSHClient()                                   
client.load_system_host_keys()                                         
client.set_missing_host_key_policy(paramiko.client.AutoAddPolicy())    
client.connect(hostname=server, username=user, password=pw)            

The result

The code produces the following output:

DEBUG:paramiko.transport:starting thread (client mode): 0x…
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.2.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_6.6.1
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_6.6.1)
DEBUG:paramiko.transport:kex algos:['curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: ecdh-sha2-nistp256
DEBUG:paramiko.transport:HostKey agreed: ecdsa-sha2-nistp256
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha2-256
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Trying SSH agent key b'f…'
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (publickey) failed.
Traceback (most recent call last):
  File "./min.py", line 12, in <module>
    client.connect(hostname=server, username=user, password=pw)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/client.py", line 394, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/client.py", line 617, in _auth
    key = pkey_class.from_private_key_file(filename, password)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/pkey.py", line 205, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/ed25519key.py", line 59, in __init__
    signing_key = self._parse_signing_key_data(data, password)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/ed25519key.py", line 122, in _parse_signing_key_data
    ignore_few_rounds=True,
  File "/tmp/_venv3/lib/python3.6/site-packages/bcrypt/__init__.py", line 118, in kdf
    raise TypeError("Unicode-objects must be encoded before hashing")
TypeError: Unicode-objects must be encoded before hashing
@ploxiln

This comment has been minimized.

Contributor

ploxiln commented Sep 1, 2017

That's interesting - it looks like it's using a key from the ssh agent, but then it tries to load a private key from a file ...

@ploxiln

This comment has been minimized.

Contributor

ploxiln commented Sep 1, 2017

I was a little surprised that it said "userauth is OK", but then must have thrown an exception, for it to continue on to try the standard auth-key filenames.

I think that a temporary work-around for you is pw = b"mypassword" (make password be bytes instead of a normal python3 string). For other key types, the password is made bytes-type in paramiko.util.generate_key_bytes(), so it seems paramiko.Ed25519Key._parse_signing_key_data() should probably do the same.

@bitprophet

This comment has been minimized.

Member

bitprophet commented Sep 2, 2017

IIRC userauth is OK just means "the server says it accepts publickey-style auth so yes we can actually try to submit this key we've got"; it's got no bearing on whether the key we're about to try will actually work!

It confuses people a lot (self included) so I'd like to change it sometime. Whether given log messages should be considered backwards compatible or not is something I'm torn on though, heh.


@tvannahl One of my questions is simply "which key are you intending to use for this server" :) It seems clear by now that this ed25519 key is triggering the bug, and is probably coming from your SSH agent (this can be disabled, by the way, with I think connect(..., allow_agent=False, ...) or similar). And that's probably what this ticket should be "about". But I also don't know if your real problem is that you're trying to use that other RSA key you mention! :)


back to @ploxiln - it seems like this could be the password or the salt, though I think you're probably right re: password type consistency.

ploxiln added a commit to ploxiln/paramiko that referenced this issue Sep 4, 2017

@tvannahl

This comment has been minimized.

tvannahl commented Sep 5, 2017

@bitprophet I am not intending to use any key on this server. I am intending to use password authentication because I do not have any of my public keys on that server.

It does not seem to be the agent interfering, I did try out connect(…, allow_agent=False) and I did disconnect the SSH Agent by unsetting the associated enviroment variables. I got the following debug output:

DEBUG:paramiko.transport:starting thread (client mode): 0x…
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.2.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_6.6.1
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_6.6.1)
DEBUG:paramiko.transport:kex algos:['curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: ecdh-sha2-nistp256
DEBUG:paramiko.transport:HostKey agreed: ecdsa-sha2-nistp256
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha2-256
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
Traceback (most recent call last):
  File "./foo.py", line 12, in <module>
    client.connect(hostname=server, username=user, password=pw, allow_agent=False)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/client.py", line 394, in connect
    look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/client.py", line 617, in _auth
    key = pkey_class.from_private_key_file(filename, password)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/pkey.py", line 205, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/ed25519key.py", line 59, in __init__
    signing_key = self._parse_signing_key_data(data, password)
  File "/tmp/_venv3/lib/python3.6/site-packages/paramiko/ed25519key.py", line 122, in _parse_signing_key_data
    ignore_few_rounds=True,
  File "/tmp/_venv3/lib/python3.6/site-packages/bcrypt/__init__.py", line 118, in kdf
    raise TypeError("Unicode-objects must be encoded before hashing")
TypeError: Unicode-objects must be encoded before hashing

But after I did change the password encoding to a byte string, like @ploxiln suggested, the authentication did work. I must admit that I am a little bit puzzled by this because of the stacktrace in the former try:

DEBUG:paramiko.transport:starting thread (client mode): 0x34541b70
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.2.1
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_6.6.1
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_6.6.1)
DEBUG:paramiko.transport:kex algos:['curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com', 'chacha20-poly1305@openssh.com', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', 'rijndael-cbc@lysator.liu.se'] client mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-ripemd160-etm@openssh.com', 'hmac-sha1-96-etm@openssh.com', 'hmac-md5-96-etm@openssh.com', 'hmac-md5', 'hmac-sha1', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-ripemd160', 'hmac-ripemd160@openssh.com', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: ecdh-sha2-nistp256
DEBUG:paramiko.transport:HostKey agreed: ecdsa-sha2-nistp256
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha2-256
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:kex engine KexNistp256 specified hash_algo <built-in function openssl_sha256>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (password) successful!
@ploxiln

This comment has been minimized.

Contributor

ploxiln commented Sep 5, 2017

We did figure out that it has nothing to do with the ssh agent - it is just because paramiko is trying to use the password to decrypt the ed25519 auth key, and instead of failing in the normal way it's getting an unexpected exception. Another work around which I think will work for you is using the look_for_keys=False kwarg for connect(). I also have a pull-request opened which should fix it for you (though I'm not totally sure if it's the best place for the fix).

@bitprophet

This comment has been minimized.

Member

bitprophet commented Sep 5, 2017

I think that PR is the right way to fix this, left a note on there. Let's roll this into the PR. Thanks!

@bitprophet bitprophet closed this Sep 5, 2017

@bitprophet

This comment has been minimized.

Member

bitprophet commented Sep 5, 2017

Oh and re: agent, I was never thinking the agent itself was teh issue, was just trying to separate "key exploding" from "why is key even being used, I don't want it". Given that this still pops up even w/ agent in play I'm assuming the key is in ~/.ssh/. Hopefully moot now.

ploxiln added a commit to ploxiln/paramiko that referenced this issue Sep 5, 2017

bitprophet added a commit that referenced this issue Nov 29, 2017

dkhapun pushed a commit to cyberx-labs/paramiko that referenced this issue Jun 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment