Can not connect to SFTP server requiring password and publickey authentication

[pierrrrrr]

Hello

I am trying to connect to an SFTP site that requires both password and publickey authentication.
Most likely, the SFTP site is not an OpenSSH server.

Unfortunately, Paramiko is not able to authenticate despite correct credentials.
No exception is raised during auth_publickey or auth_password so I am assuming
the server responds with partial success.

OpenSSH's SFTP client can connect using the same credentials.

Is there any way I could debug this further?

Pierrr

Code:

transport = Transport((HOSTNAME, PORT))
transport.connect()

transport.set_hexdump(True)  # DEBUG

transport.auth_publickey(USERNAME, key)
transport.auth_password(USERNAME, PASSWORD)

transport.auth_publickey(USERNAME, key)  # DEBUG
transport.auth_password(USERNAME, PASSWORD)

sftp = SFTPClient.from_transport(transport)

Logs:

starting thread (client mode): 0x2e9efc50
Connected (version 2.0, client SFTP)
kex algos:['diffie-hellman-group1-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group-exchange-sha256'] server key:['ssh-rsa'] client encrypt:['blowfish-cbc', '3des-ctr', 'aes128-ctr', '3des-cbc', 'aes128-cbc', 'arcfour', 'arcfour128', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish192-cbc', 'twofish128-cbc', 'cast128-cbc'] server encrypt:['blowfish-cbc', '3des-ctr', 'aes128-ctr', '3des-cbc', 'aes128-cbc', 'arcfour', 'arcfour128', 'aes192-ctr', 'aes256-ctr', 'aes192-cbc', 'aes256-cbc', 'twofish256-cbc', 'twofish192-cbc', 'twofish128-cbc', 'cast128-cbc'] client mac:['hmac-sha1', 'hmac-md5', 'hmac-md5-96', 'hmac-sha1-96', 'hmac-sha256', 'hmac-sha2-256', 'hmac-sha256@ssh.com'] server mac:['hmac-sha1', 'hmac-md5', 'hmac-md5-96', 'hmac-sha1-96', 'hmac-sha256', 'hmac-sha2-256', 'hmac-sha256@ssh.com'] client compress:['none', 'zlib'] server compress:['none', 'zlib'] client lang:[''] server lang:[''] kex follows?False
Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
using kex diffie-hellman-group14-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
Switch to new keys ...
Write packet <service-request>, length 17
Got payload (28 bytes, 10 padding)
Read packet <service-accept>, length 17
userauth is OK
Write packet <userauth-request>, length 617
Got payload (3020 bytes, 7 padding)
Read packet <userauth--banner>, length 3012
Auth banner: b'****************************************************************************\n*                                                                          
Got payload (28 bytes, 13 padding)
Read packet <userauth-failure>, length 14
Authentication continues...
Methods: ['password']
Write packet <service-request>, length 17
Got payload (28 bytes, 10 padding)
Read packet <service-accept>, length 17
userauth is OK
Write packet <userauth-request>, length 60
Got payload (3020 bytes, 7 padding)
Read packet <userauth--banner>, length 3012
Got payload (28 bytes, 12 padding)
Read packet <userauth-failure>, length 15
Authentication continues...
Methods: ['publickey']
Write packet <service-request>, length 17
Got payload (28 bytes, 10 padding)
Read packet <service-accept>, length 17
userauth is OK
Write packet <userauth-request>, length 617Got payload (3020 bytes, 7 padding)
Read packet <userauth--banner>, length 3012
Auth banner: b'****************************************************************************\n*                                                                
Got payload (28 bytes, 13 padding)
Read packet <userauth-failure>, length 14
Authentication continues...
Methods: ['password']
Write packet <service-request>, length 17
Got payload (28 bytes, 10 padding)
Read packet <service-accept>, length 17
userauth is OK
Write packet <userauth-request>, length 60
Got payload (3020 bytes, 7 padding)
Read packet <userauth--banner>, length 3012
Auth banner: b'****************************************************************************\n*                                                                           
Got payload (28 bytes, 12 padding)
Read packet <userauth-failure>, length 15
Authentication continues...
Methods: ['publickey']
[chan 0] Max packet in: 32768 bytes
Write packet <channel-open>, length 24
Got payload (12 bytes, 6 padding)
Read packet <unimplemented>, length 5
Oops, unhandled type 3
Write packet <unimplemented>, length

[rustyscottweber]

last time I checked.. every time an auth function is called, it bull dozes the previous auth member of the transport. I have suggested that we not do this in a different issue all together. Pierrrrrr, can you investigate this further by stepping though with the debugger in the auth functions to verify that is the case?

[pierrrrrr]

@rustyscottweber sure, not problem - what objects|attributes|lines would you like me to take a look at?

[teamdoug]

I ran into the same issue and it seems caused by the service-request on the second auth reinitializing the authentication flow. If I made the auth handler directly send the second userauth-request without sending the service-request, this worked.

Unfortunately, this appears non-trivial to fix given how Transport and AuthHandler interact right now. I think you'd need to persist self.auth_handler on the Transport between auth_* calls and have some way to note that the service-request already happened and does not need to be repeated, so you can go straight to the userauth-request.

Here's my really janky code that works:

    transport = paramiko.Transport((SFTP_HOST, SFTP_PORT))
    transport.connect()
    transport.auth_publickey(username=SFTP_USER, key=private_key)
    event = threading.Event()
    auth_handler = paramiko.auth_handler.AuthHandler(transport)
    transport.auth_handler = auth_handler
    transport.lock.acquire()
    try:
        auth_handler.auth_event = event
        auth_handler.auth_method = 'password'
        auth_handler.username = SFTP_USER
        auth_handler.password = password
        message = paramiko.Message().add_string('ssh-userauth')
        message.rewind()
        auth_handler._parse_service_accept(message)
    finally:
        transport.lock.release()
    auth_handler.wait_for_response(event)

I can try to put a pull request together if that would be helpful, but I don't have a good understanding of how this all actually works (especially in error cases).

[rustyscottweber]

@pierrrrrr appologies for the delay in reply, I was referring to the auth_* methods in the fashion that the auth_handler object is over written every time that a new auth method has be called. This makes authenticating against a server which requires more than one type of authentication... I would imagine difficult or impossible.
@teamdoug The origional problem that I was running into with the auth_object being overwritten was obtaining the banner since the first auth object would read the banner in, but then subsequent auth objects would not because it was already sent over once.. This becomes a problem in the case where password authentication is not allowed because after attempting password authentication, paramiko will automatically fail over to keyboard authentication.. However if this happens, the banner has been thrown away with the password auth object. My suggestion then was to see if there was a way to make the auth object persist, just like you. However, also just like you, I was also unaware as to how or if this would cause serious issues especially if there were errors detected in the authorization.

[alexey-stankevich]

Hello guys,

I ran into the same issue and it seems caused by the service-request on the second auth reinitializing > the authentication flow. If I made the auth handler directly send the second userauth-request without > sending the service-request, this worked.

Any news about this issue? Are there plans to fix it?

[Symmetric]

@rustyscottweber I'm hitting this too. I just fired up the debugger to get the diags that you were after, and I'm seeing (on paramiko v1.16.0) that in both auth_publickey and auth_password that the code blats the self.auth_handler (e.g. https://github.com/paramiko/paramiko/blob/1.16/paramiko/transport.py#L1323).

In both cases I see a response like ['keyboard-interactive', 'publickey'] for auth_password, i.e. the other allowed methods for the connection.

Any suggestions for further diags I can help with? Is this just as simple as not overwriting the self.auth_handler, or are there more complex concerns here?

[rustyscottweber]

I suppose you could try not obliterating the auth handler, but I'm not sure
that the same handler would handle the multiple types. Worth a quick test.
On Feb 26, 2016 5:31 PM, "Paul Tiplady" notifications@github.com wrote:

@rustyscottweber https://github.com/rustyscottweber I'm hitting this
too. I just fired up the debugger to get the diags that you were after, and
I'm seeing (on paramiko v1.16.0) that in both auth_publickey and
auth_password that the code blats the self.auth_handler (e.g.
https://github.com/paramiko/paramiko/blob/1.16/paramiko/transport.py#L1323
).

In both cases I see a response like ['keyboard-interactive', 'publickey']
for auth_password, i.e. the other allowed methods for the connection.

Any suggestions for further diags I can help with? Is this just as simple
as not overwriting the self.auth_handler, or are there more complex
concerns here?

—
Reply to this email directly or view it on GitHub
#519 (comment).

[repayjason]

Would love to see a fix for this. Does anyone have any ideas what might be causing this?

[bitprophet]

Skimming this, it feels extremely related to #387, aka "the auth module is incredibly old and crusty and poorly architected for nontrivial use cases". Basically needs a modern rewrite with an eye for making non-base cases easier to handle and/or for users to implement arbitrary subclasses/workarounds.

That ticket is pretty high on my list FWIW.

[danielbrownridge]

@teamdoug Thank you very much for your work around in the earlier comment. I've converted this into a helper function that returns you a SFTP client. Including here if anyone finds it useful.

def multifactor_auth_sftp_client(host, username, keyfile, passphrase, password):
    """
    Return an open paramiko SFTP client to a host that requires multifactor
    authentication.
    """

    paramiko_logger = getLogger('paramiko')
    paramiko_logger.setLevel(DEBUG)

    handler = StreamHandler()
    handler.setLevel(DEBUG)
    handler.setFormatter(Formatter(LOG_FORMAT))

    paramiko_logger.addHandler(handler)

    logger.debug("Create private key object from file")

    key = None
    try:
        key = RSAKey.from_private_key(keyfile, password=passphrase)
    except PasswordRequiredException as exception:
        logger.critical(exception)

    logger.debug("Create an SSH transport configured to the host")
    transport = Transport(host)

    logger.debug("Negotiate an SSH2 session")
    transport.connect()

    logger.debug("Attempt authenticating using a private key")
    transport.auth_publickey(username, key)

    logger.debug("Create an event for password auth")
    password_auth_event = Event()

    logger.debug("Create password auth handler from transport")
    password_auth_handler = AuthHandler(transport)

    logger.debug("Set transport auth_handler to password handler")
    transport.auth_handler = password_auth_handler

    logger.debug("Aquire lock on transport")
    transport.lock.acquire()

    logger.debug("Register the password auth event with handler")
    password_auth_handler.auth_event = password_auth_event

    logger.debug("Set the auth handler method to 'password'")
    password_auth_handler.auth_method = 'password'

    logger.debug("Set auth handler username")
    password_auth_handler.username = username

    logger.debug("Set auth handler password")
    password_auth_handler.password = password

    logger.debug("Create an SSH userauth message")
    userauth_message = Message()
    userauth_message.add_string('ssh-userauth')
    userauth_message.rewind()

    logger.debug("Make the password auth attempt")
    password_auth_handler._parse_service_accept(userauth_message)

    logger.debug("Release lock on transport")
    transport.lock.release()

    logger.debug("Wait for passwort auth response")
    password_auth_handler.wait_for_response(password_auth_event)

    logger.debug("Create an open SFTP client channel")
    sftp = transport.open_sftp_client()

    return sftp

[matt-pazien]

Would love to see a fix for this too. 👍

[rorykoehler]

What's the latest on this?

[tleemans]

We ran into this issue and fixed it in a derived class, running fine at the moment. The issue, as stated above, is that paramiko sends a MSG_SERVICE_REQUEST message every time. So, we added a variable that indicates if the MSG_SERVICE_REQUEST has already been sent. When it is, the code will just continue with a new MSG_USERAUTH_REQUEST.
I added a pull request that should fix this: #1121

[ghost]

Anyone else struggling with this, @teamdoug has posted the workaround solution above and it's confirmed working for me.

There has been lot of chatter across multiple tickets and posts about needing Paramiko to handle this and pending PR's to fix it, but his solution is the only one that is a straightforward true solution that can be used today.

Thank you @teamdoug I can now use Paramiko to connect to my client's sftp server that uses both key and unix user creds. Would have never figured this out on my own.

[ezrast]

I've pushed my own fix for this issue at ezrast@67521f6. @bitprophet, is a rewrite of the auth system still pending? If you'd like a PR let me know; otherwise I'll assume you're still planning on fixing this internally.