Handle Paramiko-level "logger" issues

[bitprophet]

Description

This has come up at least twice now: a user sets env.user to a list instead of a string by mistake, and instead of an actual useful error, we get a funky No handlers could be found for logger "paramiko.transport" message.

Quick research implies that this is a Paramiko level bug, as we don't use the logger module ourselves and that particular logger name is used in Paramiko itself. However, if it is, it's not caught in their test suite.

Attack plan:

• Look at Paramiko's mailing list or bug tracker and see if this is a known issue
• Attempt to troubleshoot ourselves if not
• Once solved, inform upstream
• Failing all else, deal with this specific problem at our level: issue our own useful alert when user is set to a list value.

---

Originally submitted by Jeff Forcier (bitprophet) on 2009-08-14 at 09:58am EDT

Relations

• Related to Allow display of full stack trace on error #52: Allow display of full stack trace on error
• Related to Support full logging to file #57: improved logging

[bitprophet]

This is still an occasional problem in many ways, and I think it's because any time Paramiko/ssh try emitting a statement above the default visibility level, this comes up. It's not really a "bug" to be fixed at that level, I don't think.

Instead we should integrate it with #57, though likely shunting all ssh-level log traffic to its own file or even to a null handler (i.e. so it doesn't go anywhere but still doesn't raise the warning.)

[cjw296]

This is definitely worth addressing. here's an example:

"""
fab -k -H h1,h2,h3 something:aparam
[h1] Executing task 'something'
[h1] run: ls some??pattern
No handlers could be found for logger "ssh.transport"
[h1] Login password:

Fatal error: No existing session

Underlying exception:
No existing session
"""

The problem here was that I'd never ssh'ed into h1 before, and I think the message being logged is something about verifying the host key.
No idea why it then went on to fail.

I manually ssh'ed into h1, that worked fine (give or take a yes, yes, yes) and could then run the fabric command just fine.

[cjw296]

Sorry, should have said: this is with Fabric 1.4.0

[bitprophet]

@cjw296 In that case I'd actually argue ssh should be raising an exception that we could then handle, since it's a bona fide error condition. (Whereas other instances of this probably are just "warnings" we want to log/ignore.)

So maybe there's a few different ways we need to update either ourselves or ssh, re: "things that currently generate the No Handlers message".

[cjw296]

Agreed about the ssh package, where should I submit that issue?

For the fab script, a StreamHandler should be configured up front so
anything logged ends up going to the console rather than resulting in an
error.

[bitprophet]

I can make a ticket on the ssh tracker (currently it's living under my account here on Github) but I want to verify your claim first re: that it really is simply an issue with known hosts. Fabric does already have an option for loading or not loading your known hosts file, so I'm wondering now if the problem is really that simple or if you ran into something tangential.

I'll try to replicate myself.

Re: handlers: yea, we intend to implement logging for Fabric itself which would then be capable of catching other libs' log messages too. #57 IIRC.

[bitprophet]

I can't replicate this on my end, unknown hosts don't cause any logging handler errors at all. Can you verify on your end & provide the fabfile + invocation + full output?

[cjw296]

I can't verify either. This was with a new user who hadn't properly had his account setup.
That logging sure would have been helpful to have ;-)

Sadly, the user has now been correctly set up and I can no longer reproduce.
I can only speculate as to some combination of first-time-login, kerberos tickets and not-created user accounts :-S

To be clear: this issue here is the lack of a configured logger, any other issues can be raised separately if they re-occur...

[bitprophet]

Well, configured logger would be #57 :) though I notice that's not mentioned in the description here, I'll add a note.

[vipulob]

 import paramiko
 import multiprocessing


 class sshConn(object):

     def __init__(self):

         pass

    def sshconn(self):
        print("Inside sshconn")
        IPAddress = '172.17.27.39'
        user = {'username':'root','password':'root123'}
        self.ssh = paramiko.SSHClient()

        # Set the auto add policy
        self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

        self.ssh.connect(IPAddress, username=user['username'], password=user['password'])

        self.transport = self.ssh.get_transport()

        #self.chan = self.transport.open_session()

        cmd = "ls"

        #self.chan.exec_command(cmd)


   def tran(self):

        self.chan = self.transport.open_session()  # Thows error : No handlers could be found for  
                                                                  # logger "paramiko.transport"


        cmd = "ls"
        # Execute the given command
        print  self.chan.exec_command(cmd)



def inter1(ssh):
    #ssh.sshconn()
    ssh.tran()

def inter(ssh):

    inter1(ssh)


def interface():

    conn = sshConn()
    conn.sshconn()

    p = multiprocessing.Process(target=inter,args=(conn,))
    p.start()
    p.join()

interface()    

####################################################

The above program stucks and give statement as:
No handlers could be found for logger "paramiko.transport"

If i do CTRL C

File "processparamiko.py", line 83, in
Process Process-1:
interface()
File "processparamiko.py", line 79, in interface
Traceback (most recent call last):
p.join()
File "/usr/lib/python2.7/multiprocessing/process.py", line 145, in join
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
res = self._popen.wait(timeout)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 154, in wait
return self.poll(0)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 135, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(_self._args, *_self._kwargs)
File "processparamiko.py", line 69, in inter
inter1(ssh)
File "processparamiko.py", line 65, in inter1
ssh.tran()
File "processparamiko.py", line 55, in tran
self.chan = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/transport.py", line 662, in open_session
return self.open_channel('session')
File "/usr/local/lib/python2.7/dist-packages/paramiko-1.11.0-py2.7.egg/paramiko/transport.py", line 759, in open_channel
event.wait(0.1);
File "/usr/lib/python2.7/threading.py", line 620, in wait
self.__cond.wait(timeout)
File "/usr/lib/python2.7/threading.py", line 358, in wait
_sleep(delay)
KeyboardInterrupt

If i call self.chan.exec_command(cmd) in sshconn function and comment the trans function then the program works.

I am using Paramiko 1.11 on Python 2.7 on Ubuntu 13.04

[vipulob]

I am getting the same error using Fabric-1.7.0.

[svolle]

The error also happens using 1.8.0.

[manuel-rubio]

The error happens using Fabric 1.10.1 & Paramiko 1.15.2:

No handlers could be found for logger "paramiko.transport"
Fatal error: Error reading SSH protocol banner
...

Looks like this happens when is not possible to connect via SSH to the server :-P

[leafonsword]

I also have the problem -- Fabric 1.10.1 & Paramiko 1.15.1~
This happens when is not possible to connect via SSH to the remote server~

[fgimian]

Same problem here with the latest of everything, it seems to happen when the connection to the host hangs 😄

Here's how I worked around it:

import logging

logging.basicConfig()
paramiko_logger = logging.getLogger("paramiko.transport")
paramiko_logger.disabled = True

This was added to the top of my fabfile.py file.

[ananthaa-advisory]

@fgimian - I am still not able to do it and I get the same error. Can you help.

#!/usr/bin/env python
import logging
logging.basicConfig()
paramiko_logger = logging.getLogger("paramiko.transport")
paramiko_logger.disabled = True
import paramiko

from fabric.api import env, run

env.hosts = [ '10.0.2.2' ]

def uptime():
run('uptime')

[fgimian]

I think it might be your order. Here's the start of my fabfile.py 😄

import logging

from fabric.api import env, task, abort
from fabric.exceptions import CommandTimeout, NetworkError
from fabric.state import output

# Workaround for: No handlers could be found for logger "paramiko.transport"
# (see https://github.com/fabric/fabric/issues/51#issuecomment-96341022)
logging.basicConfig()
paramiko_logger = logging.getLogger("paramiko.transport")
paramiko_logger.disabled = True

Cheers
fotis

[ssli08]

I got the same error because of the 'env.port' variable. The value of 'env.port' which I specified is different from the server which contained in the 'env.roles'