Exception in thread #17

Open
jumping opened this Issue May 24, 2011 · 40 comments

Comments

Projects
None yet
@jumping

jumping commented May 24, 2011

Exception in thread Thread-12 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-13 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-9 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-8 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
@jumping

This comment has been minimized.

Show comment
Hide comment
@jumping

jumping May 25, 2011

more information:

Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29fa9e6cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29f9db4cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29f9ca2cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
Exception in thread Thread-8 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
 File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-7 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
 File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

jumping commented May 25, 2011

more information:

Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29fa9e6cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29f9db4cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
Unhandled exception in thread started by <bound method Transport.__bootstrap of <paramiko.Transport at 0x29f9ca2cL (unconnected)>>
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 462, in __bootstrap
   self.__bootstrap_inner()
 File "/usr/local/lib/python2.5/threading.py", line 499, in __bootstrap_inner
   (self.getName(), _format_exc()))
 File "/usr/local/lib/python2.5/traceback.py", line 235, in format_exc
   etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'
Exception in thread Thread-8 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
 File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-7 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
 File "build/bdist.freebsd-8.2-RELEASE-i386/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
@jonozzz

This comment has been minimized.

Show comment
Hide comment
@jonozzz

jonozzz Jul 28, 2011

This usually happens when you don't close() your ssh connection(s) before the main thread exits.

jonozzz commented Jul 28, 2011

This usually happens when you don't close() your ssh connection(s) before the main thread exits.

@murphyke

This comment has been minimized.

Show comment
Hide comment
@murphyke

murphyke Oct 12, 2011

My app often crashes when it exits with a paramiko-related traceback. The crashes happen with Paramiko 1.7.7.1 and Python 2.7.2 (on Mac OS X 10.6.8). The crashes do not happen with Python 2.6.7, although they did happen with Python 2.6.4.

My app creates, uses, and closes an SFTPClient, which I've confirmed by turning on the paramiko.transport logger. The app does not use threads itself.

Here you can see the last two paramiko.transport log messages, followed by the traceback:

INFO [chan 1] sftp session closed.
DEBUG [chan 1] EOF sent (1)
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/Users/murphyke/Documents/code/python/lib/python2.7/site-packages/paramiko/transport.py", line 1574, in run
Segmentation fault

My app often crashes when it exits with a paramiko-related traceback. The crashes happen with Paramiko 1.7.7.1 and Python 2.7.2 (on Mac OS X 10.6.8). The crashes do not happen with Python 2.6.7, although they did happen with Python 2.6.4.

My app creates, uses, and closes an SFTPClient, which I've confirmed by turning on the paramiko.transport logger. The app does not use threads itself.

Here you can see the last two paramiko.transport log messages, followed by the traceback:

INFO [chan 1] sftp session closed.
DEBUG [chan 1] EOF sent (1)
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/Users/murphyke/Documents/code/python/lib/python2.7/site-packages/paramiko/transport.py", line 1574, in run
Segmentation fault
@ifischer

This comment has been minimized.

Show comment
Hide comment
@ifischer

ifischer Jan 11, 2012

I'm running Ubuntu 11.04 and could reproduce the error with this test-script, which does close the connection:
https://bugs.launchpad.net/paramiko/+bug/786808/+attachment/2534903/+files/demo_simple.py
When run in a loop for ~100 times, the exception is raised from time to time. This happens as well as on Python 2.7.1 as on Python 2.6.6.

Any workarounds?

I'm running Ubuntu 11.04 and could reproduce the error with this test-script, which does close the connection:
https://bugs.launchpad.net/paramiko/+bug/786808/+attachment/2534903/+files/demo_simple.py
When run in a loop for ~100 times, the exception is raised from time to time. This happens as well as on Python 2.7.1 as on Python 2.6.6.

Any workarounds?

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Aug 27, 2012

Member

Consolidating to this ticket from other downstream sources:

  • ssh 28
  • Fabric #513 (has a bunch more example tracebacks, all similar; plus an actual look at the root error which is pretty wacky -- import socket somehow resulting in a None value for the socket binding.)

I'll be investigating potential sources of this bug in core Python/stdlib, and failing that, looking at ways to reproduce and handle the issue more gracefully. (It's also possible that the traceback'd error is hiding actual, more solvable errors -- which might require asking for more user feedback.)

That work will initially be done in ssh but the module in question has not diverged much/at all, so I'll be sure to copy it into paramiko as soon as it's useful.

Member

bitprophet commented Aug 27, 2012

Consolidating to this ticket from other downstream sources:

  • ssh 28
  • Fabric #513 (has a bunch more example tracebacks, all similar; plus an actual look at the root error which is pretty wacky -- import socket somehow resulting in a None value for the socket binding.)

I'll be investigating potential sources of this bug in core Python/stdlib, and failing that, looking at ways to reproduce and handle the issue more gracefully. (It's also possible that the traceback'd error is hiding actual, more solvable errors -- which might require asking for more user feedback.)

That work will initially be done in ssh but the module in question has not diverged much/at all, so I'll be sure to copy it into paramiko as soon as it's useful.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

Started digging a little; will update this comment as I go:

  • Googling and already found an unrelated instance of the exact same problem albeit one with no solution.
  • More instances, also under Python 2.6, also in Trac, can be found in Trac #3957.
    • One comment in that thread mentions they see "the traceback" frequently once switching to Python 2.6, so while they don't say WHICH traceback (there are multiple ones in that thread, only one of which is ours) it further implicates possibility of core Python bug & Python version playing a part.
    • Sadly since this only ever seems to come up in production systems, trying different Python versions is going to be hard/impossible to test.
  • Django #9582 is the same issue, which points to Python #1731. Unfortunately, that says it was fixed in Python 2.5.2, and I'm 99% sure I've seen this in Python 2.6 and possibly 2.7 as well, on top of the previous bullet point mentioning 2.6.
    • The commit message in the core Python patch fixing #1731 mentions the core problem was a daemon thread trying to clean up when much of the runtime environment is already gone. Sounds like a decent explanation, so I wonder if there's a regression or if the patch simply didn't fix all possible occurrences of the problem.
  • This archived paramiko ML discussion mentions it too and implies it's a bug in 2.5 (given the date and the fact that Python on OS X of the time was 2.5.1, guessing it's the same as above) fixed in 2.6.
  • Manually searching the Python bug tracker for all (not just open) socket none error issues finds a lot of hits but none that appear related.
  • https://bugs.launchpad.net/bzr/+bug/56316 is more of the same: yup, looks like the module unloaded by the time your code ran! Thanks, that much was obvious :( Starting to wonder if a dumb workaround might simply be to catch this actual AttributeError and attempt to re-import socket...
  • http://stackoverflow.com/questions/1745232/solving-thread-cleanup-on-paramiko - conflicting messages here, yes the issue seems to be a thread not shutting down correctly, but unclear if it's possible to protect against that reliably.
  • https://bugs.launchpad.net/paramiko/+bug/786808 - an old Paramiko bug report which includes a demo they claim can run in a loop & cause the issue.
    • EDIT: This is the same script as @ifischer pasted above. Good job @bitprophet!
    • This also reminds me that many of the hits so far mention SFTP -- be interesting to see if it is in fact always limited to situations where SFTP usage (in Fabric, e.g. put/get) is invoked.
  • http://www.mail-archive.com/bazaar-commits@lists.canonical.com/msg06998.html - search for 'error' and you'll find an interesting comment about how in general, interpreter shutdown sets all module globals to None and that reimporting is apparently a legit way to work around it.
Member

bitprophet commented Sep 12, 2012

Started digging a little; will update this comment as I go:

  • Googling and already found an unrelated instance of the exact same problem albeit one with no solution.
  • More instances, also under Python 2.6, also in Trac, can be found in Trac #3957.
    • One comment in that thread mentions they see "the traceback" frequently once switching to Python 2.6, so while they don't say WHICH traceback (there are multiple ones in that thread, only one of which is ours) it further implicates possibility of core Python bug & Python version playing a part.
    • Sadly since this only ever seems to come up in production systems, trying different Python versions is going to be hard/impossible to test.
  • Django #9582 is the same issue, which points to Python #1731. Unfortunately, that says it was fixed in Python 2.5.2, and I'm 99% sure I've seen this in Python 2.6 and possibly 2.7 as well, on top of the previous bullet point mentioning 2.6.
    • The commit message in the core Python patch fixing #1731 mentions the core problem was a daemon thread trying to clean up when much of the runtime environment is already gone. Sounds like a decent explanation, so I wonder if there's a regression or if the patch simply didn't fix all possible occurrences of the problem.
  • This archived paramiko ML discussion mentions it too and implies it's a bug in 2.5 (given the date and the fact that Python on OS X of the time was 2.5.1, guessing it's the same as above) fixed in 2.6.
  • Manually searching the Python bug tracker for all (not just open) socket none error issues finds a lot of hits but none that appear related.
  • https://bugs.launchpad.net/bzr/+bug/56316 is more of the same: yup, looks like the module unloaded by the time your code ran! Thanks, that much was obvious :( Starting to wonder if a dumb workaround might simply be to catch this actual AttributeError and attempt to re-import socket...
  • http://stackoverflow.com/questions/1745232/solving-thread-cleanup-on-paramiko - conflicting messages here, yes the issue seems to be a thread not shutting down correctly, but unclear if it's possible to protect against that reliably.
  • https://bugs.launchpad.net/paramiko/+bug/786808 - an old Paramiko bug report which includes a demo they claim can run in a loop & cause the issue.
    • EDIT: This is the same script as @ifischer pasted above. Good job @bitprophet!
    • This also reminds me that many of the hits so far mention SFTP -- be interesting to see if it is in fact always limited to situations where SFTP usage (in Fabric, e.g. put/get) is invoked.
  • http://www.mail-archive.com/bazaar-commits@lists.canonical.com/msg06998.html - search for 'error' and you'll find an interesting comment about how in general, interpreter shutdown sets all module globals to None and that reimporting is apparently a legit way to work around it.
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

So, all of that taken under consideration, it sounds like:

  • the cause is, as we sort of knew, threads terminating incorrectly / too late, and this setting socket to None;
  • one fix might be to try and prevent threads from terminating poorly, though this is not well defined, and given how intermittent it is, it may not be preventable at all;
  • another fix may be to catch this exception itself and try "re"-handling the error, either by reimporting socket (though I actually am not sure that would work -- it's chicken and egg, we can't tell we've been given a socket.error exception subclass without socket existing) or, while it's kind of ugly, add a new socket import inside the method in question.
    • I have to re-examine what the socket.error line is even doing, but it may make sense to simply treat an outer AttributeError the same as socket.error and take the same actions.
Member

bitprophet commented Sep 12, 2012

So, all of that taken under consideration, it sounds like:

  • the cause is, as we sort of knew, threads terminating incorrectly / too late, and this setting socket to None;
  • one fix might be to try and prevent threads from terminating poorly, though this is not well defined, and given how intermittent it is, it may not be preventable at all;
  • another fix may be to catch this exception itself and try "re"-handling the error, either by reimporting socket (though I actually am not sure that would work -- it's chicken and egg, we can't tell we've been given a socket.error exception subclass without socket existing) or, while it's kind of ugly, add a new socket import inside the method in question.
    • I have to re-examine what the socket.error line is even doing, but it may make sense to simply treat an outer AttributeError the same as socket.error and take the same actions.
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

Using the demo script, I can get this to occur under Python 2.6.7, but not Python 2.5.6. Also can't get it to happen via a fabfile doing roughly, but not exactly, the same thing. Latter not a huge deal, if we can fix this in the base case I'm sure it will help for when the issue crops up via Fabric.

Using Python 2.6, will start testing potential workarounds.

Member

bitprophet commented Sep 12, 2012

Using the demo script, I can get this to occur under Python 2.6.7, but not Python 2.5.6. Also can't get it to happen via a fabfile doing roughly, but not exactly, the same thing. Latter not a huge deal, if we can fix this in the base case I'm sure it will help for when the issue crops up via Fabric.

Using Python 2.6, will start testing potential workarounds.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

I now seem to need to run the test script 1000x to get 1-2 errors, not 100x, but it does still seem to barf at least once every 1000x run. (My netbook's CPU & fan is kind of pissed at me right now, but whatever. It can deal.)

Interestingly, the test script's try/except does not function; the core threading/attribute related error prints itself and the print statement in the script never shows up. Not sure what that means for workaround attempts, but I'll find out.

Member

bitprophet commented Sep 12, 2012

I now seem to need to run the test script 1000x to get 1-2 errors, not 100x, but it does still seem to barf at least once every 1000x run. (My netbook's CPU & fan is kind of pissed at me right now, but whatever. It can deal.)

Interestingly, the test script's try/except does not function; the core threading/attribute related error prints itself and the print statement in the script never shows up. Not sure what that means for workaround attempts, but I'll find out.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member

Quick notes from initial look at workarounds:

  • transport.py explicitly tries to handle thread cleanup, which I hadn't noticed. In the test script this always runs every time for the one thread that gets created.
  • The error handling within run feels like it could be made more thorough; I'm guessing the assumption is the handful of cascading except clauses handle every possible error (esp given it ends with except Exception) and so it doesn't need an overarching try/finally.
    • Given the above comment re: the test script, this may be technically correct: it may not be possible to capture the exception this ticket is about. If so, we're kind of hosed :(
  • I tried adding a simple if socket is None: import socket to the top of run but that yields an UnboundLocalError. Not sure why; surely socket should exist at that point since it's a top level import?
    • If memory serves, re-importing an already-imported module should be a no-op, so perhaps a vanilla import socket inside run would be kosher. But years of "don't import things inside functions" makes me cringe at the thought & assume it'd still be biting somebody somewhere.
Member

bitprophet commented Sep 12, 2012

Quick notes from initial look at workarounds:

  • transport.py explicitly tries to handle thread cleanup, which I hadn't noticed. In the test script this always runs every time for the one thread that gets created.
  • The error handling within run feels like it could be made more thorough; I'm guessing the assumption is the handful of cascading except clauses handle every possible error (esp given it ends with except Exception) and so it doesn't need an overarching try/finally.
    • Given the above comment re: the test script, this may be technically correct: it may not be possible to capture the exception this ticket is about. If so, we're kind of hosed :(
  • I tried adding a simple if socket is None: import socket to the top of run but that yields an UnboundLocalError. Not sure why; surely socket should exist at that point since it's a top level import?
    • If memory serves, re-importing an already-imported module should be a no-op, so perhaps a vanilla import socket inside run would be kosher. But years of "don't import things inside functions" makes me cringe at the thought & assume it'd still be biting somebody somewhere.
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 12, 2012

Member
  • Changed to regular import socket at top of run; went a whole 1000-cycle run with no exceptions. Sadly the 2nd 1000-cycle run had one exception. Boo-urns.
  • Minor note: moved to developing on a cloud instance, that happens to be running Python 2.6.6. Getting the exception significantly more frequently, as in 15 times per 1000x (in the 1st run anyway) instead of 1-2 times. (This is sans the import addition, so far.)
  • Adding the import socket inside run on the cloud instance, I now only seem to get a different but very similar error, packet.py complaining about socket being None. This may have been what I was seeing up above when running on the Mac, I don't recall. Stands to reason an undead thread, with the original/first "imported module is None" error fixed, would then encounter it elsewhere.
Member

bitprophet commented Sep 12, 2012

  • Changed to regular import socket at top of run; went a whole 1000-cycle run with no exceptions. Sadly the 2nd 1000-cycle run had one exception. Boo-urns.
  • Minor note: moved to developing on a cloud instance, that happens to be running Python 2.6.6. Getting the exception significantly more frequently, as in 15 times per 1000x (in the 1st run anyway) instead of 1-2 times. (This is sans the import addition, so far.)
  • Adding the import socket inside run on the cloud instance, I now only seem to get a different but very similar error, packet.py complaining about socket being None. This may have been what I was seeing up above when running on the Mac, I don't recall. Stands to reason an undead thread, with the original/first "imported module is None" error fixed, would then encounter it elsewhere.
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 13, 2012

Member

Fun times: literally everything top-level is in fact set to None during these interpreter explosions.

Once the first two socket.* related errors are "fixed" by local import socket calls, we start getting things like len and even sys.stdout (when trying to verify the len via a regular print statement!) being None and causing their own explosions.

Think we need a different solution here -- even if we tried doing something awful like if socket is None: sys.exit(0) or whatever, I doubt it'd work because sys.exit would be None too.

Googling for "most likely raised during interpreter shutdown" to see if there is any generic wisdom for this type of exception.

Member

bitprophet commented Sep 13, 2012

Fun times: literally everything top-level is in fact set to None during these interpreter explosions.

Once the first two socket.* related errors are "fixed" by local import socket calls, we start getting things like len and even sys.stdout (when trying to verify the len via a regular print statement!) being None and causing their own explosions.

Think we need a different solution here -- even if we tried doing something awful like if socket is None: sys.exit(0) or whatever, I doubt it'd work because sys.exit would be None too.

Googling for "most likely raised during interpreter shutdown" to see if there is any generic wisdom for this type of exception.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 13, 2012

Member

Found Python #1856 which seems to imply it's a generic, known Python core bug that isn't fixed until Python 3.2. Ugh. And sadly their topic was how to fix it in the interpreter, not how to work around it in userland (with a few small exceptions.)

More results, most of them strongly related to each other:

Member

bitprophet commented Sep 13, 2012

Found Python #1856 which seems to imply it's a generic, known Python core bug that isn't fixed until Python 3.2. Ugh. And sadly their topic was how to fix it in the interpreter, not how to work around it in userland (with a few small exceptions.)

More results, most of them strongly related to each other:

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 13, 2012

Member

Tried the "hold references to things we use" approach, but it's really about the same as the "reimport stuff" approach: every time I patch up one object reference, others take its place. Practically any method within e.g. packet.Packetizer seems liable to run in this dead-interpreter state, and obviously that's a lot of potential references.

Next step: try the "check if sys.modules is None" test for squashing the exceptions.

Member

bitprophet commented Sep 13, 2012

Tried the "hold references to things we use" approach, but it's really about the same as the "reimport stuff" approach: every time I patch up one object reference, others take its place. Practically any method within e.g. packet.Packetizer seems liable to run in this dead-interpreter state, and obviously that's a lot of potential references.

Next step: try the "check if sys.modules is None" test for squashing the exceptions.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 13, 2012

Member

A preliminary test which wraps most of the body of run in the sys.modules-testing approach outlined in Python 9205 seems to yield good results, I ran one 1000-cycle test with zero errors. Will continue to up the # cycles until I'm positive it's helping. Will also try to counter-test to make sure it's not swallowing real exceptions, though the sys.modules bit should be near bulletproof.

EDIT: 2nd 1000-cycle ran 100% OK. Doing a 10,000-cycle test now, if that has no errors I'm gonna say this is fixed & hope the extra layer of try/except is sufficiently stable re: normal operations. EDIT 2: 10,000 cycles with no errors. Woo.

FWIW the test suite also still passes. Will try running some fabfile commands as well for a final sanity test.

Member

bitprophet commented Sep 13, 2012

A preliminary test which wraps most of the body of run in the sys.modules-testing approach outlined in Python 9205 seems to yield good results, I ran one 1000-cycle test with zero errors. Will continue to up the # cycles until I'm positive it's helping. Will also try to counter-test to make sure it's not swallowing real exceptions, though the sys.modules bit should be near bulletproof.

EDIT: 2nd 1000-cycle ran 100% OK. Doing a 10,000-cycle test now, if that has no errors I'm gonna say this is fixed & hope the extra layer of try/except is sufficiently stable re: normal operations. EDIT 2: 10,000 cycles with no errors. Woo.

FWIW the test suite also still passes. Will try running some fabfile commands as well for a final sanity test.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 13, 2012

Member

Commit over in ssh repo: bitprophet/ssh@3709d2e

About to port to Paramiko & push there. Then hopefully find some brave souls to test it out pre-release.

Member

bitprophet commented Sep 13, 2012

Commit over in ssh repo: bitprophet/ssh@3709d2e

About to port to Paramiko & push there. Then hopefully find some brave souls to test it out pre-release.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 14, 2012

Member

There's the commit. What I did:

  • Confirmed existence of bug in Paramiko pre-patch
  • Confirmed test suite ran OK
  • Patched
  • Confirmed test suite still OK
  • Confirmed 1000-cycle run with no exceptions
  • Confirmed 10,000-cycle run with no exceptions.
Member

bitprophet commented Sep 14, 2012

There's the commit. What I did:

  • Confirmed existence of bug in Paramiko pre-patch
  • Confirmed test suite ran OK
  • Patched
  • Confirmed test suite still OK
  • Confirmed 1000-cycle run with no exceptions
  • Confirmed 10,000-cycle run with no exceptions.
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 14, 2012

Member

So, if anybody who was seeing this issue wants to nab latest master and confirm the fix, it'd be greatly appreciated!

Member

bitprophet commented Sep 14, 2012

So, if anybody who was seeing this issue wants to nab latest master and confirm the fix, it'd be greatly appreciated!

@jumping

This comment has been minimized.

Show comment
Hide comment
@jumping

jumping Sep 25, 2012

still has Exception output:

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-23 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-11 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-38 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-34 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1602, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-13 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1602, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

jumping commented Sep 25, 2012

still has Exception output:

Exception in thread Thread-3 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-23 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-11 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-38 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1599, in run
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1402, in _log
  File "/usr/local/lib/python2.5/logging/__init__.py", line 1048, in log
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'IntType'
Exception in thread Thread-34 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1602, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-13 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/local/lib/python2.5/threading.py", line 486, in __bootstrap_inner
  File "build/bdist.freebsd-8.2-RELEASE-i386/egg/ssh/transport.py", line 1602, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Sep 26, 2012

Member

@jumping Which exact checkout SHA are you using in the above, and have you made 100% sure you're importing that checkout? (e.g. nuke virtualenv & reinstall, make sure no 'ssh' exists in system site-packages, that sort of thing.)

Can you clarify exactly how you are reproducing this & how frequently it's occurring for you?

Finally -- if this is easily reproducible & the above checks out, can you try adding a print statement to the sys.modules check we added, and see if that is somehow non-None when these exceptions occur?


tl;dr I cannot replicate this any longer with the fix in place, so the burden's back on you to help figure out what's still missing :)

Member

bitprophet commented Sep 26, 2012

@jumping Which exact checkout SHA are you using in the above, and have you made 100% sure you're importing that checkout? (e.g. nuke virtualenv & reinstall, make sure no 'ssh' exists in system site-packages, that sort of thing.)

Can you clarify exactly how you are reproducing this & how frequently it's occurring for you?

Finally -- if this is easily reproducible & the above checks out, can you try adding a print statement to the sys.modules check we added, and see if that is somehow non-None when these exceptions occur?


tl;dr I cannot replicate this any longer with the fix in place, so the burden's back on you to help figure out what's still missing :)

@vincentbernat

This comment has been minimized.

Show comment
Hide comment
@vincentbernat

vincentbernat Feb 14, 2013

I get a similar problem. I am using Paramiko through Fabric through py.test. I get at the following exceptions:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/home/bernat/.virtualenvs/pytest/local/lib/python2.7/site-packages/paramiko/transp
ort.py", line 1641, in run
    self.sock.close()
  File "/home/bernat/.virtualenvs/pytest/local/lib/python2.7/site-packages/paramiko/proxy.
py", line 86, in close
    os.kill(self.process.pid, signal.SIGTERM)
AttributeError: 'NoneType' object has no attribute 'kill'

self.sys.modules is not empty. self.sys.modules["os"] even exists. To workaround the issue, I have modified proxy.py:

    def close(self):
        self.os.kill(self.process.pid, self.signal.SIGTERM)

And in __init__():

        self.os = os
        self.signal = signal

I didn't have any problem since. This may help find the root cause (sys.modules in one module does not reflect sys.modules in another?).

I get a similar problem. I am using Paramiko through Fabric through py.test. I get at the following exceptions:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/home/bernat/.virtualenvs/pytest/local/lib/python2.7/site-packages/paramiko/transp
ort.py", line 1641, in run
    self.sock.close()
  File "/home/bernat/.virtualenvs/pytest/local/lib/python2.7/site-packages/paramiko/proxy.
py", line 86, in close
    os.kill(self.process.pid, signal.SIGTERM)
AttributeError: 'NoneType' object has no attribute 'kill'

self.sys.modules is not empty. self.sys.modules["os"] even exists. To workaround the issue, I have modified proxy.py:

    def close(self):
        self.os.kill(self.process.pid, self.signal.SIGTERM)

And in __init__():

        self.os = os
        self.signal = signal

I didn't have any problem since. This may help find the root cause (sys.modules in one module does not reflect sys.modules in another?).

@jizhang

This comment has been minimized.

Show comment
Hide comment
@jizhang

jizhang May 20, 2013

I'm using Fabric in a background job and facing the same problem. Sometime it'll get the main thread hung, so I have to write another job to help killing it...

Pakcage version info:
Fabric==1.6.0
paramiko==1.10.1
pycrypto==2.6

Any follow up on this issue?

jizhang commented May 20, 2013

I'm using Fabric in a background job and facing the same problem. Sometime it'll get the main thread hung, so I have to write another job to help killing it...

Pakcage version info:
Fabric==1.6.0
paramiko==1.10.1
pycrypto==2.6

Any follow up on this issue?

@agnivade

This comment has been minimized.

Show comment
Hide comment
@agnivade

agnivade Aug 13, 2013

Hi, any chance this bug fix can be moved to the release codebase so that its available publicly? I need a fix for this urgently. Or please suggest a workaround for this.

Hi, any chance this bug fix can be moved to the release codebase so that its available publicly? I need a fix for this urgently. Or please suggest a workaround for this.

@johaven

This comment has been minimized.

Show comment
Hide comment
@johaven

johaven Sep 14, 2013

On windows, there is a bug :

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 808, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\site-packages\paramiko-1.11.0-py2.7.egg\paramiko\transport.py", line 1641, in run
    self.sock.close()
  File "C:\Python27\lib\site-packages\paramiko-1.11.0-py2.7.egg\paramiko\proxy.py", line 86, in close
    os.kill(self.process.pid, signal.SIGTERM)
WindowsError: [Error 5] Access denied

To resolve it in proxy.py :

    def close(self):
         self.process.kill()
        #os.kill(self.process.pid, signal.SIGTERM)

johaven commented Sep 14, 2013

On windows, there is a bug :

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 808, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\site-packages\paramiko-1.11.0-py2.7.egg\paramiko\transport.py", line 1641, in run
    self.sock.close()
  File "C:\Python27\lib\site-packages\paramiko-1.11.0-py2.7.egg\paramiko\proxy.py", line 86, in close
    os.kill(self.process.pid, signal.SIGTERM)
WindowsError: [Error 5] Access denied

To resolve it in proxy.py :

    def close(self):
         self.process.kill()
        #os.kill(self.process.pid, signal.SIGTERM)
@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Oct 16, 2013

Member

@agnivade If you mean the original bugfix it was published a year ago! If you mean @vincentbernat 's suggestion, I have to see if I can recreate the problem he is having to verify the fix.

Vincent - any help would be appreciated. All I'll be able to do is look up what I was doing a year ago & see if that test setup yields errors again (& if so, whether your change helps.) No guarantee these two issues are actually linked, though.

Member

bitprophet commented Oct 16, 2013

@agnivade If you mean the original bugfix it was published a year ago! If you mean @vincentbernat 's suggestion, I have to see if I can recreate the problem he is having to verify the fix.

Vincent - any help would be appreciated. All I'll be able to do is look up what I was doing a year ago & see if that test setup yields errors again (& if so, whether your change helps.) No guarantee these two issues are actually linked, though.

@agnivade

This comment has been minimized.

Show comment
Hide comment
@agnivade

agnivade Oct 16, 2013

No I meant the original bug fix. In my ubuntu, I don't see the paramiko version to be 1.11 when I download the latest debian. That's why I was asking this. Anyway, even with the fix from the latest branch, the problem persists. But currently, I can live with it because the program crashes only after the last item of the loop.

No I meant the original bug fix. In my ubuntu, I don't see the paramiko version to be 1.11 when I download the latest debian. That's why I was asking this. Anyway, even with the fix from the latest branch, the problem persists. But currently, I can live with it because the program crashes only after the last item of the loop.

@bitprophet

This comment has been minimized.

Show comment
Hide comment
@bitprophet

bitprophet Oct 17, 2013

Member

@agnivade I can do nothing about your operating system's version :) but thanks for the update. Sounds like this (or related...there are a LOT of things that can cause this error message) issue is still popping up sometimes. This sort of bug is hard to eradicate :(

Member

bitprophet commented Oct 17, 2013

@agnivade I can do nothing about your operating system's version :) but thanks for the update. Sounds like this (or related...there are a LOT of things that can cause this error message) issue is still popping up sometimes. This sort of bug is hard to eradicate :(

@agnivade

This comment has been minimized.

Show comment
Hide comment
@agnivade

agnivade Oct 17, 2013

I don't think I was clear. Its not about the version of my OS. Its the version of paramiko which is old, when I do an apt-get install. I am saying that you haven't pushed the latest version to the public ubuntu repository.

I don't think I was clear. Its not about the version of my OS. Its the version of paramiko which is old, when I do an apt-get install. I am saying that you haven't pushed the latest version to the public ubuntu repository.

@zertrin

This comment has been minimized.

Show comment
Hide comment
@zertrin

zertrin Nov 16, 2013

@agnivade it is the job of the package maintainers of Debian and Ubuntu to update the package for their respective distributions based on the updated source code upstream (here).

Unless the maintainer or the upstream code is also one of the maintainer of the packages in the distributions, the only thing that can be done is to ask the package maintainer to update the package with the latest version of paramiko and include it in the distribution.

For Debian, you'll find the appropriate informations here : http://packages.qa.debian.org/p/paramiko.html

zertrin commented Nov 16, 2013

@agnivade it is the job of the package maintainers of Debian and Ubuntu to update the package for their respective distributions based on the updated source code upstream (here).

Unless the maintainer or the upstream code is also one of the maintainer of the packages in the distributions, the only thing that can be done is to ask the package maintainer to update the package with the latest version of paramiko and include it in the distribution.

For Debian, you'll find the appropriate informations here : http://packages.qa.debian.org/p/paramiko.html

@jbouse

This comment has been minimized.

Show comment
Hide comment
@jbouse

jbouse Nov 16, 2013

@zertrin, I am the Debian package maintainer, and use I haven't gotten the upgrade from 1.10.1 to 1.12.0 done yet but it is on my TODO list.

jbouse commented Nov 16, 2013

@zertrin, I am the Debian package maintainer, and use I haven't gotten the upgrade from 1.10.1 to 1.12.0 done yet but it is on my TODO list.

@zertrin

This comment has been minimized.

Show comment
Hide comment
@zertrin

zertrin Nov 16, 2013

@jbouse :)

For the record, I wasn't complaining here, just explaining to @agnivade that it's not because upstream has been updated that all the packages in all the distros are updated right away. Maintaining a package isn't always trivial, takes time and above all isn't necessarily done by the upstream developers.

Thanks for the heads-up anyway!

zertrin commented Nov 16, 2013

@jbouse :)

For the record, I wasn't complaining here, just explaining to @agnivade that it's not because upstream has been updated that all the packages in all the distros are updated right away. Maintaining a package isn't always trivial, takes time and above all isn't necessarily done by the upstream developers.

Thanks for the heads-up anyway!

@jbouse

This comment has been minimized.

Show comment
Hide comment
@jbouse

jbouse Nov 17, 2013

@zertrin there is also the fact that depending on which release of a given distro it may never be updated unless it's a security fix that is backported to the version in that release. ie- Debian stable will never version than 1.7.7.1 as that was what was in stable when it was released. Only Debian testing and unstable can be updated.

jbouse commented Nov 17, 2013

@zertrin there is also the fact that depending on which release of a given distro it may never be updated unless it's a security fix that is backported to the version in that release. ie- Debian stable will never version than 1.7.7.1 as that was what was in stable when it was released. Only Debian testing and unstable can be updated.

@QuentinJi

This comment has been minimized.

Show comment
Hide comment
@QuentinJi

QuentinJi Jan 14, 2014

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
  File "build/bdist.linux-i686/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

On ubuntu13.10 + paramiko-1.7.6

Fixed by update to paramiko-1.12.1

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
  File "build/bdist.linux-i686/egg/paramiko/transport.py", line 1574, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

On ubuntu13.10 + paramiko-1.7.6

Fixed by update to paramiko-1.12.1

@philippkeller

This comment has been minimized.

Show comment
Hide comment
@philippkeller

philippkeller Feb 20, 2014

I ran into the same issue, got solved through upgrading paramiko, thanks guys!

I ran into the same issue, got solved through upgrading paramiko, thanks guys!

openstack-gerrit pushed a commit to openstack/fuel-main that referenced this issue Apr 7, 2014

Igor Shishkin
Version of paramiko removed from requirements.txt
In recent version fixed thread shutdown exceptions:
paramiko/paramiko#17

Change-Id: Ibdbc0fee74ff630475900c977367d29b7dff5ebe
@jbouse

This comment has been minimized.

Show comment
Hide comment
@jbouse

jbouse May 28, 2014

I'm just looking to see if this issue is still current in latest versions as Debian BTS 698075 that references this issue hasn't seen any updates since Jan 2013 and I've since gotten 1.14.0 packaged and pushed out.

jbouse commented May 28, 2014

I'm just looking to see if this issue is still current in latest versions as Debian BTS 698075 that references this issue hasn't seen any updates since Jan 2013 and I've since gotten 1.14.0 packaged and pushed out.

@bitprophet bitprophet added the Bug label Aug 8, 2014

@leopoul leopoul referenced this issue in ncclient/ncclient Oct 25, 2014

Closed

Error during Python "shutdown" #38

@khenderick

This comment has been minimized.

Show comment
Hide comment
@khenderick

khenderick May 8, 2015

Using paramiko 1.15.2, I'm still having this issue:

Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1642, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

Using paramiko 1.15.2, I'm still having this issue:

Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/local/lib/python2.7/dist-packages/paramiko/transport.py", line 1642, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
@mksharma

This comment has been minimized.

Show comment
Hide comment
@mksharma

mksharma May 14, 2015

I installed latest Paramiko 1.15.2 and using python 2.7
Got this error:
Captured stdout:
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
File "/opt/move/epg-tools/local/lib/python2.7/site-packages/paramiko/transport.py", line 1613, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

I installed latest Paramiko 1.15.2 and using python 2.7
Got this error:
Captured stdout:
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
File "/opt/move/epg-tools/local/lib/python2.7/site-packages/paramiko/transport.py", line 1613, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

@curtmcd

This comment has been minimized.

Show comment
Hide comment
@curtmcd

curtmcd May 29, 2015

I'm still suffering from this problem under Python 2.7.6, Fabric 1.8.2, Paramiko 1.10.1, and have developed the following workaround that seems to be working so far.

My program allocates an object in a main() function from a class that uses Fabric calls. I have Fabric close all ssh connections in this object's destructor because the object goes out of scope when main() returns.

import fabric.network
import fabric.state

class MyObject(...):
    ... methods that use fabric ...
    def __del__(self):
        fabric.state.output.status = False
        fabric.network.disconnect_all()

curtmcd commented May 29, 2015

I'm still suffering from this problem under Python 2.7.6, Fabric 1.8.2, Paramiko 1.10.1, and have developed the following workaround that seems to be working so far.

My program allocates an object in a main() function from a class that uses Fabric calls. I have Fabric close all ssh connections in this object's destructor because the object goes out of scope when main() returns.

import fabric.network
import fabric.state

class MyObject(...):
    ... methods that use fabric ...
    def __del__(self):
        fabric.state.output.status = False
        fabric.network.disconnect_all()
@styk-tv

This comment has been minimized.

Show comment
Hide comment
@styk-tv

styk-tv Jun 16, 2016

Happens to me on Mac (10.11.5) dev when using execute @parallel tasks on multiple hosts. Task execute successfully but this exception happens on exit and is followed with Process finished with exit code 0 message which is somewhat confusing.

Fabric (1.11.1)
paramiko (1.15.2)
Exception in thread Thread-9 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner File "/usr/local/lib/python2.7/site-packages/paramiko/transport.py", line 1642, in run

styk-tv commented Jun 16, 2016

Happens to me on Mac (10.11.5) dev when using execute @parallel tasks on multiple hosts. Task execute successfully but this exception happens on exit and is followed with Process finished with exit code 0 message which is somewhat confusing.

Fabric (1.11.1)
paramiko (1.15.2)
Exception in thread Thread-9 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner File "/usr/local/lib/python2.7/site-packages/paramiko/transport.py", line 1642, in run

@eandersson eandersson referenced this issue in eandersson/amqpstorm Feb 7, 2017

Closed

Cant publish multiple messages #37

@kvanhijf kvanhijf referenced this issue in openvstorage/framework Apr 24, 2017

Closed

Attribute error of paramiko after ovs setup #1536

@noob-4-life noob-4-life referenced this issue in balloob/pychromecast Sep 13, 2017

Open

Exception on exit... #115

@dependabot dependabot bot referenced this issue in edx/configuration Mar 19, 2018

Merged

Bump paramiko from 2.4.0 to 2.4.1 #4388

@mpetronic

This comment has been minimized.

Show comment
Hide comment
@mpetronic

mpetronic Apr 5, 2018

I'm running Python 2.7.14 and paramiko==2.4.1 and Fabric==1.14.0 and using fabric in an integration test to manage setting up external resources for the test. If I add the following, it seems to eliminate the exception. I can run the testing repeatedly without the tearDownClass() method, and I get the exception probably about 50% of the time, on exiting the test successfully. It is mainly annoying to see this exception print out as the last output from a successful test. :) With the tearDownClass below, I don't see the exception occurring:

@classmethod
def tearDownClass(cls):
    # This is part of trying to suppress a paramiko shutdown bug
    # https://github.com/paramiko/paramiko/issues/17
    logger.info('tearDownClass')
    fabric.state.output.status = False
    fabric.network.disconnect_all()

Kudos to @curtmcd for the suggestion of the two calls to fabric to disconnect_all()

I'm running Python 2.7.14 and paramiko==2.4.1 and Fabric==1.14.0 and using fabric in an integration test to manage setting up external resources for the test. If I add the following, it seems to eliminate the exception. I can run the testing repeatedly without the tearDownClass() method, and I get the exception probably about 50% of the time, on exiting the test successfully. It is mainly annoying to see this exception print out as the last output from a successful test. :) With the tearDownClass below, I don't see the exception occurring:

@classmethod
def tearDownClass(cls):
    # This is part of trying to suppress a paramiko shutdown bug
    # https://github.com/paramiko/paramiko/issues/17
    logger.info('tearDownClass')
    fabric.state.output.status = False
    fabric.network.disconnect_all()

Kudos to @curtmcd for the suggestion of the two calls to fabric to disconnect_all()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment