tqdm has no attribute 'disable'

[midnightmagic]

System is a slightly dated NetBSD environment. tqdm is the latest version, installed via pip and Python version is as follows. It can be corrected (but then a bunch of other missing attribute errors suddenly show up) by adding an explicit attribute in the tqdm class definition.

(:dom13:01:32:36 /tmp#) pip3.5 list | grep -i tq
tqdm (4.11.2)
(:dom13:01:31:54 /tmp#) python3.5 --version
Python 3.5.3

Traceback follows:

(:dom13:00:56:11 /tmp#) wormhole receive
Enter receive wormhole code: blah-blah
 (note: you can use <Tab> to complete words)
Receiving file (20 Bytes) into: test_file
ok? (y/n): y
Receiving (->tcp:10.0.0.x:60133)..
Exception ignored in: <object repr() failed>
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.5/site-packages/tqdm/_tqdm.py", line 764, in __del__
    self.close()
  File "/usr/pkg/lib/python3.5/site-packages/tqdm/_tqdm.py", line 994, in close
    if self.disable:
AttributeError: 'tqdm' object has no attribute 'disable'
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.5/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/internet/defer.py", line 1357, in gotResult
    _inlineCallbacks(r, g, deferred)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
--- <exception caught here> ---
  File "/usr/pkg/lib/python3.5/site-packages/wormhole/cli/cli.py", line 106, in _dispatch_command
    yield maybeDeferred(command)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/pkg/lib/python3.5/site-packages/wormhole/cli/cmd_receive.py", line 78, in go
    yield d
  File "/usr/pkg/lib/python3.5/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/pkg/lib/python3.5/site-packages/wormhole/cli/cmd_receive.py", line 115, in _go
    yield self._parse_offer(them_d[u"offer"], w)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/pkg/lib/python3.5/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/pkg/lib/python3.5/site-packages/wormhole/cli/cmd_receive.py", line 191, in _parse_offer
    datahash = yield self._transfer_data(rp, f)
builtins.TypeError: 'NoneType' object is not callable
ERROR: 'NoneType' object is not callable

[warner]

Thats.. weird. It seems like a bug in tqdm, but looking at their __init__.py, I can't see a way for .disable to be missing.

Is this reproducible? Could you maybe add some print statements into tqdm, in __init__, to see if it makes it all the way to the end of the method? Maybe something is throwing an exception early, before self.disable = disable is set, but the code that's calling it is eating the exception.

[midnightmagic]

Explicitly creating a "disable" attribute in the class definition outside the method fixes this particular problem. Perhaps some kind of Python3 class issue? It's completely reproducible. wormhole on that machine is non-functional as a result. :-)

[warner]

could you paste the full pip list for that virtualenv? I haven't been able to reproduce this locally yet, and I think I'll need to before I can figure out what's going on.

[warner]

Thanks to @cyli, we tracked this down to a tqdm bug, where it's somehow misidentifying your operating system, and then fails to handle a return value that's None.

Could you run this snippet and see if it fails on your box?

import tqdm, sys, time

progress = tqdm.tqdm(file=sys.stderr, unit="B", unit_scale=True, total=100)
with progress:
    for i in range(100):
        progress.update(i)
        time.sleep(0.01)
print "exiting"

If it's the bug we think it is, that will fail with something like:

% python /tmp/test-tqdm.py
Traceback (most recent call last):
  File "/tmp/test-tqdm.py", line 3, in <module>
    progress = tqdm.tqdm(file=sys.stderr, unit="B", unit_scale=True, total=100)
  File "/Users/warner/stuff/tahoe/magic-wormhole/ve/lib/python2.7/site-packages/tqdm/_tqdm.py", line 677, in __init__
    ncols = _environ_cols_wrapper()(file)
TypeError: 'NoneType' object is not callable
Exception AttributeError: "'tqdm' object has no attribute 'disable'" in <object repr() failed> ignored

The tqdm object has a __del__ which calls self.close(), which seems unwise, because the object may be halfway deleted by the time it tries to call itself, and self.close() checks for self.disable. So if an exception is raised during __init__ (by that _environ_cols_wrapper being None, because it misrecognized your OS), when the object goes out of scope, __del__ is called, then close(), then an exception happens.

And then our use of @inlineCallbacks makes the error get displayed weird.

The root problem is probably in tqdm._utils, in this code:

from platform import system as _curos
CUR_OS = _curos()
IS_WIN = CUR_OS in ['Windows', 'cli']
IS_NIX = (not IS_WIN) and any(
    CUR_OS.startswith(i) for i in
    ['CYGWIN', 'MSYS', 'Linux', 'Darwin', 'SunOS', 'FreeBSD', 'NetBSD'])

and then this:

def _environ_cols_wrapper():  # pragma: no cover
    """
    Return a function which gets width and height of console
    (linux,osx,windows,cygwin).
    """
    _environ_cols = None
    if IS_WIN:
        _environ_cols = _environ_cols_windows
        if _environ_cols is None:
            _environ_cols = _environ_cols_tput
    if IS_NIX:
        _environ_cols = _environ_cols_linux
    return _environ_cols

I think that on your machine, neither IS_WIN nor IS_NIX are True, so _environ_cols_windows() returns None, and _tqdm.__init__ can't deal with that.

[warner]

I just took a look at the current tqdm issues and pull requests, and didn't see this problem anywhere. tqdm/tqdm#383 might be related, as it shows an internal use of an attribute failing.

What does platform.system() report on your box?

If you patch your tqdm/_utils.py to force IS_NIX = True, does the problem go away?

If that helps, then we can go file a tqdm bug and recommend that they tolerate an unrecognized system better. I think the __del__ is a potential problem too, but probably wouldn't happen too frequently.

[midnightmagic]

Hello!

I guess I'll answer your notes each in turn.

(:dom13:03:32:40 /#) pip3.5 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
appdirs (1.4.3)
asn1crypto (0.22.0)
attrs (17.1.0)
autobahn (17.5.1)
Automat (0.6.0)
certifi (2017.4.17)
cffi (1.10.0)
click (6.7)
constantly (15.1.0)
cryptography (1.8.1)
hkdf (0.0.3)
humanize (0.5.1)
hypothesis (3.6.1)
idna (2.5)
incremental (16.10.1)
ipaddress (1.0.18)
magic-wormhole (0.9.2)
packaging (16.8)
pip (9.0.1)
py (1.4.30)
pyasn1 (0.2.3)
pyasn1-modules (0.0.8)
pycparser (2.17)
PyNaCl (1.1.2)
pyOpenSSL (17.0.0)
pyparsing (2.2.0)
pytest (3.0.7)
PyYAML (3.12)
service-identity (16.0.0)
setuptools (34.3.3)
six (1.10.0)
spake2 (0.7)
tqdm (4.11.2)
Twisted (17.1.0)
txaio (2.7.1)
zope.interface (4.4.1)

Changing your snippet to:

print("exiting")

... and then runing it results in:

python3.5 test
Traceback (most recent call last):
  File "test", line 3, in <module>
    progress = tqdm.tqdm(file=sys.stderr, unit="B", unit_scale=True, total=100)
  File "/usr/pkg/lib/python3.5/site-packages/tqdm/_tqdm.py", line 678, in __init__
    ncols = _environ_cols_wrapper()(file)
TypeError: 'NoneType' object is not callable
Exception ignored in: <object repr() failed>
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.5/site-packages/tqdm/_tqdm.py", line 765, in __del__
    self.close()
  File "/usr/pkg/lib/python3.5/site-packages/tqdm/_tqdm.py", line 1002, in close
    pos = self.pos
AttributeError: 'tqdm' object has no attribute 'pos'

Subsequently, I ran the additional bits:

>>> CUR_OS 
'NetBSD'
>>> IS_NIX = (not IS_WIN) and any(
    CUR_OS.startswith(i) for i in
    ['CYGWIN', 'MSYS', 'Linux', 'Darwin', 'SunOS', 'FreeBSD', 'NetBSD'])... ... 
>>> IS_NIX
True

For the platform:

>>> import platform
>>> platform.system()
'NetBSD'

HOWEVER in examining the code, I do see the IS_NIX check does not check for NetBSD. Instead, it looks like this:

IS_NIX = (not IS_WIN) and any(
    CUR_OS.startswith(i) for i in
    ['CYGWIN', 'MSYS', 'Linux', 'Darwin', 'SunOS', 'FreeBSD'])

I corrected that to include NetBSD, and it now works.

(:dom13:03:43:03 /tmp#) wormhole receive
Enter receive wormhole code: 2-informant-tycoon
 (note: you can use <Tab> to complete words)
Receiving file (20 Bytes) into: test_file
ok? (y/n): y
Receiving (->tcp:10.x.x.x:46244)..
100%|██████████████| 20.0/20.0 [00:00<00:00, 103B/s]
Received file written to test_file
 20.0/20.0 [00:00<00:00, 103B/s]
Received file written to test_file

I don't believe I need to file an issue with the tqdm people, because another NetBSD type has already done so, here:

tqdm/tqdm@a379e33
tqdm/tqdm#344

So.. looks like all I need is just a simple fix after all.

[casperdcl]

Hi guys, thanks for your great work looking into this. tqdm should not fail when it fails to detect your system, it should fallback to a sensible default. I'll add a patch.

Secondly, are you sure it is unsafe for __del__() to call object methods? It seems unlikely that python would have the option of allowing a user-defined __del__() function which cannot safely access member objects for fear of them already being deleted.

[warner]

No problem! Thanks for making such a great tool, it's been really convenient for us.

__del__() is in a funny space among python's "magic" methods. My information may be out of date, but a couple of years ago, in Foolscap, I remember seeing problems where the __del__ method got called inside nominally-indivisible operations (as a result of GC pressure triggering a collection cycle), and it could easily observe the containing object being in a funny indeterminate state. My rule at time was to treat __del__ like you would a C signal handler: the only safe thing to do in one is to arrange to wake up a different object: basically write a byte to a pre-opened pipe-to-self, so that some event loop would wake up (at a "good" place, not in the middle of some python GC cycle) and do whatever it is you wanted to do. For Foolscap, it was about a proxy being GCed: the proxy referenced some remote object, and when the last proxy goes away, the remote object should be GCed too. I was using weakrefs and their callbacks, but I think the timing behavior was the same as __del__, and in the end I made the callback set a timer, and let the Twisted reactor fire a function which sent the decrement-refcount message, rather than trying to do it from inside the weakref callback.

So I guess I'd recommend something conservative, and also looking carefully at the __del__ docs to see what sorts of guarantees they make or don't make. And maybe walk through the ways that errors might cause the object to be re-referenced or re-deleted: maybe errors inside the __del__, or something inside del which causes the refcount to be incremented again, then decremented again, possibly causing a reentrant call. I don't know for sure that this is what we're seeing.. if we can explain it purely in terms of an exception being thrown in the middle of __init__, then maybe all we need to do is fix that. But I did see several tqdm bugs filed about ".. has no attribute X" for various Xes, so maybe it's a more significant problem.

[casperdcl]

well that was a fun way to see how many people are using pyup-bot and tqdm...

I recall reading up about __del__() in detail but it's possible I misread or there are discrepancies between different python versions. Seems more like a python problem than a tqdm one - we need to use a destructor to clear up properly after a progressbar completes!

[warner]

You might consider having a separate object which implements the cleanup code, and have the main tqdm.__del__ invoke a method on it, rather than looking carefully at self. Or, have that other object create a weakref pointing at the tqdm object, and react to the weakref's callback. The cleanup object would need a second copy of some of the state (maybe just the file descriptor? but maybe also the "total size" data, to figure out what to write in that last line), but it wouldn't be in the middle of being destroyed when it accesses that data. Or maybe just have the outer tqdm object have no state: everything lives on some inner object, all method calls get passed through. Only the application code references the outer object, so a weakref on it (held by the inner object) can discover when it gets dropped, and then can clean up without racing with object destruction.

Anyways, for this ticket, I think the fix will be to bump our dependency to the current version (4.14.0). I'll make a patch for that in a minute. Thanks to everyone for digging into this one!