dumbdbm __del__ bug

[janeaustine50]

BPO	702775
Nosy	@tim-one, @brettcannon, @rhettinger

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tim-one'
closed_at = <Date 2003-07-14.15:15:45.000>
created_at = <Date 2003-03-13.06:27:19.000>
labels = ['library']
title = 'dumbdbm __del__ bug'
updated_at = <Date 2003-07-14.15:15:45.000>
user = 'https://bugs.python.org/janeaustine50'

bugs.python.org fields:

activity = <Date 2003-07-14.15:15:45.000>
actor = 'tim.peters'
assignee = 'tim.peters'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-03-13.06:27:19.000>
creator = 'janeaustine50'
dependencies = []
files = []
hgrepos = []
issue_num = 702775
keywords = []
message_count = 10.0
messages = ['15107', '15108', '15109', '15110', '15111', '15112', '15113', '15114', '15115', '15116']
nosy_count = 6.0
nosy_names = ['tim.peters', 'nnorwitz', 'brett.cannon', 'rhettinger', 'juneaftn', 'janeaustine50']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue702775'
versions = ['Python 2.3']

[janeaustine50]

I used shelve.py and it falls back on dumbdbm when no
possible alternatives are found on the system.

I found this error, which is recurrent and deterministic:

Exception exceptions.AttributeError: "'NoneType' object
has no
attribute 'error'" in <bound method _Database.__del__ of
<dumbdbm._Database instance at 0x820c71c>> ignored

The problem seems to reside in the __del__ of
dumbdbm._Database:

class _Database:
...
    def __del__(self):
        if self._index is not None:
            self._commit()
...
    def _commit(self):
        try: _os.unlink(self._bakfile)
        except _os.error: pass
        try: _os.rename(self._dirfile, self._bakfile)
        except _os.error: pass
        f = _open(self._dirfile, 'w', self._mode)
        for key, (pos, siz) in self._index.items():
            f.write("%s, (%s, %s)\n" % (`key`, `pos`, `siz`))
        f.close()

My investigation showed that the error was from
_commit. When
it was called, _os or _open was both None. And the
exception
catch didn't work quite safely cause its in the "except"
clause.

The reason I suspect is when the time that
_Database.__del__ was
called the os module(which is imported as _os) is
already removed out.

I changed the code as:

    def _commit(self):
        global _os
        if _os is None:
            import os as _os
            import __builtin__
            _open = __builtin__.open
        try: _os.unlink(self._bakfile)
        except _os.error: pass
        try: _os.rename(self._dirfile, self._bakfile)
        except _os.error: pass
        ......

Now it works without any problems, AFAIK.

[juneaftn]

Logged In: YES
user_id=116941

see the thread at http://groups.google.com/groups?
selm=ba1e306f.0303111337.72a696c7%
40posting.google.com , esp. by my name.

[nnorwitz]

Logged In: YES
user_id=33168

Can you provide a test case which triggers this problem? I
can't see how _os becomes None. Also I would like to add a
test. Thanks.

[janeaustine50]

Logged In: YES
user_id=732903

A test case that triggers this problem:

#foobar.py
def open(filename, flag='c'):
    import dumbdbm
    return dumbdbm.open(filename,flag)

c=open('test.dbm')

#main.py
import foobar

$ python main.py
>>> 
Exception exceptions.TypeError: "'NoneType' object is not 
callable" in <bound method _Database.__del__ of 
<dumbdbm._Database instance at 0x401e482c>> ignored

[janeaustine50]

Logged In: YES
user_id=732903

Run the main.py in Python 2.3+ putting the two files in the
same place.

For Python 2.2+, put the two files in the same package and
make a new file that imports main.py and run it as follows:

=======
from <packagename> import main
=======

[janeaustine50]

Logged In: YES
user_id=732903

Tested on linux and windows xp with Python2.2.2 and
Python2.3a2.

[tim-one]

Logged In: YES
user_id=31435

Neal, this is actually a common problem in __del__
methods, and the OP's analysis is on target. When Python
is shutting down, it tries to tear down module dicts in a
safe-as-possible order, but modules are full of reference
cycles and there is no *wholly* safe order.

In general, a __del__ method should never reference
globals because of this. The usual solution can be seen in
tempfile.py: store the global objects __del__ will need as
class attributes when the class is created, and refer to
these bindings in __del__ via self.attrname
(ClassName.attrname is also no good, because it refers to
the global ClassName! that may also become None).

Reimporting a module instead may not be effective (if it's in
a partially torn-doiwn state but its name is still a key in
sys.modules, importing again will just retrieve the partially
torn-down module object). The class-attr trick is robust.

[brettcannon]

Logged In: YES
user_id=357491

OK, so according to Tim all the accessing of _os should be
removed and instead call instance or class attributes that are
storing what del is going to need when it is cleaning up by
calling them off of self since the namespace might be in the
process of being cleared. Right? So
_os.unlink(self._bakfile) should be something more like
self._delattrs['unlink'](self._bakfile) where self._delattrs
stores everything del is going to need. Or should it just
store a reference to the os module at self._os since that might be
a little cleaner looking?

[rhettinger]

Logged In: YES
user_id=80475

Tim, is this what you just fixed?

[tim-one]

Logged In: YES
user_id=31435

Raymond, yes, and thanks for the reminder. This is fixed in
current CVS, via the approach I sketched in an comment on
this report.