gdbm reorganize() leaves hanging file descriptor

[themoken]

BPO	13947
Nosy	@jcea, @sam-s

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-02-06.04:48:24.064>
created_at = <Date 2012-02-06.04:13:35.479>
labels = ['type-bug', 'library']
title = 'gdbm reorganize() leaves hanging file descriptor'
updated_at = <Date 2019-04-08.19:39:39.826>
user = 'https://bugs.python.org/themoken'

bugs.python.org fields:

activity = <Date 2019-04-08.19:39:39.826>
actor = 'sam-s'
assignee = 'none'
closed = True
closed_date = <Date 2012-02-06.04:48:24.064>
closer = 'jcea'
components = ['Library (Lib)']
creation = <Date 2012-02-06.04:13:35.479>
creator = 'themoken'
dependencies = []
files = []
hgrepos = []
issue_num = 13947
keywords = []
message_count = 7.0
messages = ['152719', '152720', '152721', '152722', '153050', '339398', '339668']
nosy_count = 3.0
nosy_names = ['jcea', 'themoken', 'sam-s']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13947'
versions = ['Python 3.6']

[themoken]

I've discovered that using either 2.7.2 or 3.2.2 (from Arch) reorganizing gdbm databases leave hanging file descriptors that will cause EAGAIN errors trying to reopen it until the process is killed. For example:

Python 2.7.2 (default, Jan 31 2012, 13:19:49) 
[GCC 4.6.2 20120120 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdbm
>>> o = gdbm.open("test27", "c")
>>> o.reorganize()
>>> o.close()
>>> o = gdbm.open("test27", "w")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
gdbm.error: (11, 'Resource temporarily unavailable')
>>>

By using lsof you can see that after reorganize() test27 has two open file descriptors, and after close() one is still open with no (obvious) way to close it. This happens with freshly created and populated databases and regardless of open mode flags.

I also tested this on 3.2.2 with dbm.gnu and got identical behavior.

[jcea]

I can not reproduce the issue with GDBM 1.9.1, neither GDBM 1.8.3, neither GDBM 1.10, under Solaris.

I don't see the file descriptor leak either.

According to http://sunsite.ualberta.ca/Documentation/Gnu/gdbm-1.8.0/html_node/gdbm_10.html, all file descriptor management is done inside GDBM.

What GDBM are you using?. Could you possibly upgrade?. Beware, 1.8 and 1.9/1.10 fileformats are *NOT* compatible.

[jcea]

Can't reproduce either in Ubuntu 10.04, 64 bits, with GDBM 1.8.3.

Closing bug.

Reporter, please indicate your GDBM version. Can you possibly upgrade it?.

[themoken]

I am using gdbm 1.10, the latest available.

pacman says:
core/gdbm 1.10-1 [installed]
GNU database library

Which also contains the file (libgdbm.so.4) that Python's gdbm.so is linked to (just in case there was some version trickery going on here).

I guess it's possible this is Arch brokenness, but the only patch applied to the 1.10 code in the PKGBUILD is malloc -> calloc for a known bug.

[jcea]

I can't reproduce. Sorry.

Reading the GDBM documentation, all filedescriptors management seem to be inside gdbm, not Python. Anyway, if I can't reproduce, I can not diagnose neither patch. Sorry.

[sam-s]

I observe the same behavior on Linux 4.18.0-17-generic #18-Ubuntu
with Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0].
gdbm 1.14.1-6:
after

db = dbm.gnu.open("asdf","n")
db["a"] = "124"
db.reorganize()
db.close()

I see

$ lsof ~/asdf 
COMMAND   PID USER  FD   TYPE DEVICE SIZE/OFF    NODE NAME
python3 22071  sds mem-W  REG    8,2    16384 4328326 /home/sds/asdf

and

>>> db = dbm.gnu.open("asdf","r")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sds/src/pyapp/tests/fileconv_test.py", line 1053, in <module>
_gdbm.error: [Errno 11] Resource temporarily unavailable

[sam-s]

Upstream claims that this is fixed by http://git.gnu.org.ua/cgit/gdbm.git/commit/?id=a0d6c1a8