intermitent failures with test_dbm

[ethanfurman]

BPO	20094
Nosy	@jcea, @ethanfurman

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-04-30.18:37:59.197>
created_at = <Date 2013-12-29.16:46:38.291>
labels = []
title = 'intermitent failures with test_dbm'
updated_at = <Date 2017-01-09.23:48:20.071>
user = 'https://github.com/ethanfurman'

bugs.python.org fields:

activity = <Date 2017-01-09.23:48:20.071>
actor = 'asottile'
assignee = 'none'
closed = True
closed_date = <Date 2014-04-30.18:37:59.197>
closer = 'ethan.furman'
components = []
creation = <Date 2013-12-29.16:46:38.291>
creator = 'ethan.furman'
dependencies = []
files = []
hgrepos = []
issue_num = 20094
keywords = []
message_count = 7.0
messages = ['207079', '207080', '217630', '217639', '285081', '285082', '285083']
nosy_count = 3.0
nosy_names = ['jcea', 'ethan.furman', 'asottile']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue20094'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[ethanfurman]

Following errors occur about half the time:

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 75, in test_anydbm_creation
    self.read_helper(f)
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 117, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 90, in test_anydbm_modification
    self.read_helper(f)
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 117, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 96, in test_anydbm_read
    self.read_helper(f)
  File "/home/ethan/source/python/issue19995/Lib/test/test_dbm.py", line 117, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

[ethanfurman]

Actually, make that about 1/5 of the time.

[jcea]

I can't reproduce on Linux 12.04. I tried the test a thousand times.

Ethan, what is your build environment?

[ethanfurman]

Actually, I haven't had this issue in quite a while now, so closing.

Thanks for taking a look at it, Jesús.

[asottile]

I'm seeing this same failure in python3.5 on 16.04 about 20% of the time:

$ python3.5 -m test -v test_dbm
== CPython 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]
==   Linux-4.4.0-57-generic-x86_64-with-Ubuntu-16.04-xenial little-endian
==   hash algorithm: siphash24 64bit
==   /tmp/test_python_8169
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[1/1] test_dbm
test_keys (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb_ndbm (test.test_dbm.WhichDBTestCase) ... BDB0004 fop_read_meta: @test_8169_tmp_ndbm.db: unexpected file type or format
ok
test_anydbm_access (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.gnu) ... ok
test_error (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8169_tmp.db: unable to flush: No such file or directory
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8169_tmp.db: unable to flush: No such file or directory
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
test_error (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.dumb) ... ok
test_error (test.test_dbm.TestCase-dbm.dumb) ... ok

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/test/test_dbm.py", line 74, in test_anydbm_creation
    self.read_helper(f)
  File "/usr/lib/python3.5/test/test_dbm.py", line 115, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/test/test_dbm.py", line 89, in test_anydbm_modification
    self.read_helper(f)
  File "/usr/lib/python3.5/test/test_dbm.py", line 115, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/test/test_dbm.py", line 95, in test_anydbm_read
    self.read_helper(f)
  File "/usr/lib/python3.5/test/test_dbm.py", line 115, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

----------------------------------------------------------------------
Ran 27 tests in 0.093s

FAILED (errors=3)
test test_dbm failed
1 test failed:
    test_dbm

Under 3.6 (built from source), it seems to fail 100% of the time

$ ./python -m test test_dbm -v
== CPython 3.6.0 (default, Jan 9 2017, 15:00:38) [GCC 5.4.0 20160609]
==   Linux-4.4.0-57-generic-x86_64-with-debian-stretch-sid little-endian
==   hash algorithm: siphash24 64bit
==  cwd: /home/asottile/workspace/cpython/build/test_python_8246
==  encodings: locale=UTF-8, FS=utf-8
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_dbm
test_keys (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb_ndbm (test.test_dbm.WhichDBTestCase) ... BDB0004 fop_read_meta: @test_8246_tmp_ndbm.db: unexpected file type or format
ok
test_anydbm_access (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.gnu) ... ok
test_error (test.test_dbm.TestCase-dbm.gnu) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8246_tmp.db: unable to flush: No such file or directory
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_8246_tmp.db: unable to flush: No such file or directory
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
test_error (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.dumb) ... ok
test_error (test.test_dbm.TestCase-dbm.dumb) ... ok

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 73, in test_anydbm_creation
    self.read_helper(f)
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 88, in test_anydbm_modification
    self.read_helper(f)
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 94, in test_anydbm_read
    self.read_helper(f)
  File "/home/asottile/workspace/cpython/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

----------------------------------------------------------------------
Ran 27 tests in 0.084s

FAILED (errors=3)
test test_dbm failed
test_dbm failed

1 test failed:
    test_dbm

Total duration: 112 ms
Tests result: FAILURE

[asottile]

Stepping through the code, it seems under ndbm it is creating a file with a '.db' extension:

(Pdb) list
 47  	             'g': b'intended',
 48  	             }
 49  	
 50  	    def init_db(self):
 51  	        import pdb; pdb.set_trace()
 52  ->	        with dbm.open(_fname, 'n') as f:
 53  	            for k in self._dict:
 54  	                f[k.encode("ascii")] = self._dict[k]
 55  	
 56  	    def keys_helper(self, f):
 57  	        keys = sorted(k.decode("ascii") for k in f.keys())
(Pdb) n
> /home/asottile/workspace/cpython/Lib/test/test_dbm.py(53)init_db()
-> for k in self._dict:
(Pdb) os.listdir('.')
['@test_8371_tmp.db']
(Pdb) _fname
'@test_8371_tmp'

[asottile]

That doesn't seem to be the problem though, that occurs in both the successful and failure case