test_dbm failure: KeyError: b'0' (intermittent in 3.5, reliable in 3.6)

[gward]

BPO	28700
Nosy	@facundobatista, @arekm, @vadmium, @asottile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-11-15.19:27:18.765>
labels = ['type-bug', 'tests']
title = "test_dbm failure: KeyError: b'0' (intermittent in 3.5, reliable in 3.6)"
updated_at = <Date 2017-10-16.15:36:01.071>
user = 'https://bugs.python.org/gward'

bugs.python.org fields:

activity = <Date 2017-10-16.15:36:01.071>
actor = 'facundobatista'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2016-11-15.19:27:18.765>
creator = 'gward'
dependencies = []
files = []
hgrepos = []
issue_num = 28700
keywords = []
message_count = 6.0
messages = ['280877', '280879', '280880', '280888', '284138', '304476']
nosy_count = 5.0
nosy_names = ['gward', 'facundobatista', 'arekm', 'martin.panter', 'Anthony Sottile']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28700'
versions = ['Python 3.6']

[gward]

test_dbm.py fails reliably for me in 3.6, but in 3.5 it passes ~80% of the time. The failure in both cases is KeyError: b'0', which has come up previously in http://bugs.python.org/issue20094 and http://bugs.python.org/issue14120.

But since we've switched from 20% failure rate to 100% failure rate, I figured something must have changed. I used "hg bisect" to track it down to a recent commit:

changeset: 103360:0bd618fe0639
user: Victor Stinner <victor.stinner@gmail.com>
date: Wed Sep 07 17:40:12 2016 -0700
summary: Implement compact dict

Here is how it fails:

$ ./python -m test -v test_dbm 
== CPython 3.6.0a4+ (default:0bd618fe0639, Nov 15 2016, 14:07:07) [GCC 5.4.0 20160609]
==   Linux-4.4.0-47-generic-x86_64-with-debian-stretch-sid little-endian
==   hash algorithm: siphash24 64bit
==   /home/data/src/cpython/3.6/build/test_python_10093
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_dbm
test_keys (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb (test.test_dbm.WhichDBTestCase) ... ok
test_whichdb_ndbm (test.test_dbm.WhichDBTestCase) ... BDB0004 fop_read_meta: @test_10093_tmp_ndbm.db: unexpected file type or format
ok
test_anydbm_access (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_10093_tmp.db: unable to flush: No such file or directory
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
BDB3028 @test_10093_tmp.db: unable to flush: No such file or directory
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm) ... ERROR
test_error (test.test_dbm.TestCase-dbm.ndbm) ... ok
test_anydbm_access (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_keys (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_modification (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_not_existing (test.test_dbm.TestCase-dbm.dumb) ... ok
test_anydbm_read (test.test_dbm.TestCase-dbm.dumb) ... ok
test_error (test.test_dbm.TestCase-dbm.dumb) ... ok

======================================================================
ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 73, in test_anydbm_creation
    self.read_helper(f)
  File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 88, in test_anydbm_modification
    self.read_helper(f)
  File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

======================================================================
ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 94, in test_anydbm_read
    self.read_helper(f)
  File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 114, in read_helper
    self.assertEqual(self._dict[key], f[key.encode("ascii")])
KeyError: b'0'

---

Ran 19 tests in 0.052s

FAILED (errors=3)
test test_dbm failed
test_dbm failed

1 test failed:
test_dbm

Total duration: 77 ms
Tests result: FAILURE

[gward]

Forgot to mention: I'm running:

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

with

$ dpkg-query -W | grep dbm
libgdbm3:amd64	1.8.3-13.1

[gward]

As suggested in http://bugs.python.org/issue14120, I installed libgdbm-dev, re-configured, and re-compiled. That fixes the problem.

IMHO that's not good enough: if we're missing a dependency, then either configuring or building should fail. It's nice that the test failure is now rock-solid reliable rather than intermittent, but it's still a test failure due to missing dependency. Yuck.

[vadmium]

Is the problem something like a missing C function prototype? Maybe you see compiler warnings, but the compiler and linker carry on with the wrong prototype. If you build with “make -s”, warnings might be easier to see.

If my guess is right, this would be similar to bpo-27659, where a module half builds with warnings about a missing crypt() function prototype, although it later fails when linking. Maybe more configure or setup.py checks? (I’m not a fan of configure, but it often seems the easiest short-term solution.)

[arekm]

Using configure option

--with-dbmliborder=gdbm:bdb

(so no ndbm in it) is another reason for such failure as seems that test suite has no conditional for it.

[facundobatista]

I have this failure on my machine too (Ubuntu 17.04, kernel 4.10.0-37-generic).

Installing libgdbm-dev and running configure with --with-dbmliborder=gdbm:bdb didn't help.

[kumaraditya303]

Python 3.5 and 3.6 are not supported anymore. Feel free to create a new issue if this issue exists on supported python versions.