Bug in dbm - long strings in keys and values

[jdcrunchman]

BPO	827760
Nosy	@loewis

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2006-07-03.12:48:16.000>
created_at = <Date 2003-10-21.20:05:57.000>
labels = ['library']
title = 'Bug in dbm - long strings in keys and values'
updated_at = <Date 2006-07-03.12:48:16.000>
user = 'https://bugs.python.org/jdcrunchman'

bugs.python.org fields:

activity = <Date 2006-07-03.12:48:16.000>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-10-21.20:05:57.000>
creator = 'jdcrunchman'
dependencies = []
files = []
hgrepos = []
issue_num = 827760
keywords = []
message_count = 4.0
messages = ['18706', '18707', '18708', '18709']
nosy_count = 3.0
nosy_names = ['loewis', 'jdcrunchman', 'tinolange']
pr_nums = []
priority = 'high'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue827760'
versions = []

[jdcrunchman]

#!/usr/local/bin/python
python/issues-test-cpython#2003-10-19. Feedback to crunch@shopip.com
import dbm

print """
Python dbm bugs summary:

1. Long strings cause weirdness.
2. Long keys fail without returning error.

This demonstrates serious bugs in the Python dbm
module.
Present in OpenBSD versions 2.2, 2.3, and 2.3.2c1.

len(key+string)>61231 results in the item being 'lost',
without warning.
If the key or string is one character shorter, it is fine.
Writing multiple long strings causes unpredictable results
(none, some, or all of the items are lost without warning).

Curiously, keys of length 57148 return an error, but
longer keys fail without warning
(sounds like an = instead of a > somewhere).
"""

mdb=dbm.open("mdb","n")
print "Writing 1 item to database, but upon reading,"
k='k'
v='X'*61230 #Long string
mdb[k]=v
mdb.close()

md=dbm.open("mdb","r")
print "database contains %i items"%len(md.keys())
md.close()

[jdcrunchman]

Logged In: YES
user_id=891874

I upped the priority since this bug causes dbm database
corruption, which could be disastrous for users of dbm or
shelve.

[tinolange]

Logged In: YES
user_id=212920

Hi!

This is no bug - at least no python bug.
It seems to be a limitation in the underlying dbm engine.

I tested it with different dbm implementations under Linux
today. You are right - when using dbm as ndbm.h (which
belongs to libc6, i.e. db1/ndbm.h) this error/limitation occurs.
Please test: At least for me the dbm engine still finds the
data if you specify it, a print md[k] shows the long v
string. Only the keys() and len() methods which iterate over
the database fail.

I also tested without Python, i. e. just using the C-API and
it's just the same (linking against libdb1.so). It is the
underlying dbm_firstkey() which fails when using that HUGE
keys. It just returns NULL.

Context:
--------

drec=dbm_firstkey(dbm);
while(drec.dptr) {
  size++;
  drec = dbm_nextkey(dbm);
}

If you happen to have a system that uses gdbm/ndbm.h as
dbm.h (for example SuSE) or if you change the links in your
include and lib directories accordingly and recompile Python
then this bug disappears and your program is working as
expected. The GNU gdbm engine (I tested 1.73 and 1.8.3)
seems to be much better in handling huge keys as the old
libc dbm.

If possible change to gdbm or assure that dbm is linked
against gdbm (-compat). setup.py tries to be smart about
that but sometimes chooses the old dbm even if it could use
gdbm instead.

ldd /usr/local/lib/python2.3/lib-dynload/dbm.so

shows you what lib you're using.

On my Debian system an
ln -sf /usr/include/gdbm/ndbm.h /usr/include/ndbm.h
ln -sf /usr/lib/libgdbm.so /usr/lib/libndbm.so
fixed it.

Ciao

Tino

[loewis]

Logged In: YES
user_id=21627

I'm closing this as a third-party bug. If there is any
evidence that there is something that Python could do to
reduce the impact of this problem, please submit this as a
new bug report.