shelve.open/bsddb.hashopen exception with unicode paths

[wjm251]

BPO	9393
Nosy	@jcea, @vstinner, @bitdancer, @serhiy-storchaka
Files	dbm_open_unicode-27.patch dbm_open_unicode-32.patch bsddb_unicode_filename-27.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-07-28.09:10:41.547>
labels = ['type-bug', 'library']
title = 'shelve.open/bsddb.hashopen exception with unicode paths'
updated_at = <Date 2015-01-18.17:03:35.417>
user = 'https://bugs.python.org/wjm251'

bugs.python.org fields:

activity = <Date 2015-01-18.17:03:35.417>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-07-28.09:10:41.547>
creator = 'wjm251'
dependencies = []
files = ['18230', '18231', '18232']
hgrepos = []
issue_num = 9393
keywords = ['patch']
message_count = 14.0
messages = ['111779', '111795', '111797', '111801', '111809', '111812', '111814', '111817', '227673', '234236', '234247', '234249', '234250', '234260']
nosy_count = 5.0
nosy_names = ['jcea', 'vstinner', 'r.david.murray', 'wjm251', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9393'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[wjm251]

Windows XP Simple Chinese Version
in python2.5,Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] on win32
I have a directory "D:\你好新建文件夹"
my code is as follows:
#--------------------------------------

temppath = u"D:\\你好新建文件夹\\a"
import shelve
cache = shelve.open(temppath, 'c')
#

---

when use temppath.encode("utf-8"),it works,
but in python2.6,temppath can works properly

but I got a Error with such traceback

Traceback (most recent call last):
  File "D:\eclipse_workspace\pytest\src\test.py", line 5, in <module>
    cache = shelve.open(temppath, 'c')
  File "D:\eclipse_workspace\omstarv5r6\linksvn\src\UNPPython\pywindows\Lib\shelve.py", line 225, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "D:\eclipse_workspace\omstarv5r6\linksvn\src\UNPPython\pywindows\Lib\shelve.py", line 209, in __init__
    Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
  File "D:\eclipse_workspace\omstarv5r6\linksvn\src\UNPPython\pywindows\Lib\anydbm.py", line 83, in open
    return mod.open(file, flag, mode)
  File "D:\eclipse_workspace\omstarv5r6\linksvn\src\UNPPython\pywindows\Lib\dbhash.py", line 16, in open
    return bsddb.hashopen(file, flag, mode)
  File "D:\eclipse_workspace\omstarv5r6\linksvn\src\UNPPython\pywindows\Lib\bsddb\__init__.py", line 310, in hashopen
    d.open(file, db.DB_HASH, flags, mode)
bsddb.db.DBNoSuchFileError: (2, 'No such file or directory')

[wjm251]

I think it is supposed that unicode paths and GBK encoded str objects will be ok in Windows.
But only UTF-8 encoded str can

[vstinner]

shelve uses anydbm which uses gdbm, dbm or bsddbm. Attached patch fixes gdbm and dbm (replace "s" format by "et" with Py_FileSystemDefaultEncoding).

bsddbm is harder to fix: bsdhashopen(), bsdbtopen() and bsdrnopen() have to be fixed, and they accept None for the filename ("z" format).

[wjm251]

sorry I donot know exactly your meaning,
what does these mean: "s" format by "et " , "z" format
and I'm not familiar with the C/C++

do you mean that I can use the attached patch to complie a new Python dll?
but it seams that in My PC the shelve module always uses bsddbm automaticly,

can you explains more clearly?

thank you very much
sincerely

[vstinner]

Same patch for Python 3.2.

[bitdancer]

Victor's comments were addressed to the python development community and concern python internals. Given that only bsddb exists on windows by default, his patches unfortunately don't do you any good. I'm adding jcea as nosy in case he wants to/can deal with the problem in bsddb.

[vstinner]

It looks like bsddb (dbm.bsd) module doesn't exist anymore in Python3: see issue bpo-9397. It's now maintained in the third party module pybsddb.

[vstinner]

New patch for bsddb module: create parse_filename() function, based on Python3 PyUnicode_FSConverter() but it accepts None. I didn't tested the patch because I'm unable to compile the module. It looks like it should use db_185.h instead of db.h, and link to another library, but configure or setup.py doesn't know it.

[serhiy-storchaka]

dbm_open_unicode-32.patch no longer applied cleanly due to Argument Clinic.

I'm not sure about applying patches to 2.7. I support this, but it looks as new feature, and you should ask on Python-Dev mailing list.

[serhiy-storchaka]

Could you please update the patch Victor?

[vstinner]

Could you please update the patch Victor?

You can update this old patch if you want.

[vstinner]

Python 3 is not affected:

Python 3.5.0a0 (default:61a045ac0006, Jan 15 2015, 00:05:43) 
[GCC 4.9.2 20141101 (Red Hat 4.9.2-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> temppath = u"D:\\你好新建文件夹\\a"
>>> import shelve
>>> cache = shelve.open(temppath, 'c')

(no error)

[vstinner]

Python 3 is not affected:

Oh sorry, dbm_open_unicode-32.patch is still needed. Currently, filenames are encoded to UTF-8 which "works" when the filesystem encoding is UTF-8, but it doesn't work on Windows.

[serhiy-storchaka]

And needed tests.

[serhiy-storchaka]

It was fixed in bpo-32922.