The copy_reg module becomes unexpectedly empty in test_cpickle

[doko42]

BPO	25698
Nosy	@warsaw, @doko42, @benjaminp, @vadmium, @serhiy-storchaka
Files	issue25698_skip_tests.patch issue25698_fix_remove_module.patch issue25698_fix_add_module.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-02-10.08:36:16.709>
created_at = <Date 2015-11-22.18:58:26.441>
labels = ['extension-modules', 'type-bug']
title = 'The copy_reg module becomes unexpectedly empty in test_cpickle'
updated_at = <Date 2017-08-01.06:09:54.294>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2017-08-01.06:09:54.294>
actor = 'Soujanya. Pasam'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-02-10.08:36:16.709>
closer = 'serhiy.storchaka'
components = ['Extension Modules']
creation = <Date 2015-11-22.18:58:26.441>
creator = 'doko'
dependencies = []
files = ['41748', '41750', '41846']
hgrepos = []
issue_num = 25698
keywords = ['patch']
message_count = 38.0
messages = ['255109', '255114', '255118', '255119', '255120', '255138', '255626', '255627', '255628', '255629', '255630', '255631', '255632', '255646', '255660', '255661', '255662', '255667', '255668', '255671', '255672', '255673', '255674', '255929', '255937', '255954', '255967', '255968', '259198', '259200', '259807', '259821', '259834', '259979', '259991', '259992', '260015', '299603']
nosy_count = 7.0
nosy_names = ['barry', 'doko', 'benjamin.peterson', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'Soujanya. Pasam']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25698'
versions = ['Python 2.7']

[doko42]

seen with the 2.7.11 release candidate; I'm not yet 100% if this is the fix for bpo-22995 causing these test regressions.

[ 93/395] test_cpickle
test test_cpickle failed -- multiple errors occurred; run in verbose mode for details
[101/395/2] test_decimal
test test_decimal failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_decimal.py", line 1774, in test_pickle
    e = pickle.loads(pickle.dumps(c, proto))
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 306, in save
    rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[108/395/4] test_dictviews
test test_dictviews failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_dictviews.py", line 201, in test_pickle
    pickle.dumps, d.viewkeys(), proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/unittest/case.py", line 473, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 306, in save
    rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[177/395/7] test_imp
test test_imp failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_imp.py", line 60, in test_source
    imp.reload(os)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/os.py", line 727, in <module>
    _copy_reg.pickle(stat_result, _pickle_stat_result, _make_stat_result)
AttributeError: 'module' object has no attribute 'pickle'

[184/395/10] test_int_literal
[185/395/10] test_ioctl
test test_ioctl crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[186/395/11] test_isinstance
[187/395/11] test_iter
[188/395/11] test_iterlen
[189/395/11] test_itertools
[190/395/11] test_json
test test_json crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[191/395/12] test_kqueue
test_kqueue skipped -- test works only on BSD
[192/395/12] test_largefile
test test_largefile crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[193/395/13] test_lib2to3
test test_lib2to3 crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[194/395/14] test_linecache
test test_linecache crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[195/395/15] test_list
test test_list crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[196/395/16] test_locale
[197/395/16] test_logging
test test_logging crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[198/395/17] test_long
[199/395/17] test_long_future
[200/395/17] test_longexp
[201/395/17] test_macos
test test_macos crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[202/395/18] test_macostools
test test_macostools crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[203/395/19] test_macpath
[204/395/19] test_macurl2path
[205/395/19] test_mailbox
test test_mailbox crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[206/395/20] test_marshal
test test_marshal crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[207/395/21] test_math
test test_math crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[208/395/22] test_md5
[209/395/22] test_memoryio
test test_memoryio failed -- multiple errors occurred; run in verbose mode for details
[210/395/23] test_memoryview
[211/395/23] test_mhlib
test test_mhlib crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'

[vadmium]

Duplicate or related to bpo-25601?

[serhiy-storchaka]

Yes, it is a duplicate of bpo-25601. And I can't reproduce it locally.

Matthias, can you reproduce test failure every time when run tests? If yes, could you please run tests in verbose mode and found what tests in test_cpickle were failed first?

[doko42]

looks like re-running the tests in verbose mode using -w is affected by this issue.

the first cpickle subtest failing is:

======================================================================
ERROR: test_recursive_dict_subclass_and_inst (test.test_cpickle.cStringIOCPicklerFastTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 175, in wrapper
    func(self)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 819, in test_recursive_dict_subclass_a
nd_inst
    self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 790, in check_recursive_collection_and
_inst
    s = self.dumps(y, proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 141, in dumps
    p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

[serhiy-storchaka]

In your report test_cpickle has number 93. What other 92 tests were ran before? Could you run tests few times and find minimal list of tests that precede failing test_cpickle?

This failure is not related to bpo-22995 because the same failure was reported in bpo-25601 5 commits before first commit in bpo-22995.

[serhiy-storchaka]

This can be related to bpo-25083. Can you look at the size of copy_reg.pyc and copy_reg.pyo files?

[warsaw]

This seems entirely reproducible in a local Xenial sbuild using the source packages in https://launchpad.net/~doko/+archive/ubuntu/toolchain/+packages

Interestingly, inspection of the built artifacts in the schroot after all is done yields:

# find . -name copy_reg.py? | xargs ls -l
-rw-rw-r-- 1 barry barry 5597 Nov 30 16:52 ./Lib/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5553 Nov 30 16:53 ./Lib/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:03 ./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:03 ./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:02 ./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:02 ./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyo

Those are debug builds, so it seems reasonable they are different sizes than the non-debug artifacts in Lib, right? And since they are self-consistent in size, it seems like it wouldn't be caused by a pyc/pyo write bug.

[warsaw]

First failure in test run afaict:

======================================================================
ERROR: test_recursive_dict_subclass_and_inst (test.test_cpickle.cStringIOCPicklerFastTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 175, in wrapper
    func(self)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 819, in test_recursive_dict_subclass_and_inst
    self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 790, in check_recursive_collection_and_inst
    s = self.dumps(y, proto)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 141, in dumps
    p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

======================================================================

[warsaw]

But if I just run test_cpickle.py, it succeeds!

[serhiy-storchaka]

Could you please add debugging output to check that the copy_reg module is empty if tests failed?

[warsaw]

I set a breakpoint where the error occurs. I'm in test_cpickle.py dumps(). Doing cPickle.dumps(arg) in the bp, I get the error. arg is:

(Pdb) arg
{<main.H object at 0x7ff8e2a870d0>: None}
(Pdb) type(arg)
<class 'test.pickletester.MyDict'>

Look again at the AttributeError:

(Pdb) cPickle.dumps(arg)
*** AttributeError: 'module' object has no attribute '_reduce_ex'

First, why is that trying to find the attribute of a module object instead of the MyDict instance? I even tried this just to discount the effects of __main__.H:

(Pdb) cPickle.dumps(type(arg)())
*** AttributeError: 'module' object has no attribute '_reduce_ex'

(as you'd expect, type(arg)() is just an instance of a MyDict.

Two questions: why is it trying to get the attribute on a module object, and why is the attribute it's trying to get '_reduce_ex' instead of '__reduce_ex__'?

The attribute to get is taken from the __reduce_ex___str in cPickle.c, which gets defined as:

#define INIT_STR(S) if (!( S ## _str=PyString_InternFromString(#S)))  return -1;
...
    INIT_STR(__reduce_ex__);

This is a compile-time definition, so it makes no sense that it would be trying to find the wrong attribute in some cases (e.g. when running the full regrtest.py) but not in others (e.g. running just test_cpickle.py). But I can find no other references to a string containing "reduce_ex" so the cPickle version has to be it. And that gets initialized in init_stuff(), called from initcPickle() so I don't see any other way for it to get corrupted.

cPickle.dumps({}) works just fine, so it's an artifact of the MyDict. That's all I've figured out for now.

[serhiy-storchaka]

Default __reduce_ex__ and __reduce__ implementations call copy_reg._reduce_ex. cPickle.dumps({}) works because __reduce_ex__ is not called, dict is supported by pickle directly.

[serhiy-storchaka]

... for protocols < 2.