Test cPickle with real files

[serhiy-storchaka]

BPO	17299
Nosy	@pitrou, @avassalotti, @benjaminp, @bitdancer, @serhiy-storchaka
Files	patch patch test_cpickle_patch: Updated to accomodate corrections in review issue17299v3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-03-18.11:24:27.797>
created_at = <Date 2013-02-26.08:39:39.279>
labels = ['type-feature', 'tests']
title = 'Test cPickle with real files'
updated_at = <Date 2013-03-18.11:24:27.796>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2013-03-18.11:24:27.796>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-03-18.11:24:27.797>
closer = 'serhiy.storchaka'
components = ['Tests']
creation = <Date 2013-02-26.08:39:39.279>
creator = 'serhiy.storchaka'
dependencies = []
files = ['29266', '29267', '29356', '29383']
hgrepos = []
issue_num = 17299
keywords = ['patch']
message_count = 19.0
messages = ['183028', '183143', '183158', '183166', '183238', '183243', '183435', '183436', '183813', '183958', '183990', '183991', '184179', '184181', '184359', '184400', '184402', '184442', '184447']
nosy_count = 8.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'benjamin.peterson', 'Arfrever', 'r.david.murray', 'python-dev', 'serhiy.storchaka', 'Aman.Shah']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17299'
versions = ['Python 2.7']

[serhiy-storchaka]

Currently cPickle module tested only with cStringIO.StringIO. However cPickle uses different code for cStringIO.StringIO, for file objects, and for general IO streams (i.e. io.BytesIO). Last two cases are not covered by tests.

[bitdancer]

Serhiy, in Python3 the corresponding test uses io.BytesIO. That means the additional tests are needed on Python3 as well, just a slightly different set, right?

[AmanShah]

Created a small patch for python 2.7 using file test_pickle.py .

[AmanShah]

Fixed the patch by removing TESTFN from tearDown.

[bitdancer]

Aman: another nit: PEP-8 calls for no unneeded parentheses around logical expressions, so things like:

if(self.f):

should be written:

   if self.f:

in order to adhere to our coding style.

Also, it's not obvious to me that there is any reason to rename the base classes (PicklerTests->AbstractIOPicklerTests, and same for Persistent test).

Finally, rereading Serhiy's note, Python3 only has "general IO streams", both non-IO files and cStringIO.StringIO are gone. So this really is a 2.7-only issue, and only worth doing because 2.7 is a long term maintenance release.

[AmanShah]

I had put in the renaming because "ioclass" is not defined in the class itself, only in the subclasses. So, instantiating it and using dumps/loads would be meaningless and would raise an error. Which is similar to the behavior of an abstract class.
Also, the "if(self.f):" part can be improved as you said and I looked at the patch for other similar mistakes. But, it's an isolated mistake (at lines 33 and 74) and so maybe it can be fixed during the merge itself?

[serhiy-storchaka]

I have added comments on Rietveld.

Perhaps it will be worth to create mixings for cStringIO.StringIO, BytesIO and file object and then mix them to other tests.

Note that there is no sense to change pure Python pickle tests. Python implementation uses the same code for all streams. It's C implementation needs specialized tests (in Lib/test/test_cpickle.py).

[serhiy-storchaka]

David, yes, this is 2.7 only issue. The code was broken recently (see msg182979 in bpo-13555) due to insufficient testing.

[AmanShah]

I have updated the patch for test_cpickle.py . Also, I would like to help out in creating mixins for the 3 but, it would be helpful if you can explain it in a bit more detail. What is the problem in using the existing pickletester.py??

[serhiy-storchaka]

DRY (do not repeat yourself). Do not implement the same method three times. Definitely the common code should be extracted into separate mixin classes: cStringIOMixin, BytesIOMixin, FileIOMixin.

What is the problem in using the existing pickletester.py??

This is just unnecessary. This is cPickle-only issue.

Aman, can you please submit a contributor form?

http://python.org/psf/contrib/contrib-form/
http://python.org/psf/contrib/

[AmanShah]

Version 3.

[serhiy-storchaka]

LGTM. You forgot to remove close() from some classes, I'll do it myself before committing.

[python-dev]

New changeset 8a0b5c9f04c2 by Serhiy Storchaka in branch '2.7':
Issue bpo-17299: Add test coverage for cPickle with file objects and general IO
http://hg.python.org/cpython/rev/8a0b5c9f04c2

[serhiy-storchaka]

I'm a little polished the patch before committing. Thank you for the patch, Aman Shah.

[benjaminp]

I think this broke the 2.7 Windows bots. Please unbreak.

[serhiy-storchaka]

I'm not sure what is wrong and can't check on Windows, but it is possible that this patch fixes tests. Please check it if you can.

[pitrou]

I think you want to open the files in binary mode, not text mode.

[serhiy-storchaka]

Oh, yes.

[serhiy-storchaka]

Benjamin has fixed this in the changeset 6aab72424063.