private dispatch table for picklers

[sbt]

BPO	14166
Nosy	@pitrou, @avassalotti
Files	pickle_dispatch.patch pickle_dispatch.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-03-04.17:39:05.160>
created_at = <Date 2012-03-01.14:34:09.532>
labels = ['type-feature']
title = 'private dispatch table for picklers'
updated_at = <Date 2012-03-04.17:39:05.158>
user = 'https://bugs.python.org/sbt'

bugs.python.org fields:

activity = <Date 2012-03-04.17:39:05.158>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2012-03-04.17:39:05.160>
closer = 'pitrou'
components = []
creation = <Date 2012-03-01.14:34:09.532>
creator = 'sbt'
dependencies = []
files = ['24697', '24729']
hgrepos = []
issue_num = 14166
keywords = ['patch']
message_count = 6.0
messages = ['154695', '154847', '154893', '154897', '154901', '154902']
nosy_count = 4.0
nosy_names = ['pitrou', 'alexandre.vassalotti', 'python-dev', 'sbt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14166'
versions = ['Python 3.3']

[sbt]

Currently the only documented way to have customised pickling for a type is to register a reduction function with the global dispatch table managed by the copyreg module. But such global changes are liable to disrupt other code which uses pickling.

Multiprocessing deals with this by defining a ForkingPickler class which subclasses the pure python _Pickler class (using undocumented features), and supports registering reduction functions specifically for that class.

I would like to see some documented alternative which works with both C and Python implementations. At least then multiprocessing can avoid using slow pure python pickling.

The attached patch allows a pickler object to have a private dispatch table which it uses *instead* of the global one. It lets one write code like

    p = pickle.Pickler(...)
    p.dispatch_table = copyreg.dispatch_table.copy()
    p.dispatch_table[SomeClass] = reduce_SomeClass

or

    class MyPickler(pickle.Pickler):
        dispatch_table = copyreg.dispatch_table.copy()

MyPickler.dispatch_table[SomeClass] = reduce_SomeClass
p = MyPickler(...)

The equivalent using copyreg would be

    copyreg.pickle(SomeClass, reduce_SomeClass)
    p = pickle.Pickler(...)

[pitrou]

That looks like a good idea.
I don't understand the following code:

+ try:
+ self._dispatch_table = self.dispatch_table
+ except AttributeError:
+ self._dispatch_table = dispatch_table

... since self.dispatch_table is a property returning self._dispatch_table. Did you mean type(self).dispatch_table?

Also, you need to update the docs (which will also make the intended semantics of the patch clearer :-)).

[sbt]

I don't understand the following code:
...
since self.dispatch_table is a property returning
self._dispatch_table. Did you mean type(self).dispatch_table?

More or less. That code was a botched attempt to match the behaviour of the C implementation.

The C implementation does not expose the dispatch table unless it has been explicitly set (on the pickler or the pickler class), and it ignores any "dispatch_table" (or "persistent_id") attribute on the metaclass.

I will do a fixed patch with docs.

[sbt]

Updated patch with docs.

[python-dev]

New changeset f7a9a10ae0c0 by Antoine Pitrou in branch 'default':
Issue bpo-14166: Pickler objects now have an optional dispatch_table attribute which allows to set custom per-pickler reduction functions.
http://hg.python.org/cpython/rev/f7a9a10ae0c0

[pitrou]

I've replaced occurrences of "pickle.Pickler" with "self.pickler_class" in the tests. Otherwise, the patch looks perfect and I've now committed it. Thank you!