single: persistence pair: persistent; objects pair: serializing; objects pair: marshalling; objects pair: flattening; objects pair: pickling; objects
pickle
Jim Kerr <jbkerr@sr.hp.com>.
Barry Warsaw <barry@zope.com>
The pickle
module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. "Pickling" is the process whereby a Python object hierarchy is converted into a byte stream, and "unpickling" is the inverse operation, whereby a byte stream is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as "serialization", "marshalling,"1 or "flattening", however, to avoid confusion, the terms used here are "pickling" and "unpickling"..
Warning
The pickle
module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.
The pickle
module has an transparent optimizer (_pickle
) written in C. It is used whenever available. Otherwise the pure Python implementation is used.
Python has a more primitive serialization module called marshal
, but in general pickle
should always be the preferred way to serialize Python objects. marshal
exists primarily to support Python's .pyc
files.
The pickle
module differs from marshal
in several significant ways:
The
pickle
module keeps track of the objects it has already serialized, so that later references to the same object won't be serialized again.marshal
doesn't do this.This has implications both for recursive objects and object sharing. Recursive objects are objects that contain references to themselves. These are not handled by marshal, and in fact, attempting to marshal recursive objects will crash your Python interpreter. Object sharing happens when there are multiple references to the same object in different places in the object hierarchy being serialized.
pickle
stores such objects only once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects.marshal
cannot be used to serialize user-defined classes and their instances.pickle
can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.- The
marshal
serialization format is not guaranteed to be portable across Python versions. Because its primary job in life is to support.pyc
files, the Python implementers reserve the right to change the serialization format in non-backwards compatible ways should the need arise. Thepickle
serialization format is guaranteed to be backwards compatible across Python releases.
Note that serialization is a more primitive notion than persistence; although pickle
reads and writes file objects, it does not handle the issue of naming persistent objects, nor the (even more complicated) issue of concurrent access to persistent objects. The pickle
module can transform a complex object into a byte stream and it can transform the byte stream into an object with the same internal structure. Perhaps the most obvious thing to do with these byte streams is to write them onto a file, but it is also conceivable to send them across a network or store them in a database. The module shelve
provides a simple interface to pickle and unpickle objects on DBM-style database files.
single: External Data Representation
The data format used by pickle
is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can't represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.
By default, the pickle
data format uses a relatively compact binary representation. If you need optimal size characteristics, you can efficiently compress <archiving>
pickled data.
The module pickletools
contains tools for analyzing data streams generated by pickle
. pickletools
source code has extensive comments about opcodes used by pickle protocols.
There are currently 4 different protocols which can be used for pickling.
- Protocol version 0 is the original "human-readable" protocol and is backwards compatible with earlier versions of Python.
- Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.
- Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of
new-style class
es. Refer to307
for information about improvements brought by protocol 2. - Protocol version 3 was added in Python 3. It has explicit support for
bytes
objects and cannot be unpickled by Python 2.x. This is the default as well as the current recommended protocol; use it whenever possible.
To serialize an object hierarchy, you simply call the dumps
function. Similarly, to de-serialize a data stream, you call the loads
function. However, if you want more control over serialization and de-serialization, you can create a Pickler
or an Unpickler
object, respectively.
The pickle
module provides the following constants:
HIGHEST_PROTOCOL
The highest protocol version available. This value can be passed as a protocol value.
DEFAULT_PROTOCOL
The default protocol used for pickling. May be less than HIGHEST_PROTOCOL. Currently the default protocol is 3, a new protocol designed for Python 3.0.
The pickle
module provides the following functions to make the pickling process more convenient:
dump(obj, file, protocol=None, *, fix_imports=True)
Write a pickled representation of obj to the open file object
file. This is equivalent to Pickler(file, protocol).dump(obj)
.
The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3. The default protocol is 3; a backward-incompatible protocol designed for Python 3.0.
Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.
The file argument must have a write() method that accepts a single bytes argument. It can thus be an on-disk file opened for binary writing, a io.BytesIO
instance, or any other custom object that meets this interface.
If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x.
dumps(obj, protocol=None, *, fix_imports=True)
Return the pickled representation of the object as a bytes
object, instead of writing it to a file.
The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3. The default protocol is 3; a backward-incompatible protocol designed for Python 3.0.
Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.
If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x.
load(file, *, fix_imports=True, encoding="ASCII", errors="strict")
Read a pickled object representation from the open file object
file and return the reconstituted object hierarchy specified therein. This is equivalent to Unpickler(file).load()
.
The protocol version of the pickle is detected automatically, so no protocol argument is needed. Bytes past the pickled object's representation are ignored.
The argument file must have two methods, a read() method that takes an integer argument, and a readline() method that requires no arguments. Both methods should return bytes. Thus file can be an on-disk file opened for binary reading, a io.BytesIO
object, or any other custom object that meets this interface.
Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2.x. If fix_imports is True, pickle will try to map the old Python 2.x names to the new names used in Python 3.x. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2.x; these default to 'ASCII' and 'strict', respectively.
loads(bytes_object, *, fix_imports=True, encoding="ASCII", errors="strict")
Read a pickled object hierarchy from a bytes
object and return the reconstituted object hierarchy specified therein
The protocol version of the pickle is detected automatically, so no protocol argument is needed. Bytes past the pickled object's representation are ignored.
Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2.x. If fix_imports is True, pickle will try to map the old Python 2.x names to the new names used in Python 3.x. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2.x; these default to 'ASCII' and 'strict', respectively.
The pickle
module defines three exceptions:
PickleError
Common base class for the other pickling exceptions. It inherits Exception
.
PicklingError
Error raised when an unpicklable object is encountered by Pickler
. It inherits PickleError
.
Refer to pickle-picklable
to learn what kinds of objects can be pickled.
UnpicklingError
Error raised when there is a problem unpickling an object, such as a data corruption or a security violation. It inherits PickleError
.
Note that other exceptions may also be raised during unpickling, including (but not necessarily limited to) AttributeError, EOFError, ImportError, and IndexError.
The pickle
module exports two classes, Pickler
and Unpickler
:
This takes a binary file for writing a pickle data stream.
The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3. The default protocol is 3; a backward-incompatible protocol designed for Python 3.0.
Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.
The file argument must have a write() method that accepts a single bytes argument. It can thus be an on-disk file opened for binary writing, a io.BytesIO
instance, or any other custom object that meets this interface.
If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3.x names to the old module names used in Python 2.x, so that the pickle data stream is readable with Python 2.x.
dump(obj)
Write a pickled representation of obj to the open file object given in the constructor.
persistent_id(obj)
Do nothing by default. This exists so a subclass can override it.
If persistent_id
returns None
, obj is pickled as usual. Any other value causes Pickler
to emit the returned value as a persistent ID for obj. The meaning of this persistent ID should be defined by Unpickler.persistent_load
. Note that the value returned by persistent_id
cannot itself have a persistent ID.
See pickle-persistent
for details and examples of uses.
dispatch_table
A pickler object's dispatch table is a registry of reduction functions of the kind which can be declared using copyreg.pickle
. It is a mapping whose keys are classes and whose values are reduction functions. A reduction function takes a single argument of the associated class and should conform to the same interface as a ~object.__reduce__
method.
By default, a pickler object will not have a dispatch_table
attribute, and it will instead use the global dispatch table managed by the copyreg
module. However, to customize the pickling for a specific pickler object one can set the dispatch_table
attribute to a dict-like object. Alternatively, if a subclass of Pickler
has a dispatch_table
attribute then this will be used as the default dispatch table for instances of that class.
See pickle-dispatch
for usage examples.
3.3
fast
Deprecated. Enable fast mode if set to a true value. The fast mode disables the usage of memo, therefore speeding the pickling process by not generating superfluous PUT opcodes. It should not be used with self-referential objects, doing otherwise will cause Pickler
to recurse infinitely.
Use pickletools.optimize
if you need more compact pickles.
This takes a binary file for reading a pickle data stream.
The protocol version of the pickle is detected automatically, so no protocol argument is needed.
The argument file must have two methods, a read() method that takes an integer argument, and a readline() method that requires no arguments. Both methods should return bytes. Thus file can be an on-disk file object opened for binary reading, a io.BytesIO
object, or any other custom object that meets this interface.
Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2.x. If fix_imports is True, pickle will try to map the old Python 2.x names to the new names used in Python 3.x. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2.x; these default to 'ASCII' and 'strict', respectively.
load()
Read a pickled object representation from the open file object given in the constructor, and return the reconstituted object hierarchy specified therein. Bytes past the pickled object's representation are ignored.
persistent_load(pid)
Raise an UnpicklingError
by default.
If defined, persistent_load
should return the object specified by the persistent ID pid. If an invalid persistent ID is encountered, an UnpicklingError
should be raised.
See pickle-persistent
for details and examples of uses.
find_class(module, name)
Import module if necessary and return the object called name from it, where the module and name arguments are str
objects. Note, unlike its name suggests, find_class
is also used for finding functions.
Subclasses may override this to gain control over what type of objects and how they can be loaded, potentially reducing security risks. Refer to pickle-restrict
for details.
The following types can be pickled:
None
,True
, andFalse
- integers, floating point numbers, complex numbers
- strings, bytes, bytearrays
- tuples, lists, sets, and dictionaries containing only picklable objects
- functions defined at the top level of a module
- built-in functions defined at the top level of a module
- classes that are defined at the top level of a module
- instances of such classes whose
__dict__
or the result of calling__getstate__
is picklable (see sectionpickle-inst
for details).
Attempts to pickle unpicklable objects will raise the PicklingError
exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RuntimeError
will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit
.
Note that functions (built-in and user-defined) are pickled by "fully qualified" name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.2
Similarly, classes are pickled by named reference, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute attr
is not restored in the unpickling environment:
class Foo:
attr = 'A class attribute'
picklestring = pickle.dumps(Foo)
These restrictions are why picklable functions and classes must be defined in the top level of a module.
Similarly, when class instances are pickled, their class's code and data are not pickled along with them. Only the instance data are pickled. This is done on purpose, so you can fix bugs in a class or add methods to the class and still load objects that were created with an earlier version of the class. If you plan to have long-lived objects that will see many versions of a class, it may be worthwhile to put a version number in the objects so that suitable conversions can be made by the class's __setstate__
method.
In this section, we describe the general mechanisms available to you to define, customize, and control how class instances are pickled and unpickled.
In most cases, no additional code is needed to make instances picklable. By default, pickle will retrieve the class and the attributes of an instance via introspection. When a class instance is unpickled, its __init__
method is usually not invoked. The default behaviour first creates an uninitialized instance and then restores the saved attributes. The following code shows an implementation of this behaviour:
def save(obj):
return (obj.__class__, obj.__dict__)
def load(cls, attributes):
obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj
Classes can alter the default behaviour by providing one or several special methods:
object.__getnewargs__()
In protocol 2 and newer, classes that implements the __getnewargs__
method can dictate the values passed to the __new__
method upon unpickling. This is often needed for classes whose __new__
method requires arguments.
object.__getstate__()
Classes can further influence how their instances are pickled; if the class defines the method __getstate__
, it is called and the returned object is pickled as the contents for the instance, instead of the contents of the instance's dictionary. If the __getstate__
method is absent, the instance's __dict__
is pickled as usual.
object.__setstate__(state)
Upon unpickling, if the class defines __setstate__
, it is called with the unpickled state. In that case, there is no requirement for the state object to be a dictionary. Otherwise, the pickled state must be a dictionary and its items are assigned to the new instance's dictionary.
Note
If __getstate__
returns a false value, the __setstate__
method will not be called upon unpickling.
Refer to the section pickle-state
for more information about how to use the methods __getstate__
and __setstate__
.
Note
At unpickling time, some methods like __getattr__
, __getattribute__
, or __setattr__
may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement __getnewargs__
to establish such an invariant; otherwise, neither __new__
nor __init__
will be called.
pair: copy; protocol
As we shall see, pickle does not use directly the methods described above. In fact, these methods are part of the copy protocol which implements the __reduce__
special method. The copy protocol provides a unified interface for retrieving the data necessary for pickling and copying objects.3
Although powerful, implementing __reduce__
directly in your classes is error prone. For this reason, class designers should use the high-level interface (i.e., __getnewargs__
, __getstate__
and __setstate__
) whenever possible. We will show, however, cases where using __reduce__
is the only option or leads to more efficient pickling or both.
object.__reduce__()
The interface is currently defined as follows. The __reduce__
method takes no argument and shall return either a string or preferably a tuple (the returned object is often referred to as the "reduce value").
If a string is returned, the string should be interpreted as the name of a global variable. It should be the object's local name relative to its module; the pickle module searches the module namespace to determine the object's module. This behaviour is typically useful for singletons.
When a tuple is returned, it must be between two and five items long. Optional items can either be omitted, or None
can be provided as their value. The semantics of each item are in order:
- A callable object that will be called to create the initial version of the object.
- A tuple of arguments for the callable object. An empty tuple must be given if the callable does not accept any argument.
- Optionally, the object's state, which will be passed to the object's
__setstate__
method as previously described. If the object has no such method then, the value must be a dictionary and it will be added to the object's__dict__
attribute. - Optionally, an iterator (and not a sequence) yielding successive items. These items will be appended to the object either using
obj.append(item)
or, in batch, usingobj.extend(list_of_items)
. This is primarily used for list subclasses, but may be used by other classes as long as they haveappend
andextend
methods with the appropriate signature. (Whetherappend
orextend
is used depends on which pickle protocol version is used as well as the number of items to append, so both must be supported.) - Optionally, an iterator (not a sequence) yielding successive key-value pairs. These items will be stored to the object using
obj[key] = value
. This is primarily used for dictionary subclasses, but may be used by other classes as long as they implement__setitem__
.
object.__reduce_ex__(protocol)
Alternatively, a __reduce_ex__
method may be defined. The only difference is this method should take a single integer argument, the protocol version. When defined, pickle will prefer it over the __reduce__
method. In addition, __reduce__
automatically becomes a synonym for the extended version. The main use for this method is to provide backwards-compatible reduce values for older Python releases.
single: persistent_id (pickle protocol) single: persistent_load (pickle protocol)
For the benefit of object persistence, the pickle
module supports the notion of a reference to an object outside the pickled data stream. Such objects are referenced by a persistent ID, which should be either a string of alphanumeric characters (for protocol 0)4 or just an arbitrary object (for any newer protocol).
The resolution of such persistent IDs is not defined by the pickle
module; it will delegate this resolution to the user defined methods on the pickler and unpickler, persistent_id
and persistent_load
respectively.
To pickle objects that have an external persistent id, the pickler must have a custom persistent_id
method that takes an object as an argument and returns either None
or the persistent id for that object. When None
is returned, the pickler simply pickles the object as normal. When a persistent ID string is returned, the pickler will pickle that object, along with a marker so that the unpickler will recognize it as a persistent ID.
To unpickle external objects, the unpickler must have a custom persistent_load
method that takes a persistent ID object and returns the referenced object.
Here is a comprehensive example presenting how persistent ID can be used to pickle external objects by reference.
../includes/dbpickle.py
If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.
The global dispatch table managed by the copyreg
module is available as copyreg.dispatch_table
. Therefore, one may choose to use a modified copy of copyreg.dispatch_table
as a private dispatch table.
For example :
f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass
creates an instance of pickle.Pickler
with a private dispatch table which handles the SomeClass
class specially. Alternatively, the code :
class MyPickler(pickle.Pickler):
dispatch_table = copyreg.dispatch_table.copy()
dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)
does the same, but all instances of MyPickler
will by default share the same dispatch table. The equivalent code using the copyreg
module is :
copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)
single: __getstate__() (copy protocol) single: __setstate__() (copy protocol)
Here's an example that shows how to modify pickling behavior for a class. The TextReader
class opens a text file, and returns the line number and line contents each time its readline
method is called. If a TextReader
instance is pickled, all attributes except the file object member are saved. When the instance is unpickled, the file is reopened, and reading resumes from the last location. The __setstate__
and __getstate__
methods are used to implement this behavior. :
class TextReader:
"""Print and number lines in a text file."""
def __init__(self, filename):
self.filename = filename
self.file = open(filename)
self.lineno = 0
def readline(self):
self.lineno += 1
line = self.file.readline()
if not line:
return None
if line.endswith('\n'):
line = line[:-1]
return "%i: %s" % (self.lineno, line)
def __getstate__(self):
# Copy the object's state from self.__dict__ which contains
# all our instance attributes. Always use the dict.copy()
# method to avoid modifying the original state.
state = self.__dict__.copy()
# Remove the unpicklable entries.
del state['file']
return state
def __setstate__(self, state):
# Restore instance attributes (i.e., filename and lineno).
self.__dict__.update(state)
# Restore the previously opened file's state. To do so, we need to
# reopen it and read from it until the line count is restored.
file = open(self.filename)
for _ in range(self.lineno):
file.readline()
# Finally, save the file.
self.file = file
A sample usage might be something like this:
>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'
single: find_class() (pickle protocol)
By default, unpickling will import any class or function that it finds in the pickle data. For many applications, this behaviour is unacceptable as it permits the unpickler to import and invoke arbitrary code. Just consider what this hand-crafted pickle data stream does when loaded:
>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0
In this example, the unpickler imports the os.system
function and then apply the string argument "echo hello world". Although this example is inoffensive, it is not difficult to imagine one that could damage your system.
For this reason, you may want to control what gets unpickled by customizing Unpickler.find_class
. Unlike its name suggests, find_class
is called whenever a global (i.e., a class or a function) is requested. Thus it is possible to either completely forbid globals or restrict them to a safe subset.
Here is an example of an unpickler allowing only few safe classes from the builtins
module to be loaded:
import builtins
import io
import pickle
safe_builtins = {
'range',
'complex',
'set',
'frozenset',
'slice',
}
class RestrictedUnpickler(pickle.Unpickler):
def find_class(self, module, name):
# Only allow safe classes from builtins.
if module == "builtins" and name in safe_builtins:
return getattr(builtins, name)
# Forbid everything else.
raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
(module, name))
def restricted_loads(s):
"""Helper function analogous to pickle.loads()."""
return RestrictedUnpickler(io.BytesIO(s)).load()
A sample usage of our unpickler working has intended:
>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
... b'(S\'getattr(__import__("os"), "system")'
... b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
...
pickle.UnpicklingError: global 'builtins.eval' is forbidden
As our examples shows, you have to be careful with what you allow to be unpickled. Therefore if security is a concern, you may want to consider alternatives such as the marshalling API in xmlrpc.client
or third-party solutions.
For the simplest code, use the dump
and load
functions. :
import pickle
# An arbitrary collection of objects supported by pickle.
data = {
'a': [1, 2.0, 3, 4+6j],
'b': ("character string", b"byte string"),
'c': set([None, True, False])
}
with open('data.pickle', 'wb') as f:
# Pickle the 'data' dictionary using the highest protocol available.
pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)
The following example reads the resulting pickled data. :
import pickle
with open('data.pickle', 'rb') as f:
# The protocol version used is detected automatically, so we do not
# have to specify it.
data = pickle.load(f)
- Module
copyreg
Pickle interface constructor registration for extension types.
- Module
pickletools
Tools for working with and analyzing pickled data.
- Module
shelve
Indexed databases of objects; uses
pickle
.- Module
copy
Shallow and deep object copying.
- Module
marshal
High-performance serialization of built-in types.
Footnotes
Don't confuse this with the
marshal
module↩The exception raised will likely be an
ImportError
or anAttributeError
but it could be something else.↩The
copy
module uses this protocol for shallow and deep copying operations.↩The limitation on alphanumeric characters is due to the fact the persistent IDs, in protocol 0, are delimited by the newline character. Therefore if any kind of newline characters occurs in persistent IDs, the resulting pickle will become unreadable.↩