Skip to content

Commit

Permalink
update load_module dump_module docs
Browse files Browse the repository at this point in the history
  • Loading branch information
mmckerns committed Jul 10, 2022
1 parent 0fa524a commit c23e049
Showing 1 changed file with 118 additions and 77 deletions.
195 changes: 118 additions & 77 deletions dill/_dill.py
Original file line number Diff line number Diff line change
Expand Up @@ -413,44 +413,60 @@ def dump_module(
) -> None:
"""Pickle the current state of :py:mod:`__main__` or another module to a file.
Save the interpreter session (the contents of the built-in module
:py:mod:`__main__`) or the state of another module to a pickle file. This
can then be restored by calling the function :py:func:`load_module`.
Runtime-created modules, like the ones constructed by
:py:class:`~types.ModuleType`, can also be saved and restored thereafter.
Save the contents of :py:mod:`__main__` (e.g. from an interactive
interpreter session), an imported module, or a module-type object (e.g.
built with :py:class:`~types.ModuleType`), to a file. The pickled
module can then be restored with the function :py:func:`load_module`.
Parameters:
filename: a path-like object or a writable stream.
main: a module object or an importable module name.
refimported: if `True`, all imported objects in the module's namespace
are saved by reference. *Note:* this is different from the ``byref``
option of other "dump" functions and is not affected by
``settings['byref']``.
main: a module object or the name of an importable module.
refimported: if `True`, all objects imported into the module's

This comment has been minimized.

Copy link
@leogama

leogama Jul 12, 2022

Contributor

I'm merging this commit with my last changes to the filtering feature. This description is not strictly correct, there's a list of object types:

dill/dill/_dill.py

Lines 326 to 327 in c23e049

SESSION_IMPORTED_AS_TYPES = (ModuleType, ClassType, TypeType, Exception,
FunctionType, MethodType, BuiltinMethodType)
that are looked after in the loaded modules to be saved by reference (the Exception type shouldn't be there, I removed it as exception classes are common types, and ClassType == TypeType now).

I'm exploring a variety of modules to see if there are other types of objects with a __module__ attribute that also should be saved by reference with this option. Ideally, these should include all objects that are not "simple values".

namespace are saved by reference. *Note:* this is similar but
independent from ``dill.settings[`byref`]``, as ``refimported``
refers to all imported objects, while ``byref`` only affects
select objects.
**kwds: extra keyword arguments passed to :py:class:`Pickler()`.
Raises:
:py:exc:`PicklingError`: if pickling fails.
Examples:
- Save current session state:
- Save current interpreter session state:
>>> import dill
>>> dill.dump_module() # save state of __main__ to /tmp/session.pkl
>>> squared = lambda x:x*x
>>> dill.dump_module() # save state of __main__ to /tmp/session.pkl
- Save the state of an imported/importable module:
>>> import my_mod as m
>>> m.var = 'new value'
>>> dill.dump_module('my_mod_session.pkl', main='my_mod')
>>> import dill
>>> import pox
>>> pox.plus_one = lambda x:x+1
>>> dill.dump_module('pox_session.pkl', main=pox)
- Save the state of a non-importable, runtime-created module:
- Save the state of a non-importable, module-type object:
>>> import dill
>>> from types import ModuleType
>>> runtime = ModuleType('runtime')
>>> runtime.food = ['bacon', 'eggs', 'spam']
>>> runtime.process_food = m.process_food
>>> dill.dump_module('runtime_session.pkl', main=runtime, refimported=True)
>>> foo = ModuleType('foo')
>>> foo.values = [1,2,3]
>>> import math
>>> foo.sin = math.sin
>>> dill.dump_module('foo_session.pkl', main=foo, refimported=True)
- Restore the state of the saved modules:
>>> import dill
>>> dill.load_module()
>>> squared(2)
4
>>> pox = dill.load_module('pox_session.pkl')
>>> pox.plus_one(1)
2
>>> foo = dill.load_module('foo_session.pkl')
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]
*Changed in version 0.3.6:* the function ``dump_session()`` was renamed to
``dump_module()``.
Expand All @@ -460,13 +476,12 @@ def dump_module(
"""
if 'byref' in kwds:
warnings.warn(
"The parameter 'byref' was renamed to 'refimported', use this"
" instead. Note: the underlying dill.Pickler do accept a 'byref'"
" argument, but it has no effect on session saving.",
"The argument 'byref' has been renamed 'refimported'"
" to distinguish it from dill.settings['byref'].",
PendingDeprecationWarning
)
if refimported:
raise ValueError("both 'refimported' and 'byref' arguments were used.")
raise TypeError("both 'refimported' and 'byref' were used")
refimported = kwds.pop('byref')
from .settings import settings
protocol = settings['protocol']
Expand Down Expand Up @@ -494,7 +509,7 @@ def dump_module(

# Backward compatibility.
def dump_session(filename=str(TEMPDIR/'session.pkl'), main=None, byref=False, **kwds):
warnings.warn("dump_session() was renamed to dump_module()", PendingDeprecationWarning)
warnings.warn("dump_session() has been renamed dump_module()", PendingDeprecationWarning)
dump_module(filename, main, refimported=byref, **kwds)
dump_session.__doc__ = dump_module.__doc__

Expand Down Expand Up @@ -565,68 +580,96 @@ def load_module(
"""Update :py:mod:`__main__` or another module with the state from the
session file.
Restore the interpreter session (the built-in module :py:mod:`__main__`) or
the state of another module from a pickle file created by the function
:py:func:`dump_module`.
Restore a module to the state saved with :py:func:`dump_module`. The
saved module can be :py:mod:`__main__` (e.g. an interpreter session),
an imported module, or a module-type object (e.g. created with
:py:class:`~types.ModuleType`).
If loading the state of a (non-importable) runtime-created module, a version
of this module may be passed as the argument ``main``. Otherwise, a new
module object is created with :py:class:`~types.ModuleType` and returned
after it's updated.
When restoring the state of a non-importable module-type object, the
current instance of this module may be passed as the argument ``main``.
Otherwise, a new instance is created with :py:class:`~types.ModuleType`
and returned.
Parameters:
filename: a path-like object or a readable stream.
main: an importable module name or a module object.
main: a module object or the name of an importable module.
**kwds: extra keyword arguments passed to :py:class:`Unpickler()`.
Raises:
:py:exc:`UnpicklingError`: if unpickling fails.
:py:exc:`ValueError`: if the ``main`` argument and the session file's
module are incompatible.
:py:exc:`ValueError`: if the argument ``main`` and module saved
at ``filename`` are incompatible.
Returns:
The restored module if it's different from :py:mod:`__main__` and
wasn't passed as the ``main`` argument.
A module object, if the saved module is not :py:mod:`__main__` or
a module instance wasn't provided with the argument ``main``.
Examples:
- Load a saved session state:
>>> import dill, sys
>>> dill.load_module() # updates __main__ from /tmp/session.pkl
>>> restored_var
'this variable was created/updated by load_module()'
- Save the state of some modules:
>>> import dill
>>> squared = lambda x:x*x
>>> dill.dump_module() # save state of __main__ to /tmp/session.pkl
>>>
>>> import pox # an imported module
>>> pox.plus_one = lambda x:x+1
>>> dill.dump_module('pox_session.pkl', main=pox)
>>>
>>> from types import ModuleType
>>> foo = ModuleType('foo') # a module-type object
>>> foo.values = [1,2,3]
>>> import math
>>> foo.sin = math.sin
>>> dill.dump_module('foo_session.pkl', main=foo, refimported=True)
- Restore the state of the interpreter:
>>> import dill
>>> dill.load_module() # updates __main__ from /tmp/session.pkl
>>> squared(2)
4
- Load the saved state of an importable module:
>>> my_mod = dill.load_module('my_mod_session.pkl')
>>> my_mod.var
'new value'
>>> my_mod in sys.modules.values()
>>> import dill
>>> pox = dill.load_module('pox_session.pkl')
>>> pox.plus_one(1)
2
>>> import sys
>>> pox in sys.modules.values()
True
- Load the saved state of a non-importable, runtime-created module:
- Load the saved state of a non-importable module-type object:
>>> runtime = dill.load_module('runtime_session.pkl')
>>> runtime.process_food is my_mod.process_food # was saved by reference
>>> import dill
>>> foo = dill.load_module('foo_session.pkl')
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]
>>> import math
>>> foo.sin is math.sin # foo.sin was saved by reference
True
>>> runtime in sys.modules.values()
>>> import sys
>>> foo in sys.modules.values()
False
- Update the state of a non-importable, runtime-created module:
- Update the state of a non-importable module-type object:
>>> import dill
>>> from types import ModuleType
>>> runtime = ModuleType('runtime')
>>> runtime.food = ['pizza', 'burger']
>>> dill.load_module('runtime_session.pkl', main=runtime)
>>> runtime.food
['bacon', 'eggs', 'spam']
>>> foo = ModuleType('foo')
>>> foo.values = ['a','b']
>>> foo.sin = lambda x:x*x
>>> dill.load_module('foo_session.pkl', main=foo)
>>> [foo.sin(x) for x in foo.values]
[0.8414709848078965, 0.9092974268256817, 0.1411200080598672]
*Changed in version 0.3.6:* the function ``load_session()`` was renamed to
``load_module()``.
See also:
:py:func:`load_module_asdict` to load the contents of a saved session
(from :py:mod:`__main__` or any importable module) into a dictionary.
:py:func:`load_module_asdict` to load the contents of module saved
with :py:func:`dump_module` into a dictionary.
"""
main_arg = main
if hasattr(filename, 'read'):
Expand Down Expand Up @@ -698,7 +741,7 @@ def load_module(

# Backward compatibility.
def load_session(filename=str(TEMPDIR/'session.pkl'), main=None, **kwds):
warnings.warn("load_session() was renamed to load_module().", PendingDeprecationWarning)
warnings.warn("load_session() has been renamed load_module().", PendingDeprecationWarning)
load_module(filename, main, **kwds)
load_session.__doc__ = load_module.__doc__

Expand All @@ -708,20 +751,19 @@ def load_module_asdict(
**kwds
) -> dict:
"""
Load the contents of a module from a session file into a dictionary.
``load_module_asdict()`` does the equivalent of this function::
Load the contents of a saved module into a dictionary.
lambda filename: vars(load_module(filename)).copy()
``load_module_asdict()`` is the near-equivalent of::
but without changing the original module.
lambda filename: vars(dill.load_module(filename)).copy()
The loaded module's origin is stored in the ``__session__`` attribute.
however, does not alter the original module. Also, the path of
the loaded module is stored in the ``__session__`` attribute.
Parameters:
filename: a path-like object or a readable stream
update: if `True`, the dictionary is updated with the current state of
the module before loading variables from the session file
update: if `True`, initialize the dictionary with the current state
of the module prior to loading the state stored at filename.
**kwds: extra keyword arguments passed to :py:class:`Unpickler()`
Raises:
Expand All @@ -731,8 +773,7 @@ def load_module_asdict(
A copy of the restored module's dictionary.
Note:
If the ``update`` option is used, the original module will be loaded if
it wasn't yet.
If ``update`` is True, the saved module may be imported then updated.
Example:
>>> import dill
Expand All @@ -741,18 +782,18 @@ def load_module_asdict(
>>> dill.dump_module()
>>> anum = 0
>>> new_var = 'spam'
>>> main_vars = dill.load_module_asdict()
>>> main_vars['__name__'], main_vars['__session__']
>>> main = dill.load_module_asdict()
>>> main['__name__'], main['__session__']
('__main__', '/tmp/session.pkl')
>>> main_vars is globals() # loaded objects don't reference current global variables
>>> main is globals() # loaded objects don't reference globals
False
>>> main_vars['alist'] == alist
>>> main['alist'] == alist
True
>>> main_vars['alist'] is alist # was saved by value
>>> main['alist'] is alist # was saved by value
False
>>> main_vars['anum'] == anum # changed after the session was saved
>>> main['anum'] == anum # changed after the session was saved
False
>>> new_var in main_vars # would be True if the option 'update' was set
>>> new_var in main # would be True if the option 'update' was set
False
"""
if 'main' in kwds:
Expand Down

0 comments on commit c23e049

Please sign in to comment.