pickle does not work for Function

[ghost]

Following code -

import pickle
from sympy import *
v = (x,y,z) = symbols('x','y','z')
f = Function('f')(*v)
print f
print pickle.dumps(f)

Produces output -

f(x, y, z)
Traceback (most recent call last):
  File "pickletest.py", line 14, in <module>
    print pickle.dumps(f)
  File "/usr/lib/python2.5/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.5/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.5/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.5/pickle.py", line 295, in save
    self.save_global(obj)
  File "/usr/lib/python2.5/pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle f: it's not found as sympy.core.function.f

pickle not working for Function!

Original issue for #4297: http://code.google.com/p/sympy/issues/detail?id=1198

Original author: https://code.google.com/u/100789173971217230647/

[rkern]

It is not possible to fix this in the normal ways. The pickle mechanism handles
instances of metaclasses (e.g. instances of FunctionClass) before it checks any of
the customization hooks. The only way to fix this is to not generate them on the fly.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c1

Original author: https://code.google.com/u/robert.kern@gmail.com/

[smichr]

We could make our own dump, dumps, load, loads methods for Basic/Expr that handles this more gracefully, perhaps? Those methods would basically be calls to pickle's equivalent mathods after doing any necessary preprocessing. Not that I have any idea right now what that preprocessing might be. Does anyone else have an idea?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c2
Original author: https://code.google.com/u/117933771799683895267/

[asmeurer]

It was pointed out to me that namedtuple from the standard library does something similar to Function, and is pickable.  If you look at the source for namedtuple, there is this line:

# For pickling to work, the __module__ variable needs to be set to the frame
# where the named tuple is created.  Bypass this step in enviroments where
# sys._getframe is not defined (Jython for example) or sys._getframe is not
# defined for arguments greater than 0 (IronPython).
try:
    result.__module__ = _sys._getframe(1).f_globals.get('__name__', '__main__')
except (AttributeError, ValueError):
    pass


Would this work for Functions?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c3
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Actually, it looks like namedtuples cannot be pickled interactively.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c4
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Ronan, any idea how to fix this?  I think Robert's right.  We have to make Function() not return a class.  The above hack does work if you pickle from within a file (you have to adjust the argument of _getframe() to get to the module where the Function lives), but not interactively.  Otherwise, pickle tries to import obj.__module__, which I don't think can be anything interactively.  And it does seem to do this before handling any kind of __getstate__; defining one on the UndefinedFunction does nothing.

And by the way, Mateusz, for SymPy Live, I think Chris's idea would be better than what we currently do (basically recreating things from strings from what you told me).

**Cc:** Ronan.L...@gmail.com matt...@gmail.com  
**Labels:** -Priority-Medium Priority-High Live Milestone-Release0.7.2  

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c5
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Actually, pickle has its own function called whichmodule which tries to find the module if __module__ is set to None.  So if we wanted to use the above hack, it would be better to just set __module__ to None before creating the UndefinedFunction.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c6
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

But if you do that and remove the XFAIL from the relevant tests in test_pickle.py, you get this:

______ sympy/utilities/tests/test_pickling.py:test_core_dynamicfunctions _______
  File "/users/aaronmeurer/documents/python/sympy/sympy-scratch/sympy/utilities/tests/test_pickling.py", line 124, in test_core_dynamicfunctions
    check(f)
  File "/users/aaronmeurer/documents/python/sympy/sympy-scratch/sympy/utilities/tests/test_pickling.py", line 44, in check
    b = pickle.loads(pickle.dumps(a, protocol))
  File "/sw/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/sw/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/sw/lib/python2.7/pickle.py", line 295, in save
    self.save_global(obj)
  File "/sw/lib/python2.7/pickle.py", line 739, in save_global
    module = whichmodule(obj, name)
  File "/sw/lib/python2.7/pickle.py", line 817, in whichmodule
    if name != '__main__' and getattr(module, funcname, None) is func:
  File "/sw/lib/python2.7/site-packages/py/_apipkg.py", line 159, in __getattribute__
    return getattr(getmod(), name)
  File "/sw/lib/python2.7/site-packages/py/_apipkg.py", line 144, in getmod
    x = importobj(modpath, None)
  File "/sw/lib/python2.7/site-packages/py/_apipkg.py", line 37, in importobj
    module = __import__(modpath, None, None, ['__doc__'])
ImportError: No module named pytest

So I give up.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c7
Original author: https://code.google.com/u/asmeurer@gmail.com/

[mattpap]

**Owner:** ---  

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c8
Original author: https://code.google.com/u/101069955704897915480/

[asmeurer]

I'm postponing instead of removing the milestone here, as this would be pretty nice to get fixed (unless we can find a different way to do it in SymPy Live).

**Labels:** -Milestone-Release0.7.2 Milestone-Release0.7.3  

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c9
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Would it be possible to create a custom pickler that does this, at least for SymPy Live?  We already are having to do that for SymPy Live, due to issue 5686 (see https://github.com/sympy/sympy-live/pull/46 ). This would be only putting a band-aid on the problem, but it would fix bugs in SymPy Live related to this, like:

>>> a = g(x).diff(x)
>>> a
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'a' is not defined

Referenced issues: #5686
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c10
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

**Status:** Valid  

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c11
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

**Blocking:** 5699  

Referenced issues: #5699
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c12
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Setting __module__ to None, i.e.,

diff --git a/sympy/core/function.py b/sympy/core/function.py
index 9e0f7af..fd6d935 100644
--- a/sympy/core/function.py
+++ b/sympy/core/function.py
@@ -612,7 +612,9 @@ class UndefinedFunction(FunctionClass):
     The (meta)class of undefined functions.
     """
     def __new__(mcl, name):
-        return BasicMeta.__new__(mcl, name, (AppliedUndef,), {})
+        ret = BasicMeta.__new__(mcl, name, (AppliedUndef,), {})
+        ret.__module__ = None
+        return ret


 class WildFunction(Function, AtomicExpr):

does fix SymPy Live ( issue 5699 , and also issue 6775 ), but for whatever reason, the pickling tests still fail.

Referenced issues: #5699, #6775
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c13
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

I tried monkey patching UndefinedFunction in SymPy Live, but it doesn't work, I think because it is already used by the time it is imported.  It's also possible I monkey patched it wrong, though. I tried 

diff --git a/shell.py b/shell.py
index 1de7615..93f691c 100755
--- a/shell.py
+++ b/shell.py
@@ -103,6 +103,12 @@ INITIAL_UNPICKLABLES = [
   "from google.appengine.api import users",
   "from __future__ import division",
   "from sympy import *",
+  """from sympy.core.function import UndefinedFunction
+def _picklable_new__(mcl, name):
+      ret = BasicMeta.__new__(mcl, name, (AppliedUndef,), {})
+      ret.__module__ = None
+      return ret
+UndefinedFunction.__new__ = _picklable_new__""",
 ]

 PREEXEC = """\

Assuming that monkey patching really doesn't work, what's the best way to fix this? Assuming that releasing a new version of SymPy is not an easy option, we'll have to include a patched version if SymPy in SymPy Live.  How should we do that, with the submodule?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c14
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

The patch from comment 13 doesn't lead to any test failures.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c15
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Any thoughts on this David? This fixes the worst of the SymPy Live pickling errors.

**Cc:** li.david...@gmail.com  

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c16
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

**Blockedon:** 4787  

Referenced issues: #4787
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c17
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

I went ahead and submitted the fix from comment 13 at https://github.com/sympy/sympy/pull/2114 .  It will fix SymPy live the next time we release.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c18
Original author: https://code.google.com/u/asmeurer@gmail.com/

[lidavidm]

I think this fix is fine, but hopefully we can make pickling work better overall.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c19
Original author: https://code.google.com/u/112898427768461421869/

[asmeurer]

My question is, what's the best way (short of releasing) to backport this to SymPy Live?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c20
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Of course, the easiest way is if we just release. Anyone want to help?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c21
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

It seems this breaks printing

In [1]: f
Out[1]: ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-9a8ad92c50ca> in <module>()
----> 1 f

/Users/aaronmeurer/Documents/ipython/IPython/core/displayhook.pyc in __call__(self, result)
    245             self.start_displayhook()
    246             self.write_output_prompt()
--> 247             format_dict, md_dict = self.compute_format_data(result)
    248             self.write_format_data(format_dict, md_dict)
    249             self.update_user_ns(result)

/Users/aaronmeurer/Documents/ipython/IPython/core/displayhook.pyc in compute_format_data(self, result)
    155
    156         """
--> 157         return self.shell.display_formatter.format(result)
    158
    159     def write_format_data(self, format_dict, md_dict=None):

/Users/aaronmeurer/Documents/ipython/IPython/core/formatters.pyc in format(self, obj, include, exclude)
    150             md = None
    151             try:
--> 152                 data = formatter(obj)
    153             except:
    154                 # FIXME: log the exception

/Users/aaronmeurer/Documents/ipython/IPython/core/formatters.pyc in __call__(self, obj)
    479                 type_pprinters=self.type_printers,
    480                 deferred_pprinters=self.deferred_printers)
--> 481             printer.pretty(obj)
    482             printer.flush()
    483             return stream.getvalue()

/Users/aaronmeurer/Documents/ipython/IPython/lib/pretty.pyc in pretty(self, obj)
    345                 if cls in self.type_pprinters:
    346                     # printer registered in self.type_pprinters
--> 347                     return self.type_pprinters[cls](obj, self, cycle)
    348                 else:
    349                     # deferred printer

/Users/aaronmeurer/Documents/ipython/IPython/lib/pretty.pyc in _type_pprint(obj, p, cycle)
    648         name = obj.__name__
    649     else:
--> 650         name = mod + '.' + obj.__name__
    651     p.text(name)
    652

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c22
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

As you can see from the traceback, this is entirely an IPython thing.  I guess they are a little sensitive to the __module__ being correct.  I'm not sure how to move forward with this.  I *guess* it's an IPython bug.  Even so, it would be annoying to have this out there.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c23
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

See https://github.com/ipython/ipython/issues/3376 .

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c24
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Issue 3870 has been merged into this issue.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c25
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Issue 3870 has been merged into this issue.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c26
Original author: https://code.google.com/u/asmeurer@gmail.com/

[asmeurer]

Issue 3870 has been merged into this issue.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c27
Original author: https://code.google.com/u/asmeurer@gmail.com/

[mrocklin]

**Labels:** -Milestone-Release0.7.3  

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c28
Original author: https://code.google.com/u/109882876523836932473/

[asmeurer]

For anyone following this issue, the dill ( https://github.com/uqfoundation/dill ) library can pickle just about anything. See https://github.com/sympy/sympy-live/pull/71 for some work on making it work with Function.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c29
Original author: https://code.google.com/u/asmeurer@gmail.com/

[gschwind]

Hello,

I got this issue within our python project lca_algebraic. I provide my analysis of the issue and the workaround that I using.

The issue

The issue come from the fact that when we create a sympy.Function this function will create a 'custom' type on the fly using 'type' built-in. the python pickler does not actually dump type objects i.e. class definitions but just dump the fully qualified name of the given type [1]. That mean that only type that can be imported from fully qualified name can be dump. In the case of Function, an new type is created on the fly [2]. This new on-the-fly type does not have fully qualified name.

Work arround

To solve the issue I followed python documentation here, and I wrote the following pickler:

# Overide the behaviour for pickling sympy.UndefineFunction
class Pickler(pickle.Pickler):
    from sympy.core.function import UndefinedFunction
    def reducer_override(self, obj):
        # FIXME: maybe too gready, we may check if obj is an instance of
        #        registered functions instead.
        if obj.__class__ is Pickler.UndefinedFunction:
            return type, (obj.__name__, obj.__bases__, dict(obj.__dict__))
        return NotImplemented

Note that it's working in my case but I guess it may not be appropriate for all case and may be wrong. I tried to fix the generated on-the-fly type within sympy using the special function obj.__reduce__ without success.

Best regards

[oscarbenjamin]

Probably UndefinedFunction should define __getnewargs_ex__ like Symbol does:

sympy/sympy/core/symbol.py

Lines 383 to 384 in e89ee93

	def __getnewargs_ex__(self):
	return ((self.name,), self._assumptions_orig)

[gschwind]

I given a try to that without success. It's seems override of pickle functions is ignored when it's about type, i.e. class definition instead of classical class instances. But I may did it wrong.