pickle does not work for Function #4297

ghost · 2008-11-16T09:02:24Z

Following code -

import pickle
from sympy import *
v = (x,y,z) = symbols('x','y','z')
f = Function('f')(*v)
print f
print pickle.dumps(f)

Produces output -

f(x, y, z)
Traceback (most recent call last):
  File "pickletest.py", line 14, in <module>
    print pickle.dumps(f)
  File "/usr/lib/python2.5/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.5/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.5/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.5/pickle.py", line 295, in save
    self.save_global(obj)
  File "/usr/lib/python2.5/pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle f: it's not found as sympy.core.function.f

pickle not working for Function!

Original issue for #4297: http://code.google.com/p/sympy/issues/detail?id=1198

Original author: https://code.google.com/u/100789173971217230647/

The text was updated successfully, but these errors were encountered:

rkern · 2009-01-11T11:30:11Z

It is not possible to fix this in the normal ways. The pickle mechanism handles
instances of metaclasses (e.g. instances of FunctionClass) before it checks any of
the customization hooks. The only way to fix this is to not generate them on the fly.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c1

Original author: https://code.google.com/u/robert.kern@gmail.com/

smichr · 2011-07-02T05:11:34Z

We could make our own dump, dumps, load, loads methods for Basic/Expr that handles this more gracefully, perhaps? Those methods would basically be calls to pickle's equivalent mathods after doing any necessary preprocessing. Not that I have any idea right now what that preprocessing might be. Does anyone else have an idea?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c2
Original author: https://code.google.com/u/117933771799683895267/

asmeurer · 2011-07-24T00:51:31Z

It was pointed out to me that namedtuple from the standard library does something similar to Function, and is pickable.  If you look at the source for namedtuple, there is this line:

# For pickling to work, the __module__ variable needs to be set to the frame
# where the named tuple is created.  Bypass this step in enviroments where
# sys._getframe is not defined (Jython for example) or sys._getframe is not
# defined for arguments greater than 0 (IronPython).
try:
    result.__module__ = _sys._getframe(1).f_globals.get('__name__', '__main__')
except (AttributeError, ValueError):
    pass


Would this work for Functions?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c3
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2011-07-24T01:05:00Z

Actually, it looks like namedtuples cannot be pickled interactively.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c4
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2011-07-26T04:54:17Z

Ronan, any idea how to fix this?  I think Robert's right.  We have to make Function() not return a class.  The above hack does work if you pickle from within a file (you have to adjust the argument of _getframe() to get to the module where the Function lives), but not interactively.  Otherwise, pickle tries to import obj.__module__, which I don't think can be anything interactively.  And it does seem to do this before handling any kind of __getstate__; defining one on the UndefinedFunction does nothing.

And by the way, Mateusz, for SymPy Live, I think Chris's idea would be better than what we currently do (basically recreating things from strings from what you told me).

**Cc:** Ronan.L...@gmail.com matt...@gmail.com  
**Labels:** -Priority-Medium Priority-High Live Milestone-Release0.7.2

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c5
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2011-07-26T04:59:20Z

Actually, pickle has its own function called whichmodule which tries to find the module if __module__ is set to None.  So if we wanted to use the above hack, it would be better to just set __module__ to None before creating the UndefinedFunction.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c6
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2011-07-26T05:04:05Z

But if you do that and remove the XFAIL from the relevant tests in test_pickle.py, you get this:

______ sympy/utilities/tests/test_pickling.py:test_core_dynamicfunctions _______
  File "/users/aaronmeurer/documents/python/sympy/sympy-scratch/sympy/utilities/tests/test_pickling.py", line 124, in test_core_dynamicfunctions
    check(f)
  File "/users/aaronmeurer/documents/python/sympy/sympy-scratch/sympy/utilities/tests/test_pickling.py", line 44, in check
    b = pickle.loads(pickle.dumps(a, protocol))
  File "/sw/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/sw/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/sw/lib/python2.7/pickle.py", line 295, in save
    self.save_global(obj)
  File "/sw/lib/python2.7/pickle.py", line 739, in save_global
    module = whichmodule(obj, name)
  File "/sw/lib/python2.7/pickle.py", line 817, in whichmodule
    if name != '__main__' and getattr(module, funcname, None) is func:
  File "/sw/lib/python2.7/site-packages/py/_apipkg.py", line 159, in __getattribute__
    return getattr(getmod(), name)
  File "/sw/lib/python2.7/site-packages/py/_apipkg.py", line 144, in getmod
    x = importobj(modpath, None)
  File "/sw/lib/python2.7/site-packages/py/_apipkg.py", line 37, in importobj
    module = __import__(modpath, None, None, ['__doc__'])
ImportError: No module named pytest

So I give up.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c7
Original author: https://code.google.com/u/asmeurer@gmail.com/

mattpap · 2011-10-28T01:49:10Z

**Owner:** ---

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c8
Original author: https://code.google.com/u/101069955704897915480/

asmeurer · 2011-11-16T00:38:03Z

I'm postponing instead of removing the milestone here, as this would be pretty nice to get fixed (unless we can find a different way to do it in SymPy Live).

**Labels:** -Milestone-Release0.7.2 Milestone-Release0.7.3

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c9
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2012-01-11T14:36:24Z

Would it be possible to create a custom pickler that does this, at least for SymPy Live?  We already are having to do that for SymPy Live, due to issue 5686 (see https://github.com/sympy/sympy-live/pull/46 ). This would be only putting a band-aid on the problem, but it would fix bugs in SymPy Live related to this, like:

>>> a = g(x).diff(x)
>>> a
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'a' is not defined

Referenced issues: #5686
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c10
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2012-03-20T07:51:51Z

**Status:** Valid

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c11
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-03-03T21:10:46Z

**Blocking:** 5699

Referenced issues: #5699
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c12
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-03-03T21:40:16Z

Setting __module__ to None, i.e.,

diff --git a/sympy/core/function.py b/sympy/core/function.py
index 9e0f7af..fd6d935 100644
--- a/sympy/core/function.py
+++ b/sympy/core/function.py
@@ -612,7 +612,9 @@ class UndefinedFunction(FunctionClass):
     The (meta)class of undefined functions.
     """
     def __new__(mcl, name):
-        return BasicMeta.__new__(mcl, name, (AppliedUndef,), {})
+        ret = BasicMeta.__new__(mcl, name, (AppliedUndef,), {})
+        ret.__module__ = None
+        return ret


 class WildFunction(Function, AtomicExpr):

does fix SymPy Live ( issue 5699 , and also issue 6775 ), but for whatever reason, the pickling tests still fail.

Referenced issues: #5699, #6775
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c13
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-03-03T21:54:46Z

I tried monkey patching UndefinedFunction in SymPy Live, but it doesn't work, I think because it is already used by the time it is imported.  It's also possible I monkey patched it wrong, though. I tried 

diff --git a/shell.py b/shell.py
index 1de7615..93f691c 100755
--- a/shell.py
+++ b/shell.py
@@ -103,6 +103,12 @@ INITIAL_UNPICKLABLES = [
   "from google.appengine.api import users",
   "from __future__ import division",
   "from sympy import *",
+  """from sympy.core.function import UndefinedFunction
+def _picklable_new__(mcl, name):
+      ret = BasicMeta.__new__(mcl, name, (AppliedUndef,), {})
+      ret.__module__ = None
+      return ret
+UndefinedFunction.__new__ = _picklable_new__""",
 ]

 PREEXEC = """\

Assuming that monkey patching really doesn't work, what's the best way to fix this? Assuming that releasing a new version of SymPy is not an easy option, we'll have to include a patched version if SymPy in SymPy Live.  How should we do that, with the submodule?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c14
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-03-04T01:14:57Z

The patch from comment 13 doesn't lead to any test failures.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c15
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-15T20:42:56Z

Any thoughts on this David? This fixes the worst of the SymPy Live pickling errors.

**Cc:** li.david...@gmail.com

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c16
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-16T19:43:37Z

**Blockedon:** 4787

Referenced issues: #4787
Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c17
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-16T19:44:13Z

I went ahead and submitted the fix from comment 13 at https://github.com/sympy/sympy/pull/2114 .  It will fix SymPy live the next time we release.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c18
Original author: https://code.google.com/u/asmeurer@gmail.com/

lidavidm · 2013-05-16T22:51:03Z

I think this fix is fine, but hopefully we can make pickling work better overall.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c19
Original author: https://code.google.com/u/112898427768461421869/

asmeurer · 2013-05-16T22:52:11Z

My question is, what's the best way (short of releasing) to backport this to SymPy Live?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c20
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-17T01:36:46Z

Of course, the easiest way is if we just release. Anyone want to help?

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c21
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-29T02:25:20Z

It seems this breaks printing

In [1]: f
Out[1]: ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-9a8ad92c50ca> in <module>()
----> 1 f

/Users/aaronmeurer/Documents/ipython/IPython/core/displayhook.pyc in __call__(self, result)
    245             self.start_displayhook()
    246             self.write_output_prompt()
--> 247             format_dict, md_dict = self.compute_format_data(result)
    248             self.write_format_data(format_dict, md_dict)
    249             self.update_user_ns(result)

/Users/aaronmeurer/Documents/ipython/IPython/core/displayhook.pyc in compute_format_data(self, result)
    155
    156         """
--> 157         return self.shell.display_formatter.format(result)
    158
    159     def write_format_data(self, format_dict, md_dict=None):

/Users/aaronmeurer/Documents/ipython/IPython/core/formatters.pyc in format(self, obj, include, exclude)
    150             md = None
    151             try:
--> 152                 data = formatter(obj)
    153             except:
    154                 # FIXME: log the exception

/Users/aaronmeurer/Documents/ipython/IPython/core/formatters.pyc in __call__(self, obj)
    479                 type_pprinters=self.type_printers,
    480                 deferred_pprinters=self.deferred_printers)
--> 481             printer.pretty(obj)
    482             printer.flush()
    483             return stream.getvalue()

/Users/aaronmeurer/Documents/ipython/IPython/lib/pretty.pyc in pretty(self, obj)
    345                 if cls in self.type_pprinters:
    346                     # printer registered in self.type_pprinters
--> 347                     return self.type_pprinters[cls](obj, self, cycle)
    348                 else:
    349                     # deferred printer

/Users/aaronmeurer/Documents/ipython/IPython/lib/pretty.pyc in _type_pprint(obj, p, cycle)
    648         name = obj.__name__
    649     else:
--> 650         name = mod + '.' + obj.__name__
    651     p.text(name)
    652

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c22
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-29T02:36:52Z

As you can see from the traceback, this is entirely an IPython thing.  I guess they are a little sensitive to the __module__ being correct.  I'm not sure how to move forward with this.  I *guess* it's an IPython bug.  Even so, it would be annoying to have this out there.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c23
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-05-29T02:59:35Z

See https://github.com/ipython/ipython/issues/3376 .

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c24
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-06-06T19:03:53Z

Issue 3870 has been merged into this issue.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c25
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-06-06T19:48:35Z

Issue 3870 has been merged into this issue.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c26
Original author: https://code.google.com/u/asmeurer@gmail.com/

asmeurer · 2013-06-07T22:39:48Z

Issue 3870 has been merged into this issue.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c27
Original author: https://code.google.com/u/asmeurer@gmail.com/

mrocklin · 2013-06-28T16:10:03Z

**Labels:** -Milestone-Release0.7.3

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c28
Original author: https://code.google.com/u/109882876523836932473/

asmeurer · 2013-07-13T20:41:17Z

For anyone following this issue, the dill ( https://github.com/uqfoundation/dill ) library can pickle just about anything. See https://github.com/sympy/sympy-live/pull/71 for some work on making it work with Function.

Original comment: http://code.google.com/p/sympy/issues/detail?id=1198#c29
Original author: https://code.google.com/u/asmeurer@gmail.com/

ghost added the imported label Mar 7, 2014

ghost added Live labels Mar 7, 2014

This was referenced Mar 6, 2014

Functions should be objects #4787

Open

Variable definitions do not always work at SymPy Live #5699

Open

ValueError when executing rsolve's example in SymPy Live #6775

Closed

Broken pipe when invoke solve() or nsolve() #6863

Open

asmeurer mentioned this issue Apr 10, 2014

Functions can't be pickled (or dilled) #7394

Closed

jcrist removed the Priority: High label Apr 17, 2015

moorepants mentioned this issue Oct 3, 2016

implemented_function (UndefinedFunction) fails to deserialize with cloudpickle #11692

Closed

FabioLuporini mentioned this issue May 18, 2020

Fix pickling with MPI full mode devitocodes/devito#1291

Merged

smichr removed the Valid label Nov 19, 2021

TJStienstra mentioned this issue May 3, 2023

Add simple pickle compatibility to ReferenceFrame #25102

Merged

pickle does not work for Function #4297

pickle does not work for Function #4297

ghost commented Nov 16, 2008

rkern commented Jan 11, 2009

smichr commented Jul 2, 2011

asmeurer commented Jul 24, 2011

asmeurer commented Jul 24, 2011

asmeurer commented Jul 26, 2011

asmeurer commented Jul 26, 2011

asmeurer commented Jul 26, 2011

mattpap commented Oct 28, 2011

asmeurer commented Nov 16, 2011

asmeurer commented Jan 11, 2012

asmeurer commented Mar 20, 2012

asmeurer commented Mar 3, 2013

asmeurer commented Mar 3, 2013

asmeurer commented Mar 3, 2013

asmeurer commented Mar 4, 2013

asmeurer commented May 15, 2013

asmeurer commented May 16, 2013

asmeurer commented May 16, 2013

lidavidm commented May 16, 2013

asmeurer commented May 16, 2013

asmeurer commented May 17, 2013

asmeurer commented May 29, 2013

asmeurer commented May 29, 2013

asmeurer commented May 29, 2013

asmeurer commented Jun 6, 2013

asmeurer commented Jun 6, 2013

asmeurer commented Jun 7, 2013

mrocklin commented Jun 28, 2013

asmeurer commented Jul 13, 2013

pickle does not work for Function #4297

pickle does not work for Function #4297

Comments

ghost commented Nov 16, 2008

rkern commented Jan 11, 2009

smichr commented Jul 2, 2011

asmeurer commented Jul 24, 2011

asmeurer commented Jul 24, 2011

asmeurer commented Jul 26, 2011

asmeurer commented Jul 26, 2011

asmeurer commented Jul 26, 2011

mattpap commented Oct 28, 2011

asmeurer commented Nov 16, 2011

asmeurer commented Jan 11, 2012

asmeurer commented Mar 20, 2012

asmeurer commented Mar 3, 2013

asmeurer commented Mar 3, 2013

asmeurer commented Mar 3, 2013

asmeurer commented Mar 4, 2013

asmeurer commented May 15, 2013

asmeurer commented May 16, 2013

asmeurer commented May 16, 2013

lidavidm commented May 16, 2013

asmeurer commented May 16, 2013

asmeurer commented May 17, 2013

asmeurer commented May 29, 2013

asmeurer commented May 29, 2013

asmeurer commented May 29, 2013

asmeurer commented Jun 6, 2013

asmeurer commented Jun 6, 2013

asmeurer commented Jun 7, 2013

mrocklin commented Jun 28, 2013

asmeurer commented Jul 13, 2013