Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GenericLikelihoodModel is not picklable #1873

Closed
hmgaudecker opened this issue Aug 7, 2014 · 6 comments

Comments

Projects
None yet
3 participants
@hmgaudecker
Copy link
Contributor

commented Aug 7, 2014

Apparently scipy.stats.norm is being attached to GenericLikelihoodModel causing pickle.load() to fail.

Minimal example below, also see the mailing list discussion

In [1]: 
import pickle
from statsmodels.base.model import GenericLikelihoodModel

In [2]:
model = GenericLikelihoodModel(endog=[0, 1], exog=[0.2, 3.0])

In [3]:
with open('model.pickle', mode='wb') as f:
    pickle.dump(model, f)

In [4]:
with open('model.pickle', mode='rb') as f:
    model = pickle.load(f)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-d8739e3ead39> in <module>()
      1 with open('model.pickle', mode='rb') as f:
----> 2     model = pickle.load(f)

AttributeError: 'norm_gen' object has no attribute '_parse_args_stats'

In [5]:
import statsmodels
statsmodels.__version__

Out[5]:
'0.6.0.dev-311552e'
@josef-pkt

This comment has been minimized.

Copy link
Member

commented Aug 7, 2014

@hmgaudecker

I'm not able to replicate this using the above commands, and I don't see a use of scipy.stats.norm (norm_gen) in the module.
I'm using scipy.__version__ '0.13.3' but the pickling problem should exist also there.
and I'm using python 2.7 with roughly current statsmodels master.

If I give loglike as an argument
model = GenericLikelihoodModel(endog=[0, 1], exog=[0.2, 3.0], loglike=stats.norm.logpdf)
I get a pickling error already on dump but with instance method, not directly with stats.norm. (as expected)

Can you try to find where scipy.stats.norm is used in the model.

@hmgaudecker

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2014

You should always try on current master, I guess... Doing so means the problem doesn't occur on my Macbook, will try again tomorrow on my office Linux box, where it originally showed up. In case that was it: Sorry for the noise!

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Aug 7, 2014

I don't remember any recent changes in this part, so I thought it's not relevant that your checkout is a little bit older.

as aside for docs: subclassing is better than attaching user given function because it avoids instance methods which are not picklable in python.
I'm almost sure that there is no good way to attach user given function and being able to pickle the model.

@psarka

This comment has been minimized.

Copy link

commented Aug 7, 2014

It works with python 3.3 as well:

Python 3.3.3 (default, Nov 27 2013, 17:12:35) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> from statsmodels.base.model import GenericLikelihoodModel
>>> model = GenericLikelihoodModel(endog=[0, 1], exog=[0.2, 3.0])
>>> with open('model.pickle', mode='wb') as f:
...     pickle.dump(model, f)
... 
>>> with open('model.pickle', mode='rb') as f:
...     model = pickle.load(f)
... 
>>> model
<statsmodels.base.model.GenericLikelihoodModel object at 0x7f93e8047990>
>>> import statsmodels
>>> statsmodels.__version__
'0.6.0.dev-8709f00'
@hmgaudecker

This comment has been minimized.

Copy link
Contributor Author

commented Aug 8, 2014

Works with current master. Previously, I used the parent of this commit - suspect that was it.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Aug 8, 2014

@hmgaudecker Thanks for finding the commit. impact on being able to pickle was a unintended side effect.

I opened #1876 to add the unit tests, so we know when this changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.