GenericLikelihoodModel is not picklable #1873

Closed
hmgaudecker opened this Issue Aug 7, 2014 · 6 comments

Projects

None yet

3 participants

@hmgaudecker
Contributor

Apparently scipy.stats.norm is being attached to GenericLikelihoodModel causing pickle.load() to fail.

Minimal example below, also see the mailing list discussion

In [1]: 
import pickle
from statsmodels.base.model import GenericLikelihoodModel

In [2]:
model = GenericLikelihoodModel(endog=[0, 1], exog=[0.2, 3.0])

In [3]:
with open('model.pickle', mode='wb') as f:
    pickle.dump(model, f)

In [4]:
with open('model.pickle', mode='rb') as f:
    model = pickle.load(f)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-d8739e3ead39> in <module>()
      1 with open('model.pickle', mode='rb') as f:
----> 2     model = pickle.load(f)

AttributeError: 'norm_gen' object has no attribute '_parse_args_stats'

In [5]:
import statsmodels
statsmodels.__version__

Out[5]:
'0.6.0.dev-311552e'
@josef-pkt
Member

@hmgaudecker

I'm not able to replicate this using the above commands, and I don't see a use of scipy.stats.norm (norm_gen) in the module.
I'm using scipy.__version__ '0.13.3' but the pickling problem should exist also there.
and I'm using python 2.7 with roughly current statsmodels master.

If I give loglike as an argument
model = GenericLikelihoodModel(endog=[0, 1], exog=[0.2, 3.0], loglike=stats.norm.logpdf)
I get a pickling error already on dump but with instance method, not directly with stats.norm. (as expected)

Can you try to find where scipy.stats.norm is used in the model.

@hmgaudecker
Contributor

You should always try on current master, I guess... Doing so means the problem doesn't occur on my Macbook, will try again tomorrow on my office Linux box, where it originally showed up. In case that was it: Sorry for the noise!

@josef-pkt
Member

I don't remember any recent changes in this part, so I thought it's not relevant that your checkout is a little bit older.

as aside for docs: subclassing is better than attaching user given function because it avoids instance methods which are not picklable in python.
I'm almost sure that there is no good way to attach user given function and being able to pickle the model.

@psarka
psarka commented Aug 7, 2014

It works with python 3.3 as well:

Python 3.3.3 (default, Nov 27 2013, 17:12:35) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> from statsmodels.base.model import GenericLikelihoodModel
>>> model = GenericLikelihoodModel(endog=[0, 1], exog=[0.2, 3.0])
>>> with open('model.pickle', mode='wb') as f:
...     pickle.dump(model, f)
... 
>>> with open('model.pickle', mode='rb') as f:
...     model = pickle.load(f)
... 
>>> model
<statsmodels.base.model.GenericLikelihoodModel object at 0x7f93e8047990>
>>> import statsmodels
>>> statsmodels.__version__
'0.6.0.dev-8709f00'
@hmgaudecker
Contributor

Works with current master. Previously, I used the parent of this commit - suspect that was it.

@hmgaudecker hmgaudecker closed this Aug 8, 2014
@josef-pkt
Member

@hmgaudecker Thanks for finding the commit. impact on being able to pickle was a unintended side effect.

I opened #1876 to add the unit tests, so we know when this changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment