SARIMAX gives error on model.save(): "SystemError: error return without exception set"

[dchirikov]

Hi,

I am observind an issue on deep SARIMA(X) model:

Dep. Variable:  y                                   No. Observations:   11664
Model:          SARIMAX(5, 0, 21)x(5, 1, 6, 24)     Log Likelihood      4285.492
Date:           Mon, 09 Oct 2017                    AIC                 -8494.984
Time:           23:36:52                            BIC                 -8215.142

This is quite big dataset, and if I limit Q/q and P/p to smaller values (1..2) model can be saved. But with described parameters I see the following #trace.

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-29-e65397e0cf94> in <module>()
----> 1 model.save('testmodel2.p')

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/site-packages/statsmodels/base/wrapper.pyc in save(self, fname, remove_data)
     70             self.remove_data()
     71 
---> 72         save_pickle(self, fname)
     73 
     74     @classmethod

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/site-packages/statsmodels/iolib/smpickle.pyc in save_pickle(obj, fname)
     13     """
     14     with get_file_obj(fname, 'wb') as fout:
---> 15         cPickle.dump(obj, fout, protocol=-1)
     16 
     17 

SystemError: error return without exception set

version of statsmodel is 0.8.0

[josef-pkt]

This might be difficult to debug without a test case.
Based on a quick search, I don't find any specific limitation on pickle size.

Can you try to pickle or cPickle dump it directly instead of going through the save method. And maybe try different pickle options. SystemError might indicate a operating system limitation.

[josef-pkt]

@ChadFulton ?

[dchirikov]

pickle.dump(model, open('testmodel3.p' ,'a')) gives typeError: can't pickle zStatespace objects

Full stack trace

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-44-cea77e5efb40> in <module>()
----> 1 pickle.dump(model, open('testmodel3.p' ,'a'))

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in dump(obj, file, protocol)
   1374 
   1375 def dump(obj, file, protocol=None):
-> 1376     Pickler(file, protocol).dump(obj)
   1377 
   1378 def dumps(obj, protocol=None):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in dump(self, obj)
    222         if self.proto >= 2:
    223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
    225         self.write(STOP)
    226 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    304             reduce = getattr(obj, "__reduce_ex__", None)
    305             if reduce:
--> 306                 rv = reduce(self.proto)
    307             else:
    308                 reduce = getattr(obj, "__reduce__", None)

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle zStatespace objects

[dchirikov]

pickle.dump(model, open('testmodel3.p' ,'a'), 2) returns something more interesting:
error: 'i' format requires -2147483648 <= number <= 2147483647

Full trace:

---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-47-b2ee66e58109> in <module>()
----> 1 pickle.dump(model, open('testmodel3.p' ,'a'), 2)

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in dump(obj, file, protocol)
   1374 
   1375 def dump(obj, file, protocol=None):
-> 1376     Pickler(file, protocol).dump(obj)
   1377 
   1378 def dumps(obj, protocol=None):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in dump(self, obj)
    222         if self.proto >= 2:
    223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
    225         self.write(STOP)
    226 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    685                 for k, v in tmp:
    686                     save(k)
--> 687                     save(v)
    688                 write(SETITEMS)
    689             elif n:

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    685                 for k, v in tmp:
    686                     save(k)
--> 687                     save(v)
    688                 write(SETITEMS)
    689             elif n:

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    685                 for k, v in tmp:
    686                     save(k)
--> 687                     save(v)
    688                 write(SETITEMS)
    689             elif n:

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_tuple(self, obj)
    566         write(MARK)
    567         for element in obj:
--> 568             save(element)
    569 
    570         if id(obj) in memo:

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/home/dmitry/miniconda3/envs/py2_intel/lib/python2.7/pickle.pyc in save_string(self, obj, pack)
    490                 self.write(SHORT_BINSTRING + chr(n) + obj)
    491             else:
--> 492                 self.write(BINSTRING + pack("<i", n) + obj)
    493         else:
    494             self.write(STRING + repr(obj) + '\n')

error: 'i' format requires -2147483648 <= number <= 2147483647

[josef-pkt]

The last error error: 'i' format requires -2147483648 <= number <= 2147483647 looks like something I have seen in my google search, maybe on stackoverflow searching for size limit in pickle.
I didn't see any explanations, mostly recommendation to avoid python pickle for large data.

[dchirikov]

But how to check here if data is big enough? SARIMAX(1, 0, 1)x(1, 1, 1, 24) dumps successfully.
Currently I am trying to reproduce it on Python 3.5 and with minimal amount of dependencies.

[ChadFulton]

I agree that this sounds like trying to pickle too much data. In your original specification you have a state vector of dimension 190, which means that some of the output arrays are pretty big.

For example, predicted_state_cov has dimension k_states, k_states, nobs, which is 190 x 190 x 11664 and my back of the envelope calculation suggests that's about 3 gigabytes of data.

[dchirikov]

Fair enough about calculations. It is just weird, that the data which fits to memory, doesn't fit* to disk.

*can not be saved

[ChadFulton]

The original error sounds like it might be related to numpy/numpy#2396

[ChadFulton]

Also, pickle will make much larger objects than exist in memory - for example, if create np.ones((190, 190, 5000)), then I use about 1.5gb of memory but the pickle's size is 5.23 gb.

[ChadFulton]

You may want to consider using remove_data=True in your save call, although of course that would remove many results objects (anything with some dimension of nobs).

[bashtage]

A common problem with large arrays: see numpy/numpy#2396

[dchirikov]

Yes, seems you are right. I managed to save model in Python 3.5, though. Size of the file - 13G.
Issue can be closed, I guess.
Thank you for your help.

[josef-pkt]

I'm keeping it open as a FAQ issue, given that it's the first time we get those problems and explanations.

[josef-pkt]

Separate from the technical question in this issue:

@dchirikov What's your use case? Why do you want or need to pickle?

I think in the long run we should get away from pickling as the only or recommended way to reuse the model estimate for, for example, prediction.
If it's cheap to rebuild a model or if we have standalone predict/filter functions, then we need to store/pickle only the relevant parameters, plus whatever else we need in the statespace case.

[dchirikov]

@josef-pkt To be honest I am relatively new in statistics/econometrics and can't say I am really familiar with other ways of reusing models (if any). But in my case I need to compute models for 15 timeseries arrays and every model is computing 2-5 hours on pretty powerful HW. So in order not to lose those hours I need to back them up somehow. I will use predict for sure and fittedvalues to estimate quality and/or graphs.

[josef-pkt]

@dchirikov Thanks for the reply.

I guess pickle is still the best choice for backing up a full model and results.
If the main time is spend in estimation/fit and setting up a model is not too expensive, then just saving the params and using them as start_params for a new fit might work.

[ChadFulton]

If the main time is spend in estimation/fit and setting up a model is not too expensive, then just saving the params and using them as start_params for a new fit might work.

And if you want to evaluate the model at exactly a saved set of parameters (i.e. you don't need to re-fit), then you can just do res = mod.smooth(saved_params) if you need smoothed results or res = mod.filter(saved_params) if you just need fittedvalues and predict.

[bashtage]

Closing as answered.