TypeError: cannot serialize '_io.TextIOWrapper' object

[trevorwelch]

From what I can tell, this error is related to multiprocessing. I'm on OS X, MacBook Pro.

from hpsklearn import HyperoptEstimator, any_classifier
from hyperopt import tpe
import numpy as np

print(X_train_matrix.shape, y_train_matrix.shape, X_test_matrix.shape, y_test_matrix.shape, type(X_train_matrix), type(y_train_matrix), type(X_test_matrix), type(y_test_matrix))

if __name__ == '__main__':
    estim = HyperoptEstimator( classifier=xgb,  
                                algo=tpe.suggest, 
                              preprocessing=None,
                              trial_timeout=100.0,
                             
                             )

    estim.fit(X_train_matrix, y_train_matrix)

    print( estim.score(X_test_matrix, y_test_matrix) )

    print( estim.best_model() )

Full log:

(31016, 133) (31016,) (7755, 133) (7755,) <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'>
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-151-69ac7b68d098> in <module>()
     13                              )
     14 
---> 15     estim.fit(X_train_matrix, y_train_matrix)
     16 
     17     print( estim.score(X_test_matrix, y_test_matrix) )

~/hyperopt-sklearn/hpsklearn/estimator.py in fit(self, X, y, EX_list, valid_size, n_folds, cv_shuffle, warm_start, random_state, weights)
    706             increment = min(self.fit_increment,
    707                             adjusted_max_evals - len(self.trials.trials))
--> 708             fit_iter.send(increment)
    709             if filename is not None:
    710                 with open(filename, 'wb') as dump_file:

~/hyperopt-sklearn/hpsklearn/estimator.py in fit_iter(self, X, y, EX_list, valid_size, n_folds, cv_shuffle, warm_start, random_state, weights, increment)
    617                               #    so we notice them.
    618                               catch_eval_exceptions=False,
--> 619                               return_argmin=False, # -- in case no success so far
    620                              )
    621             else:

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin)
    305             verbose=verbose,
    306             catch_eval_exceptions=catch_eval_exceptions,
--> 307             return_argmin=return_argmin,
    308         )
    309 

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/base.py in fmin(self, fn, space, algo, max_evals, rstate, verbose, pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin)
    633             pass_expr_memo_ctrl=pass_expr_memo_ctrl,
    634             catch_eval_exceptions=catch_eval_exceptions,
--> 635             return_argmin=return_argmin)
    636 
    637 

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin)
    318                     verbose=verbose)
    319     rval.catch_eval_exceptions = catch_eval_exceptions
--> 320     rval.exhaust()
    321     if return_argmin:
    322         return trials.argmin

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/fmin.py in exhaust(self)
    197     def exhaust(self):
    198         n_done = len(self.trials)
--> 199         self.run(self.max_evals - n_done, block_until_done=self.async)
    200         self.trials.refresh()
    201         return self

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/fmin.py in run(self, N, block_until_done)
    171             else:
    172                 # -- loop over trials and do the jobs directly
--> 173                 self.serial_evaluate()
    174 
    175             if stopped:

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/fmin.py in serial_evaluate(self, N)
     90                 ctrl = base.Ctrl(self.trials, current_trial=trial)
     91                 try:
---> 92                     result = self.domain.evaluate(spec, ctrl)
     93                 except Exception as e:
     94                     logger.info('job exception: %s' % str(e))

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/hyperopt/base.py in evaluate(self, config, ctrl, attach_attachments)
    838                 memo=memo,
    839                 print_node_on_error=self.rec_eval_print_node_on_error)
--> 840             rval = self.fn(pyll_rval)
    841 
    842         if isinstance(rval, (float, int, np.number)):

~/hyperopt-sklearn/hpsklearn/estimator.py in fn_with_timeout(*args, **kwargs)
    580             assert fn_rval[0] in ('raise', 'return')
    581             if fn_rval[0] == 'raise':
--> 582                 raise fn_rval[1]
    583 
    584             # -- remove potentially large objects from the rval

TypeError: cannot serialize '_io.TextIOWrapper' object

Any ideas? My searching hasn't found much...

[bjkomer]

Hmm, I haven't come across this before. You're right, it looks related to multiprocessing. My guess is an open-file object is somehow getting passed to the evaluation of the classifier or being returned from it, and since that isn't picklable it's causing this error.

What classifier are you using?

[theglob]

I dont know if anybody still cares, but i had a similar issue. I figured out that in some cases an object cant be pickled when it contains a logger object. On my mac it worked, on ubuntu i got OPs Error. Why i am writing this? Because i got the hint for that issue from some problem that was barely related:
http://databasefaq.com/index.php/answer/114012/python-windows-multiprocessing-cherrypy-python-multiprocessing-multiprocessing-works-in-ubuntu-doesnt-in-windows

I just scrolled through the shit and didnt get anything, however i kept scrolling. Somewhere he mentioned some issues with windows having not the ability to fork()... and that it is related to the logger instance. So i thought ok... i give it a shot and it worked haha
still cant believe it. This could have easily cost me another day or so to figure out since nobody encountered that shit.

So yeah... hope OP has also found a solution meanwhile :D

[PolarState]

Thank you @theglob, saved me a ton of time.