Error when importing hpsklearn

[skycrazer]

Unfortunately, i'm getting the following error message when trying to import hpsklear:

import hpsklearn
Traceback (most recent call last):

File "", line 1, in
import hpsklearn

File "C:\Users\SK\hpsklearn.py", line 41, in
import skdata.iris.view

ImportError: No module named 'skdata'

Im using Spyder 3.1.4 and Python 3.5.

Appreciate any suggestions for a solution. Thanks.

[bjkomer]

I'm not sure what is causing that error, hpsklearn shouldn't depend on skdata anymore. It's possible you might be using an older version. I just tried installing the version from master into a virtualenv and it works for both Python 2 and 3.

The notebook examples still use skdata, so if you are running those you'll need to install it as well. It wasn't designed for Python 3 so it may not work with that.

There are some other examples here on the website that don't use skdata.

[skycrazer]

Im using hpsklearn 0.0.1
Is this the latest version?

[skycrazer]

I manually copied hpsklearn folder from this git to the site-packages folder from anaconda and got this message:

import hpsklearn
C:\Program Files\Anaconda3\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
"This module will be removed in 0.20.", DeprecationWarning)
diagnosis
0 35
1 12
dtype: int64
Traceback (most recent call last):

File "", line 1, in
import hpsklearn

File "C:\Users\SK\hpsklearn.py", line 41, in
import skdata.iris.view

File "C:\Program Files\Anaconda3\lib\site-packages\skdata\iris_init_.py", line 1, in
from dataset import Iris

ImportError: cannot import name 'Iris'

Still not working.. Do I need to downgrade to Python 2? Not really a solution.. many other modules wont work then.

[bjkomer]

Unfortunately the only supported way to install hyperopt-sklearn is to clone the repo and do pip install -e . within the folder. I'm planning on cleaning some things up and doing the first PyPI release sometime in the next few weeks when I have time, which should make this easier in the future (and possibly making it conda installable as well). It should be possible to install packages within Anaconda with pip. I know that installing within virtualenv works for sure.

I did some looking around and it seems there is a really old version that is conda installable floating around. I'm not sure who put that there, but it is definitely not the correct one. There is no file called hpsklearn.py in this project, so when you are importing it seems to be somehow finding this other version to get that error. I'd suggest removing it and installing the version on master with pip.

[skycrazer]

Thanks for the advice. Now I'm getting the following warning when importing hpsklearn:

">> import hpsklearn
WARN: OMP_NUM_THREADS=None =>
If you are using openblas if you are using openblas set OMP_NUM_THREADS=1 or risk subprocess calls hanging indefinitely
C:\Program Files\Anaconda3\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
"This module will be removed in 0.20.", DeprecationWarning)"

What is this all about? Any suggestions? Thanks.

[bjkomer]

The first warning is letting you know that if you are using a particular linear algebra library (OpenBLAS) you could run into problems. I've never seen it be an issue, but you can run export OMP_NUM_THREADS=1 just in case and it will make the warning go away.

The second warning is saying a few modules in sklearn are getting moved around in the future. It doesn't affect anything right now, but I'll have to update it before version 0.20 of sklearn.

[Shuyib]

Would you advice putting the command export OMP_NUM_THREADS=1 to the .bash_profile of a mac? Because i find myself repeating this step frequently on my experiments @bjkomer.

[Shuyib]

I just added it. And now it works fine.

[parampopat]

I am getting this error
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm 2017.3.2\helpers\pydev\pydev_run_in_console.py", line 53, in run_file
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2017.3.2\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/User/PycharmProjects/summer_work/hyperopt.py", line 4, in
from hpsklearn import HyperoptEstimator, svr
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\hpsklearn_init_.py", line 9, in
from .estimator import hyperopt_estimator as HyperoptEstimator
File "C:\Users\User\AppData\Local\Programs\Python\Python35\lib\site-packages\hpsklearn\estimator.py", line 27, in
import hyperopt
File "C:/Users/User/PycharmProjects/summer_work\hyperopt.py", line 4, in
from hpsklearn import HyperoptEstimator, svr
ImportError: cannot import name 'HyperoptEstimator'

[Shuyib]

Try and run this on command line for example python nameofscript.py then you can redirect your output to a file for instance python nameofscript.py > resulttrial.txt. I'm seeing your using pycharm maybe that could be the problem. On the other hand, jupyter notebooks run these fine never done it on pycharm. @parampopat

[PhanindraPanthagani]

The first warning is letting you know that if you are using a particular linear algebra library (OpenBLAS) you could run into problems. I've never seen it be an issue, but you can run export OMP_NUM_THREADS=1 just in case and it will make the warning go away.

The second warning is saying a few modules in sklearn are getting moved around in the future. It doesn't affect anything right now, but I'll have to update it before version 0.20 of sklearn.

export OMP_NUM_THREADS=1 doesn't run .Please let me know how to run this in python 3.

[bjkomer]

@PhanindraPanthagani that command is meant to be run in a terminal to set an environment variable. In order to not have to type it every time you can set it in your .bashrc (or equivalent file for your operating system) as Shuyib mentioned above. The export works for linux/mac. If you're using windows the syntax might be different depending on your command line terminal.

You can also set it using Python if you would like. Something like this near the top of your file:

import os
os.environ['OMP_NUM_THREADS'] = "1"

[PhanindraPanthagani]

@bjkomer thanks, I have set it to 1 and I am running this in jupyter notebook. Does hpsklearn work in jupyter notebook? I am getting a strange error that its not able pickle module objects as below. I cloned the latest hyperopt and latest hpsklearn from github and trying to run the code. Please find the error below. I am trying to use "AUC_ROC" as a new continous loss function instead of default Accuracy . Please let me know how to resolve this issue :

****module 'roclossfn' from 'C:\Users\pp5950\OneDrive - Lennox International, Inc\AutoML\roclossfn.py'>
(34170, 79)
(34170,)
estim is hyperopt_estimator(algo=<function suggest at 0x00000209BAB75C80>,
classifier=None, continuous_loss_fn=True, ex_preprocs=None,
fit_increment=1, fit_increment_dump_filename=None,
loss_fn=<module 'roclossfn' from 'C:\Users\pp5950\OneDrive - Lennox International, Inc\AutoML\roclossfn.py'>,
max_evals=30, preprocessing=None, refit=True, regressor=None,
seed=300,
space=<hyperopt.pyll.base.Apply object at 0x00000209C3C945C0>,
trial_timeout=300, use_partial_fit=False, verbose=False)
0%| | 0/1 [00:00<?, ?it/s, best loss: ?]

---

TypeError Traceback (most recent call last)
in
35
36 print("estim is",estim)
---> 37 estim.fit(X_trainnp, y_trainnp,n_folds=10)
38
39

~\AppData\Roaming\Python\Python36\site-packages\hpsklearn-0.0.3-py3.6.egg\hpsklearn\estimator.py in fit(self, X, y, EX_list, valid_size, n_folds, cv_shuffle, warm_start, random_state, weights)
781 increment = min(self.fit_increment,
782 adjusted_max_evals - len(self.trials.trials))
--> 783 fit_iter.send(increment)
784 if filename is not None:
785 with open(filename, 'wb') as dump_file:

~\AppData\Roaming\Python\Python36\site-packages\hpsklearn-0.0.3-py3.6.egg\hpsklearn\estimator.py in fit_iter(self, X, y, EX_list, valid_size, n_folds, cv_shuffle, warm_start, random_state, weights, increment)
691 # so we notice them.
692 catch_eval_exceptions=False,
--> 693 return_argmin=False, # -- in case no success so far
694 )
695 else:

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar)
401 catch_eval_exceptions=catch_eval_exceptions,
402 return_argmin=return_argmin,
--> 403 show_progressbar=show_progressbar,
404 )
405

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\base.py in fmin(self, fn, space, algo, max_evals, max_queue_len, rstate, verbose, pass_expr_memo_ctrl, catch_eval_exceptions, return_argmin, show_progressbar)
651 catch_eval_exceptions=catch_eval_exceptions,
652 return_argmin=return_argmin,
--> 653 show_progressbar=show_progressbar)
654
655

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\fmin.py in fmin(fn, space, algo, max_evals, trials, rstate, allow_trials_fmin, pass_expr_memo_ctrl, catch_eval_exceptions, verbose, return_argmin, points_to_evaluate, max_queue_len, show_progressbar)
420 show_progressbar=show_progressbar)
421 rval.catch_eval_exceptions = catch_eval_exceptions
--> 422 rval.exhaust()
423 if return_argmin:
424 if len(trials.trials) == 0:

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\fmin.py in exhaust(self)
274 def exhaust(self):
275 n_done = len(self.trials)
--> 276 self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
277 self.trials.refresh()
278 return self

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\fmin.py in run(self, N, block_until_done)
239 else:
240 # -- loop over trials and do the jobs directly
--> 241 self.serial_evaluate()
242
243 try:

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\fmin.py in serial_evaluate(self, N)
139 ctrl = base.Ctrl(self.trials, current_trial=trial)
140 try:
--> 141 result = self.domain.evaluate(spec, ctrl)
142 except Exception as e:
143 logger.info('job exception: %s' % str(e))

~\AppData\Roaming\Python\Python36\site-packages\docs-0.2.1-py3.6.egg\hyperopt\base.py in evaluate(self, config, ctrl, attach_attachments)
856 memo=memo,
857 print_node_on_error=self.rec_eval_print_node_on_error)
--> 858 rval = self.fn(pyll_rval)
859
860 if isinstance(rval, (float, int, np.number)):

~\AppData\Roaming\Python\Python36\site-packages\hpsklearn-0.0.3-py3.6.egg\hpsklearn\estimator.py in fn_with_timeout(*args, **kwargs)
639 th = Process(target=partial(fn, best_loss=self._best_loss),
640 args=args, kwargs=kwargs)
--> 641 th.start()
642 if conn1.poll(self.trial_timeout):
643 fn_rval = conn1.recv()

~\AppData\Local\Continuum\anaconda3new\envs\hpsklearnbuild\lib\multiprocessing\process.py in start(self)
103 'daemonic processes are not allowed to have children'
104 _cleanup()
--> 105 self._popen = self._Popen(self)
106 self._sentinel = self._popen.sentinel
107 # Avoid a refcycle if the target function holds an indirect

~\AppData\Local\Continuum\anaconda3new\envs\hpsklearnbuild\lib\multiprocessing\context.py in _Popen(process_obj)
221 @staticmethod
222 def _Popen(process_obj):
--> 223 return _default_context.get_context().Process._Popen(process_obj)
224
225 class DefaultContext(BaseContext):

~\AppData\Local\Continuum\anaconda3new\envs\hpsklearnbuild\lib\multiprocessing\context.py in _Popen(process_obj)
320 def _Popen(process_obj):
321 from .popen_spawn_win32 import Popen
--> 322 return Popen(process_obj)
323
324 class SpawnContext(BaseContext):

~\AppData\Local\Continuum\anaconda3new\envs\hpsklearnbuild\lib\multiprocessing\popen_spawn_win32.py in init(self, process_obj)
63 try:
64 reduction.dump(prep_data, to_child)
---> 65 reduction.dump(process_obj, to_child)
66 finally:
67 set_spawning_popen(None)

~\AppData\Local\Continuum\anaconda3new\envs\hpsklearnbuild\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
58 def dump(obj, file, protocol=None):
59 '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60 ForkingPickler(file, protocol).dump(obj)
61
62 #

TypeError: can't pickle module objects****

[agatheLB-elmy]

As Hyperopt-sklearn needs a specific installation (pip install git+https://github.com/hyperopt/hyperopt-sklearn, see doc), there is an additional manipulation to install the package in Venv with Pycharm.
Add pip instruction in Options box.