Segmentation fault

[ChangbingChen]

when try the sample code:

import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
X, y = sklearn.datasets.load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)
automl = autosklearn.classification.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, y_hat))

something wrong with the automl.fit line, and got "Segmentation fault", the python command console exit.

>automl.fit(X_train, y_train)
/home/work/.pyenv/versions/py3_env/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
[WARNING] [2019-05-29 16:26:32,672:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-29 16:26:32,680:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
Segmentation fault
(py3_env) [work@*** ~]$ [WARNING] [2019-05-29 16:26:34,685:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-29 16:26:36,689:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!

does anyone come across this problem?

[mfeurer]

I'm afraid that this is a very uncommon problem. Could you please provide further information on the OS you're using and how you installed Auto-sklearn?

Also, could you please upload the log file created by Auto-sklearn?

[ChangbingChen]

os info：

LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.2.1511 (Core) 
Release:        7.2.1511
Codename:       Core

python version:

Python 3.6.6 (default, Oct 24 2018, 14:44:15) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

installation：
a）make compile and install swig-4.0.0.tar.gz
b）pip install auto-sklearn

pip list:

Package                       Version 
----------------------------- --------
alabaster                     0.7.12  
auto-sklearn                  0.5.2   
Babel                         2.7.0   
certifi                       2019.3.9
chardet                       3.0.4   
ConfigSpace                   0.4.10  
Cython                        0.29.9  
docutils                      0.14    
idna                          2.8     
imagesize                     1.1.0   
Jinja2                        2.10.1  
joblib                        0.13.2  
liac-arff                     2.4.0   
lockfile                      0.12.2  
MarkupSafe                    1.1.1   
nose                          1.3.7   
numpy                         1.16.4  
packaging                     19.0    
pandas                        0.24.2  
pip                           10.0.1  
psutil                        5.6.2   
Pygments                      2.4.2   
pynisher                      0.5.0   
pyparsing                     2.4.0   
pyrfr                         0.7.4   
python-dateutil               2.8.0   
pytz                          2019.1  
PyYAML                        5.1     
requests                      2.22.0  
scikit-learn                  0.19.2  
scipy                         1.3.0   
setuptools                    39.0.1  
six                           1.12.0  
smac                          0.8.0   
snowballstemmer               1.2.1   
Sphinx                        2.0.1   
sphinx-rtd-theme              0.4.3   
sphinxcontrib-applehelp       1.0.1   
sphinxcontrib-devhelp         1.0.1   
sphinxcontrib-htmlhelp        1.0.2   
sphinxcontrib-jsmath          1.0.1   
sphinxcontrib-qthelp          1.0.2   
sphinxcontrib-serializinghtml 1.1.3   
typing                        3.6.6   
urllib3                       1.25.3  
xgboost                       0.90   

test.py:

import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
X, y = sklearn.datasets.load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)
automl = autosklearn.classification.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, y_hat))

nohup python test.py &
nohup file cotennt:

/home/work/.pyenv/versions/py3_env/lib/python3.6/site-packages/sklearn/ensemble/weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
  from numpy.core.umath_tests import inner1d
/home/work/.pyenv/versions/py3_env/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
[WARNING] [2019-05-30 11:11:50,584:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:11:50,592:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:11:52,596:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:11:54,600:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:11:56,604:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:11:58,608:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:00,612:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:02,616:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:04,620:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:06,624:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:08,628:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:10,631:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:12,635:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:14,639:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:16,643:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:18,647:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:20,651:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:22,654:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:24,658:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:26,662:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:28,666:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:30,670:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:32,673:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:34,677:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:36,681:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:38,685:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:40,689:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:42,692:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:44,696:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:46,700:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:48,704:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:50,707:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:52,711:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:54,715:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:56,718:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:12:58,722:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:13:00,727:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:13:02,730:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:13:04,734:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:13:06,738:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-30 11:13:08,741:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!

[ChangbingChen]

automl_d74860caaa557f473ce23908ff7ba369.txt

The attached file is the detailed log file with debug level.

[mfeurer]

Thanks for the additional information. Unfortunately, the logs do not provide any further detail except that Auto-sklearn did not start training a single machine learning model. To further understand the problem, for how long does Auto-sklearn run, and when does the segfault happen? Also, would you be able to test Auto-sklearn in a conda environment to ensure that there's no issue with the installation?

[wangmoyu]

we meet the same problem also: in 2nd round, we meet segment error

[ChangbingChen]

It seems to work, thx~

1. install anaconda3, wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh and run it
2. conda activate base, and conda install gxx_linux-64 gcc_linux-64 swig

(base) [work@** anaconda3]$ which python
~/anaconda3/bin/python
(base) [work@** anaconda3]$ python
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

3. pip install auto-sklearn
4. run the sample code, and about one hour later got the result 0.9911

[mfeurer]

Thank you very reporting a solution to this issue.

Could you please do me a favor and check the swig version installed with conda? I suspect that this is an issue of the recent SWIG4.

[ChangbingChen]

perhaps, this swig version is 3.0.12

(base) [work@** ~]$ which swig
~/anaconda3/bin/swig
(base) [work@** ~]$ swig -version

SWIG Version 3.0.12

Compiled with x86_64-conda_cos6-linux-gnu-c++ [x86_64-conda_cos6-linux-gnu]

Configured options: +pcre

Please see http://www.swig.org for reporting bugs and further information

[mfeurer]

Thanks a lot for checking. I'll close this issue as we have found a solution.

@sfalkner do you happen to know if the pyrfr behaves differently with SWIG4?

[IncubatorShokuhou]

@mfeurer I also have the "Segmentation fault" problem , and it remains unfixed after I install the newest anaconda3 and downgrade the version of swig to 3.0.12.

[Stomach-ache]

I also encounter this problem, and unfortunately, the above solution seems doesn't work for me.

[adithyabsk]

I can add another data point on this issue, running into the same issue here. Tried downgrading to swig3 as well. (I am on a Mac, though)

[adithyabsk]

As an update, installing through conda seems to work (I'd be happy to post instructions if anyone needs it). My follow up question would be one for @mfeurer. Why is that so? If swig is installed through brew, I would assume that the package should be no difference no?

[adithyabsk]

I think I've isolated the issue, the segfault seems to be occuring when using the pyrfr version that the package currently specifies (0.7.4). If the package version is bumped to the latest version (0.8.0) I am not getting a segfault. @mfeurer could we get an intermediate patch with a bumped pyrfr version?

[dylancashman]

On a Mac, and I'm segfaulting as well. I've tried both downgrading swig as well as trying the package with pyrfr at version 0.8.0, no luck with either.

@adithyabsk can you provide more information about what worked for you?

edit: If anyone is stuck on this and just wants to get it working, I ended up just running the auto-sklearn docker container and trying it out in there, it ran fine.

[PGijsbers]

I had the same issue, but fixed it with help from this issue. Adding information in case it helps anyone.

OS: Windows Subsystem for Linux (WSL) on Windows 10, Python 3.7.3

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

I worked within a conda environment. Initially installed the latest swig (4.0.0) manually as well as the other requirements, then installed with pip install auto-sklearn (meaning I got pyrfr==0.7.4). No problems detected during installation. However, at runtime it failed the same way on the 'holdout example' (immediate segfault, no models).

Changed to gcc and swig with conda install gxx_linux-64 gcc_linux-64 swig. It installed swig 3.0.12. Fail again.

Finally upgraded pyrfr to 0.8.0 and the example problem ran to completion (Accuracy score ~0.951).

[adithyabsk]

@dylancashman My setup is the exact same as @PGijsbers but I'm on macOS, I think the key piece that you might be missing is swig3.0 which is the version that seems to be working with auto sklearn.

[JavierClearImageAI]

I followed all the thread and seems to be working now (fingers crossed).
Just a comment. Upgrading to pyrfr 0.8.0 was key, but having the 0.7.4 installed pip didn't want to install it ($ pip install pyrfr --> Requirement already satisfied: pyrfr in /home/javier/anaconda3/envs/autoTrain/lib/python3.6/site-packages (0.7.4)), although theoretically this should install the latest version. Not being able to find the pyrfr uninstalling commands I just ended up deleting it from the directory: anaconda3/envs/autoTrain/lib/python3.6/site-packages (probably not a good practice, but I was a bit desperate and it worked fine). I am using Python 3.6, since it seems the most compatible with other ML frameworks.

Finally I installed pyrfr from scratch. It showed this message:

Collecting pyrfr
  Using cached https://files.pythonhosted.org/packages/ed/0f/4d7e42a9dfef3a1898e03cffa8f1cfcd1f96507d718808b2db584c6f8401/pyrfr-0.8.0.tar.gz
Building wheels for collected packages: pyrfr
  Building wheel for pyrfr (setup.py) ... done
  Created wheel for pyrfr: filename=pyrfr-0.8.0-cp36-cp36m-linux_x86_64.whl size=437076 sha256=dee332a1d73f45f55c55155b23ffc74c34dd84f2d65bae2447e940d3c490dd95
  Stored in directory: /home/javier/.cache/pip/wheels/61/1a/d2/b5aee388a492a01946143d3c976b2ca810af537480e1f16999
Successfully built pyrfr
ERROR: auto-sklearn 0.5.2 has requirement pyrfr<0.8,>=0.7, but you'll have pyrfr 0.8.0 which is incompatible.
Installing collected packages: pyrfr
Successfully installed pyrfr-0.8.0

It works fine finally 👍

PD: while training, it shows this, though:

  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)

[mfeurer]

If swig is installed through brew, I would assume that the package should be no difference no?

It might not work with the compiler installed via conda, or be incompatible with the python binary from conda (as that might have been compiled with an incompatible compiler).

In total, this issue appears to be solved by using swig from conda, making sure to use swig version 3.X and by using pyrfr 0.8.

[ballcap231]

On a Mac, and I'm segfaulting as well. I've tried both downgrading swig as well as trying the package with pyrfr at version 0.8.0, no luck with either.

@adithyabsk can you provide more information about what worked for you?

edit: If anyone is stuck on this and just wants to get it working, I ended up just running the auto-sklearn docker container and trying it out in there, it ran fine.

I couldn't run it on the docker container. I just got errors of "No module named 'numpy.random._pickle'" when I tried to do prediction using a model I had built before. Did you install/prepare anything else?

Full traceback:

root@006ed697a386:/mnt/mydata/AS_inf_env# python3 AS_field_WSOG_inf.py
Traceback (most recent call last):
File "AS_field_WSOG_inf.py", line 18, in
PO_model = load_joblib(PO_folder)
File "AS_field_WSOG_inf.py", line 17, in
load_joblib = lambda folder: joblib.load(glob.glob(path.join(folder,'*.joblib'))[0])
File "/usr/local/lib/python3.6/dist-packages/joblib/numpy_pickle.py", line 598, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/usr/local/lib/python3.6/dist-packages/joblib/numpy_pickle.py", line 526, in _unpickle
obj = unpickler.load()
File "/usr/lib/python3.6/pickle.py", line 1050, in load
dispatchkey[0]
File "/usr/lib/python3.6/pickle.py", line 1338, in load_global
klass = self.find_class(module, name)
File "/usr/lib/python3.6/pickle.py", line 1388, in find_class
import(module, level=0)
ModuleNotFoundError: No module named 'numpy.random._pickle'

EDIT: Nevermind, i think this was just because I was using a different version

[rupeshchandgude]

OS : Ubuntu 14.04
I had same issue, I tried every suggestion mentioned in this issue. Install SWIG using conda also, but problem is not solved. Seems like there was SWIG installation issue in Ubuntu 14.04 , as in ubuntu 14.04 if we install swig using sudo apt-get install swig ,it install swig2 and for autosklearn I think SWIG 3 and above are required. Then I install swig using above command in another machine with Ubuntu 16.04 , SWIG 3 was installed and same codes run successfully without any error. So my question is, can we use autosklearn in ubuntu 14.04?

[mfeurer]

Yes, the pyrfr requires SWIG version 3. What's not working if you install SWIG with Conda? Also, can you do sudo apt-get install swig3 on Ubuntu 14.04?

[d2dugas]

Had the same issues on archlinux, been up- an downgrading pyrfr, swig, etc. whatsoever without success. Finally a working procedure as a combination of the above answers (not sure if first pacman/ln step is required):

pacman -S swig3
ln -s /usr/bin/swig-3 /usr/bin/swig

conda create -n test python=3.6 gxx_linux-64 gcc_linux-64 swig=3 && conda activate test

curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install
pip install auto-sklearn

[andreysfc]

I had the same issues on CentOS 7.5 (Core)

A bit confused here. What is the difference between swig from package installation (such as yum or apt) and conda?. I have both different version with conda using version 3 and yum using version 4.

Try to follow everything above without any success.

[mmaybeno]

I wanted to chime in here since I was having swig issues when using the current jupyter notebook docker image jupyter/scipy-notebook:latest and thought it could be helpful to some. Originally I was installing swig via apt-get which installs version 4 (the base image is ubuntu 20.04). When I ran auto-sklearn in a jupyter notebook, the seg fault was being swallowed and expressed as zmq.error.ZMQError: Address already in use. But based on the installation notes to use conda for installing swig, I was able to install and run auto-sklearn successfully. Below is the Dockerfile that worked for me. Swig is still version 4, but doing everything with conda resolved my issues.

FROM jupyter/scipy-notebook:latest

USER root

RUN apt-get update && apt-get install -y curl

RUN conda install --yes gxx_linux-64 gcc_linux-64 swig

RUN curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install

RUN pip install auto-sklearn

RUN fix-permissions $CONDA_DIR && fix-permissions /home/$NB_USER

USER $NB_USER

[mfeurer]

Thanks for the suggestion @mmaybeno. In case it helps, we also provide a docker image of Auto-sklearn at https://github.com/automl/auto-sklearn/packages/ So far it is only available for the development version, but we'll provide a version for the master branch with the next release.

[mmaybeno]

Ah excellent news, thanks @mfeurer. I keep forgetting github has docker images now, as I only looked at dockerhub and noticed the latest image was 2 years old.

[mfeurer]

Thanks for letting us know. I just added a push to dockerhub to our docker building github action as docker hub is used more often.