Memory Exception- unable to escape from memory exception

[r06922112]

I try to using auto-Sklearn on my own dataset,
The shape of my dataset is : Train X_train/y_train : (1400014, 17) (1400014,) and running 5-fold cv (doing by myself)
Here is how I calling auto-Sklearn :
automl = autosklearn.classification.AutoSklearnClassifier(include_estimators = ['k_nearest_neighbors'],time_left_for_this_task =3600, ml_memory_limit = 16384 )

I using only one estimator and set memory_limit to 16384 MB,
But always failed due to Memory problem shown below:

[WARNING] [2018-10-08 20:20:16,950:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,032:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2018-10-08 20:20:17,111:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,194:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 12
[WARNING] [2018-10-08 20:20:17,268:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,348:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 6
[WARNING] [2018-10-08 20:20:17,422:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,491:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 3
[WARNING] [2018-10-08 20:20:17,569:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[WARNING] [2018-10-08 20:20:17,651:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- restart with less ensemle_nbest: 1
[WARNING] [2018-10-08 20:20:17,726:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] No models better than random - using Dummy Classifier!
[CRITICAL] [2018-10-08 20:20:17,806:EnsembleBuilder(1):fe48bc5e59a4ed41003d4efb5557d165] Memory Exception -- Unable to escape from memory exception

Not sure is the data are too large or wrong using the auto-Sklearn API ?

[ice-blaze]

I had the same error and by reducing the number of different classes the error disappear. (My dataset size (982840, 5))

[mfeurer]

Will be fixed in the upcoming release via ab45a25.

[ice-blaze]

@mfeurer Thank you for your work! I was wondering when should we expect next release ? I saw the 4.0.1 came out today.

[jiahuanluo]

@mfeurer I still encounter this error when running the newest version on mnist dataset which is on LibSVM website. Is it normal?

[mfeurer]

I didn't have the time to try this. I'm re-opening this issue now to remember us to look into this.

[huangwei2013]

k_nearest_neighbors eats memory ：P
may try smaller dataset or less time_left_for_this_task

[jiahuanluo]

@huangwei2013 Thanks for your suggestion. It seems it doesn't help.
But I just use half of the mnist dataset (with 30,000 instances and 780 features) and 1 classifier (SVC).
Is there anything I get wrong?

[huangwei2013]

code from the error as below：

            if safe_ensemble_script.exit_status is pynisher.MemorylimitException:
                # if ensemble script died because of memory error,
                # reduce nbest to reduce memory consumption and try it again
                if self.ensemble_nbest == 1:
                    self.logger.critical("Memory Exception -- Unable to escape from memory exception")

while you didn't set ensemble_nbest & ensemble_size (both defaulted ：1)， so only tip I can get is reduce time_left_for_this_task & ml_memory_limit to whatever you can run

[acceleratiom]

Hi, I just tryed with the last version, just downloaded now as in "install explanations".

I do have the same issue:

[WARNING] [2019-07-24 23:53:01,502:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2019-07-24 23:53:02,188:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 12
/root/anaconda3/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
/root/anaconda3/lib/python3.7/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
[WARNING] [2019-07-25 00:02:10,870:EnsembleBuilder(1):be51e08169cbe20f58ed65c52cdac4c0] Memory Exception -- restart with less ensemle_nbest: 6

I See upside that there was a fix to correct that.
And I see that I actually do have enaugh memory. 🐙

(base) root@123456789:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          23543        2752       14498           0        6292       20407

I have some questions, I hope you will forgive my ignorance 👍

• Do the last install include this fix ?
• Should I put some parameters to profit of that fix ? here is my current config :

    clf = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=int(3600*15),
        per_run_time_limit=int(3600*1.5),
        include_estimators=['MLPClassifier'],ml_memory_limit=22000
    )

• Note: I am training with some MLPClassifier
• You have others advices ? I imagine that the more we have ensemble, the best...

[moelliDo]

Unfortunately, I've got the same issue. I am using the Dataset of the BPI Challenge 2019 (https://data.4tu.nl/repository/uuid:d06aff4b-79f0-45e6-8ec8-e19730c248f1), sort it by
df = df.sort_values(by=["case concept:name", "event time:timestamp"])
restrict it by
df = df[["case concept:name", "event concept:name"]]
label encode it with
df.iloc[:, 1] = labelencoder.fit_transform(df.iloc[:, 1])
transform/unstack it with
df_activities_trans = df.set_index( ['case concept:name', df.groupby('case concept:name').cumcount() + 1]).unstack()
and fill the NaNs with
final_df = final_df.fillna(999)
such that I receive a dataframe with 251734 rows and 990 columns. However, I iteratively compute classification models by selecting e.g.
X = df_activities_trans_ml.iloc[:,0:2] (first two columns of the dataset as features) and
Y = df_activities_trans_ml.iloc[:,2] (third column as output) such that I use a dataset with 251734 rows and 3 columns.
I defined the classifier as follows:
classifier = autosklearn.classification.AutoSklearnClassifier(time_left_for_this_task=3600, per_run_time_limit=360, ml_memory_limit=10240, n_jobs=4)
and used the following train/test split
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, Y, random_state=1)
However, I always get the following error when calling classifier.fit(X_train, y_train):

[WARNING] [2020-02-04 17:01:49,717:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:49,754:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 25
[WARNING] [2020-02-04 17:01:49,947:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:49,967:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 12
[WARNING] [2020-02-04 17:01:50,190:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,215:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 6
[WARNING] [2020-02-04 17:01:50,444:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,465:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 3
[WARNING] [2020-02-04 17:01:50,628:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] No models better than random - using Dummy Score!
[WARNING] [2020-02-04 17:01:50,649:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- restart with less ensemle_nbest: 1
[CRITICAL] [2020-02-04 17:03:53,982:EnsembleBuilder(1791095845):d0a46efbd79ae759b18ea47055ab1f34] Memory Exception -- Unable to escape from memory exception

My system is a vm running Ubuntu 16.04 with 4 cores and 15032 MB RAM.

              total        used        free      shared  buff/cache   available
Mem:          15032         176       12367           8        2487       14484
Swap:             0           0           0

EDIT: Sorry, it seems like I confused the ml_memory_limit with ensemble_memory_limit. Increasing the ensemble_memory_limit seems to work for me.