UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. "timeout or by a memory leak."

[kaustubh-ai]

Description

Fitting a GridSearchCV model with n_jobs = -1 gives the UserWarning mentioned below, while for n_jobs = 1 it runs without any warnings

Steps/Code to Reproduce

pipeline = Pipeline([('vect', CountVectorizer(tokenizer=tokenise)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(RandomForestClassifier(n_estimators=20, n_jobs=-1)))])

parameters = {'clf__estimator__max_depth': [10, 15],
              'clf__estimator__n_estimators': [100, 300]}
              
model = GridSearchCV(pipeline, parameters, make_scorer(fbeta_score, beta=2, average='macro'), -1, cv=2, verbose=3)

model.fit(X_train, y_train)

Expected Results

No error is thrown

Actual Results

/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. "timeout or by a memory leak.", UserWarning

Versions

System:
python: 3.6.8 (default, Jan 14 2019, 11:02:34) [GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
executable: /usr/bin/python3
machine: Linux-4.14.79+-x86_64-with-Ubuntu-18.04-bionic

Python deps:
pip: 19.2.1
setuptools: 41.0.1
sklearn: 0.21.3
numpy: 1.16.4
scipy: 1.3.0
Cython: 0.29.13
pandas: 0.24.2

[TomDLT]

Thanks for the report, yet it is not an error but a warning.
It should not prevent your code from running, nor should it lead to different results.
Feel free to reopen this issue if you get different results.

[yiorgos1973]

I have this error frequently when using all processors or almost all processors. Anaconda IDEs have a little matter in memory allocation and I suspect that this has also to do with that. Of course it is not an error and the results are not alterd.

[anjani-dhrangadhariya]

n_jobs=-1 causes the model to use as many available CPU resources as possible. It is better to keep this variable to -10 or even lesser in case you are sharing a server with your lab members. More information here.

[sharpe5]

In my case, exact same error, posted the fix on Stack Overflow:

# Increase timeout (tune this number to suit your use case).
timeout=99999
result_chunks = joblib.Parallel(n_jobs=njobs, timeout=timeout)(joblib.delayed(f_chunk)(i) for i in n_chunks)

[jtlz2]

@sharpe5 That's not quite the same because this is bubbling up from within sklearn - how would you apply that fix in the context of sklearn without hacking the codebase?