'task failed to un-serialize' when run cross_val_score

[xubury]

Description

I am a python newbie and recently taking a course about ANN.I follow the course but a got stuck in one part where i need to execute the cross_val_score function.
It show the error 'BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.' It work fines when the n_jobs = 1 but it takes too much time.
so i look around the repository and found similiar issue,but the solution doesn't work for me.
I try 1. import the function from another file
2. upgrade scikit-learn to 0.20.2
3. downgrade to python 3.5 (in py3.5 it doesn't pop up the error but hangs the process)
none above works.

Steps/Code to Reproduce

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('Churn_Modelling.csv')
X = dataset.iloc[:, 3:13].values
y = dataset.iloc[:, 13].values

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X_1 = LabelEncoder()
X[:, 1] = labelencoder_X_1.fit_transform(X[:, 1])
labelencoder_X_2 = LabelEncoder()
X[:, 2] = labelencoder_X_2.fit_transform(X[:, 2])
onehotencoder = OneHotEncoder(categorical_features = [1])
X = onehotencoder.fit_transform(X).toarray()
X = X[:,1:]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from keras.models import Sequential
from keras.layers import Dense

def build_classifier(self):
    classifier = Sequential()
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu', input_dim = 11))
    classifier.add(Dense(units = 6, kernel_initializer = 'uniform', activation = 'relu'))
    classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
    classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
    return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 100)

accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

Expected Results

get the trainning result

Actual Results

exception calling callback for <Future at 0x289f6f3e048 state=finished raised BrokenProcessPool>
sklearn.externals.joblib.externals.loky.process_executor.RemoteTraceback:
'''
Traceback (most recent call last):
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 391, in process_worker
call_item = call_queue.get(block=True, timeout=timeout)
File "D:\Anaconda\lib\multiprocessing\queues.py", line 113, in get
return ForkingPickler.loads(res)
File "D:\Anaconda\lib\site-packages\keras_init.py", line 3, in
from . import utils
File "D:\Anaconda\lib\site-packages\keras\utils_init.py", line 6, in
from . import conv_utils
File "D:\Anaconda\lib\site-packages\keras\utils\conv_utils.py", line 9, in
from .. import backend as K
File "D:\Anaconda\lib\site-packages\keras\backend_init.py", line 88, in
sys.stderr.write('Using TensorFlow backend.\n')
AttributeError: 'NoneType' object has no attribute 'write'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky_base.py", line 625, in _invoke_callbacks
callback(self)
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 309, in call
self.parallel.dispatch_next()
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 731, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 759, in dispatch_one_batch
self._dispatch(tasks)
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 716, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib_parallel_backends.py", line 510, in apply_async
future = self._workers.submit(SafeFunction(func))
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky\reusable_executor.py", line 151, in submit
fn, *args, **kwargs)
File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 1022, in submit
raise self._flags.broken
sklearn.externals.joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
Traceback (most recent call last):

File "", line 3, in
accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

File "D:\Anaconda\lib\site-packages\sklearn\model_selection_validation.py", line 402, in cross_val_score
error_score=error_score)

File "D:\Anaconda\lib\site-packages\sklearn\model_selection_validation.py", line 240, in cross_validate
for train, test in cv.split(X, y, groups))

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 930, in call
self.retrieve()

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)

File "D:\Anaconda\lib\concurrent\futures_base.py", line 432, in result
return self.__get_result()

File "D:\Anaconda\lib\concurrent\futures_base.py", line 384, in __get_result
raise self._exception

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky_base.py", line 625, in _invoke_callbacks
callback(self)

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 309, in call
self.parallel.dispatch_next()

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 731, in dispatch_next
if not self.dispatch_one_batch(self._original_iterator):

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 759, in dispatch_one_batch
self._dispatch(tasks)

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\parallel.py", line 716, in _dispatch
job = self._backend.apply_async(batch, callback=cb)

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib_parallel_backends.py", line 510, in apply_async
future = self._workers.submit(SafeFunction(func))

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky\reusable_executor.py", line 151, in submit
fn, *args, **kwargs)

File "D:\Anaconda\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 1022, in submit
raise self._flags.broken

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

Versions

System:
python: 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
executable: D:\Anaconda\pythonw.exe
machine: Windows-10-10.0.17134-SP0

BLAS:
macros:
lib_dirs:
cblas_libs: cblas

Python deps:
pip: 18.1
setuptools: 39.1.0
sklearn: 0.20.2
numpy: 1.15.1
scipy: 1.1.0
Cython: 0.28.5
pandas: 0.23.4

[xubury]

I upload the csv file to my repository

so today I have tested serveral version of scikit-learn. It seems that below 19.0 it would hang the process with a shortly 100% usage of CPU and above 19.0 it will prompt this error

[xubury]

well.. I finally sovled this problem.
It turns out this issue has something to do with Ipython?(i am not sure)
I ran the script outside spyder with

if __name__=='__main__':
    classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 1)
    accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

it worked perfectly!

[BHuntington7]

Hey Thanks for this! It was super helpful!

[AndersH92]

Thank you, this saved me a lot of trouble!

[xubury]

well.. I finally sovled this problem.
It turns out this issue has something to do with Ipython?(i am not sure)
I ran the script outside spyder with

if __name__=='__main__':
    classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 1)
    accuracies = cross_val_score(estimator = classifier, X = X_train, y = y_train, cv = 10, n_jobs = -1)

it worked perfectly!

It worked because of epochs=1 but in the real code epochs=100

are you sure?I remember tried 10 epochs and it worked well.I don't think epochs has anything to do with this problem. I think it caused by multithreading compatibility of Ipython, if you ran the code in jupyter notebook or windows shell, you won't have this problem.

[xubury]

Just recreate the code in a different computer.Turns out even if you ran the script outside spyder, it will still possibly throw an error.
So the best solution is:
1.import build_classifier from a different py file and wipe out the 'self' arugment.
2.Run the script outside spyder
The situation varies from computer and OS, some people can skip the first step and have 0 error.

[blurred-machine]

For me, the same error came but it was solved when I removed
n_jobs=-1
from the code, then it worked perfectly.
Maybe there is some issue with the multithreading compatibility for parallel computing

[jagbeer18]

thank you @paras009 ,
removing n_jobs = -1 was a good idea. but, i am still unable to figure out why it didn't work previously.