model.fit ValueError: I/O operation on closed file

[panw]

Any idea what could be causing this error? I've been trying to solve this for a week. Thanks in advance.

Train on 60816 samples, validate on 15204 samples
Epoch 1/20
60816/60816 [==============================] - 19s - loss: 0.1665 - acc: 0.9597 - val_loss: 0.1509 - val_acc: 0.9605
Epoch 2/20
31200/60816 [==============>...............] - ETA: 8s - loss: 0.1583 - acc: 0.9600�����������������������������������������������������������������������������������
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-e3590f9fe5e6> in <module>()
     16 
     17 my_model = model.fit(train_x, train_y, batch_size=100, nb_epoch=20,                       
---> 18                      show_accuracy=True, verbose=1, validation_data=(test_x, test_y))
     19 score = model.evaluate(test_x, test_y, show_accuracy=True, verbose=0)
     20 print('Test loss:', score[0])

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in fit(self, X, y, batch_size, nb_epoch, verbose, callbacks, validation_split, validation_data, shuffle, show_accuracy, class_weight, sample_weight)
    644                          verbose=verbose, callbacks=callbacks,
    645                          val_f=val_f, val_ins=val_ins,
--> 646                          shuffle=shuffle, metrics=metrics)
    647 
    648     def predict(self, X, batch_size=128, verbose=0):

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in _fit(self, f, ins, out_labels, batch_size, nb_epoch, verbose, callbacks, val_f, val_ins, shuffle, metrics)
    284                     batch_logs[l] = o
    285 
--> 286                 callbacks.on_batch_end(batch_index, batch_logs)
    287 
    288                 epoch_logs = {}

/usr/local/lib/python2.7/dist-packages/keras/callbacks.pyc in on_batch_end(self, batch, logs)
     58         t_before_callbacks = time.time()
     59         for callback in self.callbacks:
---> 60             callback.on_batch_end(batch, logs)
     61         self._delta_ts_batch_end.append(time.time() - t_before_callbacks)
     62         delta_t_median = np.median(self._delta_ts_batch_end)

/usr/local/lib/python2.7/dist-packages/keras/callbacks.pyc in on_batch_end(self, batch, logs)
    168         # will be handled by on_epoch_end
    169         if self.verbose and self.seen < self.params['nb_sample']:
--> 170             self.progbar.update(self.seen, self.log_values)
    171 
    172     def on_epoch_end(self, epoch, logs={}):

/usr/local/lib/python2.7/dist-packages/keras/utils/generic_utils.pyc in update(self, current, values)
     59             prev_total_width = self.total_width
     60             sys.stdout.write("\b" * prev_total_width)
---> 61             sys.stdout.write("\r")
     62 
     63             numdigits = int(np.floor(np.log10(self.target))) + 1

/usr/local/lib/python2.7/dist-packages/ipykernel/iostream.pyc in write(self, string)
    315 
    316             is_child = (not self._is_master_process())
--> 317             self._buffer.write(string)
    318             if is_child:
    319                 # newlines imply flush in subprocesses

ValueError: I/O operation on closed file

[artix41]

I had the same problem (also with IPython), and I solved it by adding a time.sleep(0.1) the line just after model.fit. It seems Python (or just IPython?) needs a time before two fitting...

[fchollet]

This looks like an IO bug with IPython, you might want to file a bug with
the devs. It seems to be triggered by the Keras progbar's use of sys.stdout.

On 3 April 2016 at 07:35, artix41 notifications@github.com wrote:

I had the same problem (also with IPython), and I solved it by adding a
time.sleep(0.1) the line just after model.fit. It seems Python (or just
IPython?) needs a time before two fitting...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2110 (comment)

[panw]

@fchollet yea that's what I found as well. Thanks for looking into it.

[luthfianto]

@fchollet Right, I'm experiencing the same with Spyder (because it uses IPython kernel too). With each epoch completes at 2s, it results on I/O operation error too. verbose = 0 does the trick. Thanks.

[AloneGu]

@rilut thanks for the trick!!

[Sohojoe]

@artix41 - Thanks! adding the time.sleep(0.1) worked for me

[statchaitya]

@rilut Setting verbose = 0 strangely results in the termination of my run in a single epoch. Any idea what is happening here? I added time.sleep(0.1) too. Isn't helping much. I am still trying to find a workaround :(

[cesarsouza]

Instead of writing to sys.stdout directly, could those calls be wrapped on an auxiliary function with a try/catch block to help prevent the entire learning from breaking down if/when this happens?

Although it is clear this is not the best solution, I have to say it is very depressing to find that after 50h of training your model was lost simply because some text could not be printed 😞

I am glad ModelCheckpoint works very well, though!

[saurav-joshi]

I get ethe same error when I am trying to call Convoluted Neural Network (CNN) via Keras library using the best estimated GridSearchCV estimator..

I cannot set the verbose parameter to false false since it is called from via GridSearchCV and cannot set the time lag as well for the same reason..

I am pasting my code snippet for reference..

from sklearn.grid_search import GridSearchCV
from sklearn.metrics import make_scorer
import numpy

grid search epochs, batch size and optimizer

optimizers = ['rmsprop', 'adam']
init = ['glorot_uniform', 'normal', 'uniform']
epochs = numpy.array([50, 100, 150])
batches = numpy.array([5, 10, 20])

Create the parameters to tune

parameters = dict(optimizer=optimizers, nb_epoch=epochs, batch_size=batches, init=init)

Initialize the classifier

clf = KerasClassifier(build_fn=ConvulatedNeuralNet)

Make an f1 scoring function using 'make_scorer'

f1_scorer = make_scorer(f1_score, pos_label = 1)

Perform grid search on the classifier using the f1_scorer as the scoring method

grid_obj = GridSearchCV(estimator=clf, param_grid=parameters, verbose = 0)

Get the estimator

#clf = grid_obj.best_estimator_

#Hot encode Yes,No to 1,0 else the library would throw error..
y_encoded = pd.DataFrame(y_all).replace(['yes', 'no'], [1, 0])

Fit the grid search object to the training data and find the optimal parameters

datasets = [train_test_split(X_all, y_encoded, train_size=x, test_size=95) for x in [100, 200, 300]]
for data in datasets:
X_train, X_test, y_train, y_test = data
grid_obj = grid_obj.fit(X_train.as_matrix(), y_train.as_matrix())
time.sleep(0.1)

[Diyago]

Where should I use time.sleep(0.1), please give some example. I faced the same problem...

[saurav-joshi]

I don't think you can set the timeout if you are using GridSerachCV since the classifier is called from within GridSearchCV and not by the user...

I tried the katest version of IPython, released two days back, and still have this issue..

You can try few alternatives..

1. Reduce number of epochs and batch size.. This would reduce number of iterations and hence the number of log messages...
2. Change the development environment-- say PyCharm-- which is bug free...

[psilva07]

I also have the same problem on IPython but it only arises if I do something on the notebook (create , delete, or modify a cell) during the execution of the fit function. Otherwise it works fine.

[Harley1980]

I fix this problem with adding %%capture as the first line of the cell

[Givemeyourgits]

Any update on this? I've been seeing the same problem on Windows 64 Anaconda 2.7 Jupyter Notebook and Lab.

[Diyago]

@Givemeyourgits you may try to update jupyter notebook, current version is 4+ and someone says that the problem was fixed, didn't help for me. So I recommend to set verbose =0

[Givemeyourgits]

Running latest version of Notebook for 2.7.x

[paulovn]

As the IPython link says (ipython/ipython#9168) this seems to be fixed in ipython trunk, but not yet in a released version (will be in version 4.4).

In the meantime, I've made a quick hack simply by wrapping the update() method in the keras ProgBar class in a simple try/except block. It seems to work (I don't get ValueError anymore and processes in Jupyter notebook finish correctly, event with verbose activated). Making a patch would be simple.

[shachar-i]

4.4 is out. Indeed fixed this.
conda install ipykernel to upgrade if you are on anaconda, like me.

I think this can be closed.

[thecolorblue]

I just ran into this issue with ipykernel 4.4.1. I am working from this example with a 2gb csv file. I only get around 2-3m lines in before I get ValueError: I/O operation on closed file error.

[vijaymanikandan]

I had this problem too and setting verbose=0 in the argument of model.fit() seems to have fixed it.

[jf003320018]

I use conda install ipykernel advised by @shachar-i, my problem is fixed.
Setting verbose=0 is not a good way.

[ymcdull]

Thanks @shachar-i , my problem fixed too. One point that may be helpful to others is that you need to check if your jupyter is installed by conda or pip. You can check with:
which jupyter

For me, I used pip to install jupyter, so use pip to install ipykernel as well:
pip install -U ipykernel

[mekustonee]

conda install ipykernel solved it for me too, thank @shachar-i and @jf003320018

[NicGian]

I experienced too this problem. As workaround I set verbose=2 in the argument of model.fit() and it has fixed it. Moreover setting verbose =2 at least logs the epoch and accuracy of the training compared to verbose = 0 which doesn't log anything.

[fehiepsi]

I can confirm that verbose=2 fixes my case.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, but feel free to re-open it if needed.

[wasil1]

how can solve this value error???

[KarthikElangovan]

IndexError Traceback (most recent call last)
in ()
32 time.sleep(0.1)
33
---> 34 model.fit(x=x, y=y, batch_size=32, epochs = 3, validation_split=0.1)

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
726 max_queue_size=max_queue_size,
727 workers=workers,
--> 728 use_multiprocessing=use_multiprocessing)
729
730 def evaluate(self,

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
222 validation_data=validation_data,
223 validation_steps=validation_steps,
--> 224 distribution_strategy=strategy)
225
226 total_samples = _get_total_number_of_samples(training_data_adapter)

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_training_inputs(model, x, y, batch_size, epochs, sample_weights, class_weights, steps_per_epoch, validation_split, validation_data, validation_steps, shuffle, distribution_strategy, max_queue_size, workers, use_multiprocessing)
495 'at same time.')
496
--> 497 adapter_cls = data_adapter.select_data_adapter(x, y)
498
499 # Handle validation_split, we want to split the data and get the training

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in select_data_adapter(x, y)
645 def select_data_adapter(x, y):
646 """Selects a data adapter than can handle a given x and y."""
--> 647 adapter_cls = [cls for cls in ALL_ADAPTER_CLS if cls.can_handle(x, y)]
648 if not adapter_cls:
649 # TODO(scottzhu): This should be a less implementation-specific error.

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in (.0)
645 def select_data_adapter(x, y):
646 """Selects a data adapter than can handle a given x and y."""
--> 647 adapter_cls = [cls for cls in ALL_ADAPTER_CLS if cls.can_handle(x, y)]
648 if not adapter_cls:
649 # TODO(scottzhu): This should be a less implementation-specific error.

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in can_handle(x, y)
454 handles_y = True
455 if y is not None:
--> 456 handles_y = ListsOfScalarsDataAdapter._is_list_of_scalars(y)
457 return handles_x and handles_y
458

~\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in _is_list_of_scalars(inp)
462 return True
463 if isinstance(inp, (list, tuple)):
--> 464 return ListsOfScalarsDataAdapter._is_list_of_scalars(inp[0])
465 return False
466

IndexError: list index out of range

i am also getting the same ..some one please help me...

[zakiyyah-ai]

how can solve this value error?

[sulaeman22]

editing in google colab and getting error in the model.fit part of the code. showing value error: in user code. anyone knows the solution

[sulaeman22]

[sulaeman22]

[sulaeman22]

[Coder-Vishali]

how can solve this value error?

Did you find solution