Skip to content

Commit

Permalink
Fixes keras-team#2110 "ValueError: I/O operation on closed file"
Browse files Browse the repository at this point in the history
This is a workaround for keras-team#2110 where calling `model.fit` with verbose=1
using IPython can intermittently raise "ValueError: I/O operation on
closed file".

This exception appears to be caused by an unknown IO bug with IPython
that is raised when updating the ProgbarLogger progress bar . To
workaround this bug and prevent users from unexpectedly losing their
model:

- The minimum progress bar refresh interval is now 0.1 seconds (up from
0.01 seconds).

- Progress bar updates are now wrapped in try/catch blocks that ignore
ValueError exceptions raised when calling progbar.update

An ideal solution would resolve the IPython at the source, however, this
is an important workaround for users who want to use IPython with
verbose=1.
  • Loading branch information
scottlawsonbc committed Jun 28, 2016
1 parent ba90107 commit 5f9f127
Showing 1 changed file with 10 additions and 3 deletions.
13 changes: 10 additions & 3 deletions keras/callbacks.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,8 @@ def on_epoch_begin(self, epoch, logs={}):
if self.verbose:
print('Epoch %d/%d' % (epoch + 1, self.nb_epoch))
self.progbar = Progbar(target=self.params['nb_sample'],
verbose=self.verbose)
verbose=self.verbose,
interval=0.1)
self.seen = 0

def on_batch_begin(self, batch, logs={}):
Expand All @@ -185,14 +186,20 @@ def on_batch_end(self, batch, logs={}):
# skip progbar update for the last batch;
# will be handled by on_epoch_end
if self.verbose and self.seen < self.params['nb_sample']:
self.progbar.update(self.seen, self.log_values)
try:
self.progbar.update(self.seen, self.log_values)
except ValueError:
pass

def on_epoch_end(self, epoch, logs={}):
for k in self.params['metrics']:
if k in logs:
self.log_values.append((k, logs[k]))
if self.verbose:
self.progbar.update(self.seen, self.log_values, force=True)
try:
self.progbar.update(self.seen, self.log_values, force=True)
except ValueError:
pass


class History(Callback):
Expand Down

0 comments on commit 5f9f127

Please sign in to comment.