TimeStopping callback

[kylemcdonald]

I've been using this callback for stopping training after a fixed period of time, useful when testing different architectures consecutively. Is it worth contributing?

One thing I noticed was self.model.stop_training only stops the model after an epoch, but with some training where the epochs are very long you might want to stop after a batch.

import time
from keras.callbacks import Callback
class TimedStopping(Callback):
    '''Stop training when enough time has passed.
    # Arguments
        seconds: maximum time before stopping.
        verbose: verbosity mode.
    '''
    def __init__(self, seconds=None, verbose=0):
        super(Callback, self).__init__()

        self.start_time = 0
        self.seconds = seconds
        self.verbose = verbose

    def on_train_begin(self, logs={}):
        self.start_time = time.time()

    def on_epoch_end(self, epoch, logs={}):
        if time.time() - self.start_time > self.seconds:
            self.model.stop_training = True
            if self.verbose:
                print('Stopping after %s seconds.' % self.seconds)

[faroit]

@kylemcdonald 👍 yes, this is super helpful. Especially if you run your jobs on a HPC cluster where a 24h job limit applies ;-)

I would even add safety_factor so that the training stops earlier by a factor of the average duration per epoch:

class TimedStopping(Callback):
    '''Stop training when enough time has passed.
    # Arguments
        seconds: maximum time before stopping.
        safety_factor: stop safety_factor * average_time_per_epoch earlier
        verbose: verbosity mode.
    '''
    def __init__(self, seconds=None, safety_factor=1, verbose=0):
        super(Callback, self).__init__()

        self.start_time = 0
        self.safety_factor = safety_factor
        self.seconds = seconds
        self.verbose = verbose
        self.time_logs = []

    def on_train_begin(self, logs={}):
        self.start_time = time.time()

    def on_epoch_end(self, epoch, logs={}):
        elapsed_time = time.time() - self.start_time
        self.time_logs.append(elapsed_time)

        avg_elapsed_time = float(sum(self.time_logs)) / \
            max(len(self.time_logs), 1)

        print(" ", self.seconds - self.safety_factor * avg_elapsed_time)
        if elapsed_time > self.seconds - self.safety_factor * avg_elapsed_time:
            self.model.stop_training = True
            if self.verbose:
                print('Stopping after %s seconds.' % self.seconds)

are submitting a PR?

[fchollet]

Please consider adding it to the contrib repo: https://github.com/farizrahman4u/keras-contrib

[faroit]

@fchollet didn't know about contrib. Maybe it contrib would get more attention if you would move both keras and contrib into a new keras github organisation.

[kylemcdonald]

Thanks @fchollet closing this and moving to keras-team/keras-contrib#87