Define summary frequency per step

[visuallization]

Hi there,

Is there a possibility to define the summary frequency per step? Kind of similar to unity ml agents.

Cheers!

[alex-petrenko]

Hi @visuallization !

Currently the summary rate per step is defined in learner.py

    def _should_save_summaries(self):
        summaries_every_seconds = self.summary_rate_decay_seconds.at(self.train_step)
        if time.time() - self.last_summary_time < summaries_every_seconds:
            return False

        return True

Where summary_rate_decay is:

self.summary_rate_decay_seconds = LinearDecay([(0, 2), (100000, 60), (1000000, 120)])

This will start saving summaries every 2 seconds at the beginning of training, linearly decay to once in 60 seconds at 100000 train steps, and to once in 120 seconds after 1M steps, and then it will stay there.

The logic behind this is following:

• at the beginning of training and for shorter runs we want high granularity of summaries to have nice informative smooth curves.
• for longer runs (hours or days) we don't want frequent summaries, at the very least to prevent tensorboard event files from growing to gigabytes.

You are welcome to modify this code to implement the desired logic. I don't think there's a command line option to control that. Contributions are welcome!

[visuallization]

@alex-petrenko Okay great thanks for the detailed explanation!