Show averages in batch loss/accuracy?

[danieldk]

While training, the batch loss/accuracy of the last trained batch is shown. However, these numbers tend to jump around quite a bit. Maybe we should show the average so far in the current epoch? This would help observing the trends a bit better.

[twuebi]

I think it's very informative to see single batch losses, it can be helpful for tuning parameters, but also to find foul training examples, as they would tend to produce outlier losses.

[danieldk]

It could be a command-line option. Overall averages are not really nice for pretraining (who cares what performance you had at batch 100 when you are at batch 10000). On the other hand, I see quite a lot of variability between batches, which is also annoying. Maybe the proper solution would be to support moving averages, they are informative on how the model performs at that point, but also easier to read than the jumpy raw loss/accuracy.

[twuebi]

It could be a command-line option. Overall averages are not really nice for pretraining (who cares what performance you had at batch 100 when you are at batch 10000). On the other hand, I see quite a lot of variability between batches, which is also annoying. Maybe the proper solution would be to support moving averages, they are informative on how the model performs at that point, but also easier to read than the jumpy raw loss/accuracy.

If it's just an option, no hard feelings.

Something related I implemented: I have a branch with tensorboard support (through tf.contrib.summary) which serves plots of accuracy, train loss, gradient norms and also visualizes the graph. It brings the overhead of a browser in, but I like the visualization it offers.

[danieldk]

Tensorboard support would be really nice!

[twuebi]

Then I'll do the PRs.