Monitoring

[bartvm]

This is just a bunch of ideas that I think would be nice for monitoring. I want to start working on some optimization stuff, for which it would be handy if I can monitor a range of different things, many of which Pylearn2 doesn't support right now.

• Monitor at different points in time (all optional)
  1. After each update (useful for monitoring on the training set e.g. the likelihood of each batch; done by GroundHog for machine translation I believe)
  2. After a fixed number of updates (useful for infinite datasets, or very large ones)
  3. After each epoch (after having seen as many samples as are in the dataset, only for finite datasets; this is what Pylearn2 has)
• Don't just monitor scalars, also allow for arrays to be monitored (e.g. gradients and Hessians) and general Python objects (e.g. samples from the translation model as strings)
• Closely related: Monitor non-Theano variables (e.g. I want to be able to use NumPy to calculate the eigenvalues, or calculate the BLEU score, and monitor that), but Theano variables still need special treatment
  • Allow monitor to save history to disk once in a while, because memory usage to store this data can quickly add up
• Allow for monitors to be pickled and continue where they left off

We need to keep in mind that Theano variable monitor channels need to be collected and compiled as a group (so that they can share a large part of the computational graph). Likewise, if we monitor the progress of each batch (useful for very long-running experiments like machine translation) it would be best if we can compile the monitoring value together with the update, so that they too can share the computational graph.

[bartvm]

Another limitation of Pylearn2 came to mind: Averaging of statistics over a validation set. Basically, there are two kinds of statistics, those that need to be accumulated over an entire validation set (e.g. average likelihood) and those that don't (e.g. weight norms). Monitoring values should be tagged as such, so that they can be collected into two separate Theano functions.

[bartvm]

For reference, also see discussion here: #68 (comment)

[bartvm]

Main features completed; time for more specific issues.