HParams KerasCallback Display Max Value for Metric

[dgrahn]

In the HParams plugin, it would be very useful to have the option of viewing the maximum reached by a metric during training. With the current KerasCallback, only the last value is stored. If you use the KerasCallback in addition to EarlyStopping, you'll end up with a lower than max value.

I'd be open to contributing to this, but I would need guidance on where the best place to add the feature would be and how to test it.

[stephanwlee]

Thanks for the feature request and willingness to contribute. Keras callback, specifically, is not owned by us but rather by the TensorFlow team. We can, too, suggest an improvement but if you were to contribute, you would have to send a PR to https://github.com/tensorflow/tensorflow and make changes to https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/callbacks.py.

BTW, @wchargin, do we only record single metric value?

[dgrahn]

@stephanwlee I believe you are incorrect. The code for the KerasCallback is a part of the Hparams Tensorboard plugin: https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/hparams/keras.py

[dgrahn]

As I'm looking through the code, I think this is how it works.

• Tensorboard callback logs metrics.
• KerasCallback logs the hyperparameters.
• HParams backend pulls the metrics in from the Tensorboard logs.

Based on this, the best way to handle this would be to add an option inside of the "Metrics" panel to allow users to select which metric is returned (last, max, or min). I believe I could make the changes, but would like them to be "generally approved" before taking the time.

It appears that the relevant code is here:

• Backend - Reading the last scalar:

  tensorboard/tensorboard/plugins/hparams/backend_context.py

  Line 138 in ba302fc

  def read_last_scalars(self, experiment_id, run_tag_filter):

• Backend - Builds the session groups:

  tensorboard/tensorboard/plugins/hparams/list_session_groups.py

  Line 74 in 65a8103

  def _build_session_groups(self):

• Frontend - Build session group query:
  

  tensorboard/tensorboard/plugins/hparams/tf_hparams_query_pane/tf-hparams-query-pane.html

  Line 827 in c6e181c

  _buildListSessionGroupsRequest() {

• Frontend - metrics panel:

  tensorboard/tensorboard/plugins/hparams/tf_hparams_query_pane/tf-hparams-query-pane.html

  Line 104 in c6e181c

  <div class="section-title">Metrics</div>

[stephanwlee]

Ah, indeed. I am very mistaken. Those place look correct, but I would not recommend making changes to the hparams UI code as it is not something we'd like to move forward with. Though it is very early to share, we are working on rewrite in Angular.

@GalOshri did you hear similar feature requests from other users?

[dgrahn]

So this isn't a great solution since it clutters up the metrics. But I created a callback which logs the maximum value of metrics. It's a stopgap measure for those who need something short-term.

https://gist.github.com/dgrahn/45acc8ee54c515d6ac28e2696db2a12d