Support hparams logging to tensorboard

[timothe-chaumont]

Description

• I created a HParam data class in the same way as Figure, Image ones. It can take any number of distinct hyperparameters and metrics as input.

• In each .write() method of HumanOutputFormat, JSONOutputFormat and CSVOutputFormat, I have raised errors when hparams format was given. e.g.

            elif isinstance(value, HParam):
                raise FormatUnsupportedError(["stdout", "log"], "hparam")

• I added a case in TensorBoardOutputFormat to log the hparams values to tensorboard:

            if isinstance(value, HParam):
                # we don't use `self.writer.add_hparams` to have control over the log_dir
                exp, ssi, sei = hparams(value.hparam_dict, metric_dict=value.metric_dict)
                self.writer.file_writer.add_summary(exp)
                self.writer.file_writer.add_summary(ssi)
                self.writer.file_writer.add_summary(sei)

As described in the code comment, I have not used self.writer.add_hparams(hparam_dict, metric_dict), provided by pytorch, but reused some of the code from that method. Using this first method, the hparams could not be saved in the same run folder as the other logs, and so metrics from SCALARS tab could not appear in HPARAMS tab.

• I created one parametrized test : test_report_hparam_to_unsupported_format_raises_error() in the same way as other data classes. I did not create a test_report_hparam_to_tensorboard test, as hparams logs are not seen by EventAccumulator (c.f. Stack Overflow and EventAccumulator implementation).

• Finally, in the tensorboard section of the documentation, I added an example of a callback that uses this new code to log hyperparameters to tensorboard.

Additionnal information - choices made

• As hyperparameters are key-value pairs, adding the support for csv, json, or human output formats could be a future developpment.

• It is not required to pass a metric_dict to hparams() or writer.add_hparams(), but if we don't, then nothing is displayed in HPARAMS tab. I have added a warning to alert the user about that.

• When adding metrics in metric_dict, the users have 2 choices:

  • Adding custom metrics -> only one value per run for each metric and no plot,
  • Using metrics from the scalar section (with value 0) -> the user can visualize the plot of that metric in HPARAMS tab and the last value of that plot is displayed (c.f. image below).

  It is not ideal to display the last value - instead of the best value, for example (issue discussed here) - but I decided to use the second one in the documentation example as I found it more relevant and intuitive.

• In the example, I put the logic in _on_training_start() as we only have to log the hyperparameters & metrics once, but I could also added it in on_step() as I did it here.

Motivation and Context

• [x] I have raised an issue to propose this change (required for new features and bug fixes) I have proposed a solution to Also log hyperparameters to the tensorboard #428 : a way of logging hyperparameters to tensorboard.

closes #428

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)
• I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[araffin]

please use explicit names, what is ssi? sei?

[timothe-chaumont]

It's done:

experiment, session_start_info, session_end_info = hparams(value.hparam_dict, metric_dict=value.metric_dict)

The content and meaning of those variables is described in hparam docstring. (And the initial short names where those used in pytorch SummaryWriter class)

[araffin]

is the new feature also tested somewhere?
currently only the failure case is tested?

[timothe-chaumont]

I tested the new feature in practice, but not programmatically. It is not as easy as for the other types of logs (e.g. images):

I did not create a test_report_hparam_to_tensorboard test, as hparams logs are not seen by EventAccumulator (c.f. Stack Overflow and EventAccumulator implementation).

Two solutions are proposed in this stackoverflow page but one requires an external library, and the other is not very clean.

[araffin]

more than reading the logged hparams, we should at least run the logger.

[araffin]

LGTM, thanks =)