Add support for on premise Tensorboard

[DavidGOrtega]

Right now only considers dev while it could use users instances.
Some users do not use cml-tensorboard-dev because of privacy reasons

[0x2b3bfa0]

Right now only considers dev while it could use users instances.

% tensorboard --help
usage: tensorboard ··· {serve,dev} ···

···

positional arguments:
  {serve,dev}           TensorBoard subcommand (defaults to 'serve')
    serve               start local TensorBoard server (default subcommand)
    dev                 upload data to TensorBoard.dev

optional arguments:
  ···

As per the help message on the tensorboard command, the only options are dev and serve, where the former would use the managed tensorboard.dev servers and the latter would run a local server on the machine where the command has been executed.

Some users do not use cml-tensorboard-dev because of privacy reasons.

Sounds logical, but it would be a bad remedy if we let users resort to the self-hosted version and expose it to the wide internet without any security measure; see tensorflow/tensorboard#267.

We can generate custom addresses for each run as proposed on tensorflow/tensorboard#267 (comment) and use cml-send-comment to publish the links as usual. Users wouldn't notice any difference, and we don't care about local risks: whoever can access the instance can also access the data we're protecting.

[0x2b3bfa0]

Once instances stop, users won't be able to access the service anymore and the link will be down. Right? 🤔

[DavidGOrtega]

Im not speaking about ephemeral tensorboard but allow users to have their own TB up and running.

Sounds logical, but it would be a bad remedy if we let users resort to the self-hosted version and expose it to the wide internet without any security measure; see tensorflow/tensorboard#267.

tensorboard.dev works that way. It's totally insecure. The user would have t think how to securize that.

Aside that even with ephemeral TB this would be awesome because the user can see the output during the training even that later on tb is not available anymore is still useful.

[0x2b3bfa0]

I'm not speaking about ephemeral tensorboard but allow users to have their own TB up and running.

Do you mean spinning up a persistent cloud instance with our Terraform provider to deploy Tensorboard, and leave it running indefinitely like if it were tensorboard.dev? As far as I can tell, the tensorboard command-line tool doesn't allow users to specify a managed service other than the official one, and the other altenative would require the server to run in the same machine that has the data; id est, our ephemeral instance.

tensorboard.dev works that way. It's totally insecure.

Agh! Well, they generate a new URL for each result with a ~128 bit unique suffix, but anybody with the URL can access the metrics.

Fortunately enough, they do have some reasonable security guidelines for the self-hosted counterpart:

• https://github.com/tensorflow/tensorboard/blob/master/SECURITY.md
• https://github.com/tensorflow/tensorflow/blob/master/SECURITY.md#running-a-tensorflow-server

The user would have to think how to securize that.

Preferably, we should be in charge of that. When you buy a car, it doesn't come with a hot shoe behind the wheel and a note suggesting to connect an airbag if you deem it necessary, or choose the optimism and prayer add-on instead.

Aside that even with ephemeral TB this would be awesome because the user can see the output during the training even that later on tb is not available anymore is still useful.

Agreed!

[DavidGOrtega]

As far as I can tell, the tensorboard command-line tool doesn't allow users to specify a managed service other than the official one, and the other altenative would require the server to run in the same machine that has the data; id est, our ephemeral instance.

I think it allow to set the host

[0x2b3bfa0]

I think it allow to set the host

I still haven't found how to do that after a cursory investigation. 🤔 This could work, but seems to be an experimental setting, and deploying a tensorboard.dev alternative on our own might not be a brilliant idea.

https://github.com/tensorflow/tensorboard/blob/29cfcec9d15c47d1bf54b15df24e65bc38c188e6/tensorboard/uploader/flags_parser.py#L47-L53