Tensorboard dev

[elleobrien]

One of the use cases we want to show for the CML site is producing a tensorboard link in the report. For example:

View your TensorBoard live at: https://tensorboard.dev/experiment/E31QaKfWTQaQuEUR6H03UA/

I'm running into some trouble using the tensorboard dev command inside my GH runner. This is my workflow (project repo is here):

name: train-my-model

on: [push]

jobs:
  run:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/cml-py3:latest

    steps:
      - uses: actions/checkout@v2

      - name: dvc_cml_run
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }}
        run: |
          pip3 install -r requirements.txt
          python train.py
          
          tensorboard dev upload --logdir logs

I've confirmed the commands work on my local machine. On the runner, I'm getting this error message:

Traceback (most recent call last):
  File "/usr/local/bin/tensorboard", line 8, in <module>
    sys.exit(run_main())
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/main.py", line 75, in run_main
    app.run(tensorboard.main, flags_parser=tensorboard.configure)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/program.py", line 289, in main
    return runner(self.flags) or 0
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/uploader/uploader_main.py", line 633, in run
    return _run(flags)
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/uploader/uploader_main.py", line 120, in _run
    _prompt_for_user_ack(intent)
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/uploader/uploader_main.py", line 76, in _prompt_for_user_ack
    response = six.moves.input("Continue? (yes/NO) ")
EOFError: EOF when reading a line
Continue? (yes/NO) 
##[error]Process completed with exit code 1.

I tried passing in a "yes" with

echo "yes" | tensorboard dev upload --logdir logs but got the same error. Does this make any sense given the setup of the runner?

[DavidGOrtega]

@andronovhopf this is a very interesting case. Probably a candidate to have its own cml wrapper @dmpetrov

First of all Tensorboard dev is expecting you to auth using OAuth, this process will generate a file inside your machine under ~/.config/tensorboard/credentials/uploader-creds.json ala google drive or gs credentials.

So, you must copy those JSON credentials and add them as a secret. Then within your approach you still need a Ctrl+c to cancel sending data to TB.

However this might not be the best approach. One of the best features is watching your model logs in real time so TB should be started first (as a fg process) and then train. However this procedure had a lots of caveats failing in many ways due to failures in the pipes. Also is very important to know whats the url to be checked during the training. Finally after some iterations this is actually the solution:

name: train-my-model

on: [push]

jobs:
  run:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/cml-py3:latest

    steps:
      - uses: actions/checkout@v2

      - name: dvc_cml_run
        shell: bash
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }} 
        run: |
          pip3 install -r requirements.txt

          TENSORBOARD_CREDENTIALS='{"refresh_token": "1//03IJCTMThPsYECgYIARAAGAMSNwF-L9Iru6PoxuqEtGTcnvbXeGwi5j4cXBrFQpXdcmBhZyvZggR1WqKeIjhs1V57g1NvpCsUFnw", "token_uri": "https://oauth2.googleapis.com/token", "client_id": "373649185512-8v619h5kft38l4456nm2dj4ubeqsrvh6.apps.googleusercontent.com", "client_secret": "pOyAuU2yq2arsM98Bw5hwYtr", "scopes": ["openid", "https://www.googleapis.com/auth/userinfo.email"], "type": "authorized_user"}'
          mkdir -p ~/.config/tensorboard/credentials/
          echo "$TENSORBOARD_CREDENTIALS" > ~/.config/tensorboard/credentials/uploader-creds.json

          script -f -c "
              tensorboard dev upload --logdir logs & \
              sleep 5 && \
              cat tb.log | grep http > report.md && \
              cml-send-comment report.md && \
              python train.py" tb.log

          cml-send-github-check report.md

This solution is very good in a PR since it's going to generate a comment telling you where to go aside of printing the url in the command line before start training.

Then is generating the final report also.

with secrets just replace

TENSORBOARD_CREDENTIALS='{"refresh_token": "1//03IJCTMThPsYECgYIARAAGAMSNwF-L9Iru6PoxuqEtGTcnvbXeGwi5j4cXBrFQpXdcmBhZyvZggR1WqKeIjhs1V57g1NvpCsUFnw", "token_uri": "https://oauth2.googleapis.com/token", "client_id": "373649185512-8v619h5kft38l4456nm2dj4ubeqsrvh6.apps.googleusercontent.com", "client_secret": "pOyAuU2yq2arsM98Bw5hwYtr", "scopes": ["openid", "https://www.googleapis.com/auth/userinfo.email"], "type": "authorized_user"}'
mkdir -p ~/.config/tensorboard/credentials/
echo "$TENSORBOARD_CREDENTIALS" > ~/.config/tensorboard/credentials/uploader-creds.json

with

mkdir -p ~/.config/tensorboard/credentials/
echo "${{ secrets.TENSORBOARD_CREDENTIALS }}" > ~/.config/tensorboard/credentials/uploader-creds.json

[elleobrien]

Great work @DavidGOrtega! I had a feeling the authentication was going to take a little work. Looks like a similar approach to authenticating with GDrive as a DVC remote?

I'll give your solution a go. Really cool

[DavidGOrtega]

Looks like a similar approach to authenticating with GDrive as a DVC remote?

Yes, exactly

[DavidGOrtega]

a cml wrapper would just give the url without having to oauth since we could be using a generic CML user. In TB dev nothing indicates the owner

[DavidGOrtega]

@andronovhopf We have just released a new feature called cml-tensorboard-dev. It setup the given credentials for you and launches an instance of tensorboard returning you a url like cml-publish does.

The workflow eequivalent to what I posted is

name: train-my-model

on: [push]

jobs:
  run:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/cml-py3:latest

    steps:
      - uses: actions/checkout@v2

      - name: dvc_cml_run
        shell: bash
        env:
          repo_token: ${{ secrets.GITHUB_TOKEN }} 
        run: |
          pip3 install -r requirements.txt
          export TB_CREDENTIALS='{"refresh_token": "1//03IJCTMThPsYECgYIARAAGAMSNwF-L9Iru6PoxuqEtGTcnvbXeGwi5j4cXBrFQpXdcmBhZyvZggR1WqKeIjhs1V57g1NvpCsUFnw", "token_uri": "https://oauth2.googleapis.com/token", "client_id": "373649185512-8v619h5kft38l4456nm2dj4ubeqsrvh6.apps.googleusercontent.com", "client_secret": "pOyAuU2yq2arsM98Bw5hwYtr", "scopes": ["openid", "https://www.googleapis.com/auth/userinfo.email"], "type": "authorized_user"}'
          
          echo 'CML' > report.md
          cml-tensorboard-dev --logdir logs --md --name "Go to tensorboard" >> report.md
          cml-send-comment report.md
          python train.py
          cml-send-github-check report.md

👀 credentials should be a secret!

You can see the comment generated here

https://github.com/DavidGOrtega/3_tensorboard/commit/b60b78e1057131f31bca456f35c671137dc3e38a#comments

[elleobrien]

it worked great for me!!! thanks. I love that this workflow starts tensorboard before python train.py so I can watch training happen. And cml-tensorboard-dev has regex built in to just extract the tensorboard link and not all the other output! So good.

for the purpose of explaining in docs how to configure secrets.TB_CREDENTIALS...which Google API should users be setting up to get these? For Google Drive access with CML, we'd had them using the Google Drive API; is there a better one?

[DavidGOrtega]

it's an google engine app. In their source code I can see their config:

OAUTH_CLIENT_CONFIG = u"""
{
  "installed": {
    "client_id": "373649185512-8v619h5kft38l4456nm2dj4ubeqsrvh6.apps.googleusercontent.com",
    "project_id": "hosted-tensorboard-prod",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_secret": "pOyAuU2yq2arsM98Bw5hwYtr",
    "redirect_uris": [
      "urn:ietf:wg:oauth:2.0:oob",
      "http://localhost"
    ]
  }
}
"""

So google probably offers a way to get the auth doing a 3-legged auth. Let me see, Im not a google engine pro.

[dmpetrov]

yeah, it is not clear how to set up TB credentials.
I copy-pasted the content of my ~/.config/gcloud/application_default_credentials.json to GH TB_CREDENTIALS and set up env to TB_CREDENTIALS: ${{ secrets.TB_CREDENTIALS }} but there is a still an auth issue (below).

@DavidGOrtega could you please provide some instruction on how to set it up?

python -u /usr/local/bin/tensorboard dev upload --logdir logs --name "Go to TB" --description ""
Data for the "graphs" plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the "--plugins" command line argument.

Traceback (most recent call last):
  File "/usr/local/bin/tensorboard", line 8, in <module>

    sys.exit(run_main())
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/main.py", line 75, in run_main

    app.run(tensorboard.main, flags_parser=tensorboard.configure)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run

    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main


....


    self._api.CreateExperiment, request
  File "/usr/local/lib/python3.6/dist-packages/tensorboard/util/grpc_util.py", line 90, in call_with_retries

    metadata=version_metadata(),
  File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in __call__

    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking

    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAUTHENTICATED
	details = "Failed to authenticate user: Invalid audience scope"
	debug_error_string = "***"created":"@1590176686.852689640","description":"Error received from peer ipv4:34.95.66.171:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Failed to authenticate user: Invalid audience scope","grpc_status":16***"

[DavidGOrtega]

@dmpetrov
they must be TB specific credentials. This mean that you have to generate proper credentials with TB. What I was trying to clarify is how we could use google 3-legged OAuth api based on TB configuration.

Until that, the procedure is creating your own credentials:

1. launch tensorboard dev upload in your machine.
2. Accept the TOS
3. Visit a URL (OAuth) that will give you and write the code that TB is going to ask for.
4. Pick your fresh generated credentials at ~/.config/tensorboard/credentials/uploader-creds.json

[DavidGOrtega]

closed via #102