[tritonbench] Add initial tritonbench benchmark config

[xuzhao9]

We would like to utilize the Blackwell runners for TritonBench benchmarking.

It will pull the latest docker image from https://github.com/meta-pytorch/tritonbench/pkgs/container/tritonbench and run the nightly benchmark. We might add more TritonBench benchmarks later.

TODOs:

• Tune Blackwell frequency and power limit so that it matches the internal benchmarking setup. ([gpu_lockdown] Add GPU lockdown script for B200 meta-pytorch/tritonbench#668)
• Add GitHub secret TRITONBENCH_SCRIBE_GRAPHQL_ACCESS_TOKEN to the repo, so that the metrics can be written to Scuba/Scribe

[huydhn]

Any permission that we need to setup to make Scribe upload works here?

[huydhn]

Ah, let me know if you need to setup TRITONBENCH_SCRIBE_GRAPHQL_ACCESS_TOKEN, not sure if that can be done self-serve

[xuzhao9]

Yes, we will need to set up TRITONBENCH_SCRIBE_GRAPHQL_ACCESS_TOKEN as a github secret.

[huydhn]

Just FYI, B200 runners are offline at the moment waiting for NVIDIA to re-image them, so it would take a few more days for them to be back to CI. The tracking issue is here pytorch/pytorch#169386

[huydhn]

One note is that I don't think the CI container has the permission to run meta-pytorch/tritonbench#668. In B200 multi-tenancy setup, the workflow has already run inside a container. Nevertheless, the daemon does fix the power and clock here https://github.com/meta-pytorch/pytorch-gha-infra/blob/main/multi-tenant/services/ghad-manager/ghad-manager.py#L578-L580 before it launches the container. But this is done outside of CI

[xuzhao9]

Thanks for the feedback! Glad to know that the GPU is tuned already in the infra.