[Unity] cuda graph support for cublas

[sunggg]

This PR enables cuda graph with cublas.

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• cc @quic-sanirudh _{See #10317 for details}

_{Generated by tvm-bot}

[LeiWang1999]

hi @masahi , @sunggg , would you mind providing some detailed information about the CUDA graph? I'm currently attempting to integrate cuDNN into Relax via BYOC, and any insights would be greatly appreciated.

[masahi]

What information do you need?

[LeiWang1999]

What information do you need?

@masahi thanks,
Trustfully, I'm not sure about the purpose of the cuda graph, is this a new concept that introduced by unity?

[masahi]

No, it is a standard CUDA concept to reduce host-side overhead. See
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs
https://www.nvidia.com/ja-jp/on-demand/session/gtcspring23-s51211/

[LeiWang1999]

@masahi thanks! seems sth like multi stream parallel.