[ENHANCEMENT] CUDA-Graph integration

[xysmlx]

🚀 Feature

CUDA-Graph is introduced in CUDA-10.1 to reduce kernel launch overhead. CUDA-Graph matches current NNFusion's design, so it could be easily integrated to cuda_codegen to improve performance.

Motivation

Pitch

Add stream in kernel_entry and capture the kernel_entry function to initialize the cuda-graph.

Note that it cannot capture default stream and there should not be host-blocking API calls (e.g., cudaDeviceSynchronize) during stream capturing.

Alternatives

Additional context

https://developer.nvidia.com/blog/cuda-graphs/
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#cuda-graphs

[nnfbot]

Thanks for the report @xysmlx! I will look into it ASAP! (I'm a bot).