Skip to content

ONNX Runtime CUDA Plugin EP 0.1.0

Choose a tag to compare

View all tags
@tianleiwu tianleiwu released this 17 Aug 23:12
plugin-ep-cuda/v0.1.0
08a7a47

This is the first release of ONNX Runtime CUDA Plugin EP, providing CUDA execution as a separately packaged plugin execution provider.

These notes cover commits affecting CUDA Plugin EP core code, CMake integration, and its primary build and package pipeline.

Please refer to QUICK_START.md for the usage.

Highlights

Plugin Runtime

  • Introduces the CUDA Plugin EP core and makes it the default CUDA provider implementation (#27816, #29544).
  • Adds arena allocation, resource accounting, available-resource reporting, and IOBinding synchronization (#27931, #28028, #28103, #27919).
  • Adds provider options for user compute streams, copy behavior, EP-level unified streams, and external allocators, with compatibility back to ONNX Runtime 1.24.4 through version-gated callbacks (#28603, #28824).

CUDA Graphs and Profiling

  • Adds CUDA Graph capture and replay, including user compute stream support, synchronization controls, and a kernel sync stream for scratch allocation (#27958, #28002, #28686, #29221, #29244).
  • Adds plugin profiling APIs and memory statistics in profiling output (#28216, #29058).

Operators and Dependencies

  • Expands model coverage with NHWC hardening, cuDNN SDPA for MHA/GQA, quantized MoE kernels, block-quantized FP4 MatMul, and native NVFP4 QMoE prefill on SM120 (#28612, #28849, #28467, #29818, #29824).
  • Adds cuDNN-free ArgMax, ArgMin, and ReduceSum and fixes LogSoftmax for plugin deployments (#29620).
  • Reduces runtime dependencies by making cuDNN and cuFFT optional and removing the nvrtc link requirement (#29252, #29808, #29705).

Build and Packaging

  • Adds dedicated build and packaging pipelines, the onnxruntime-ep-cuda Python wheel, and NuGet packages (#27959, #28152, #28299, #28378).
  • Adds Windows ARM64 packages, package-size controls, and aligned package metadata and artifact naming (#28896, #31635).
  • Improves CUDA build coverage and reliability across CUDA 12.8/13, SM90, and SM120 while reducing CI build time with per-architecture object libraries and quick-build support (#29711, #28736, #29776, #31608, #31609, #28645).

Contributors

Thank all contributors that contributed to the CUDA execution provider:

@abudup, @aciddelgado, @AdamLouly, @adamreeve, @adrianlizarraga, @Alexey-Kamenev, @amancini-N, @amathews-amd, @AndreyOrb, @apsonawane, @ashbhandare, @askhade, @baijumeswani, @BowenBao, @chenfucn, @chengchen666, @chilo-ms, @cloudhan, @cocotdf, @codemzs, @cschreib-ibex, @dakenf, @decahedron1, @derdeljan-msft, @duli2012, @e-tao, @edgchen1, @elwhyjay, @eserscor, @fajin-corp, @feihugis, @frank-dong-ms, @fs-eire, @fthielke, @gedoensmax, @georgthegreat, @gh-yewang, @gineshidalgo99, @GopalakrishnanN, @gramalingam, @guoyu-wang, @guoyuhong, @hadiFute, @hanbitmyths, @hariharans29, @harrysummer, @harshithapv, @HectorSVC, @hubertlu-tw, @jagadish-amd, @jambayk, @jchen351, @jeffdaily, @jessebenson, @jiafatom, @jiangzhuo, @jignparm, @jingyanwangms, @jixiongdeng, @johannes-rehm-snkeos, @jslhcl, @JTischbein, @justinchuby, @jywu-mysoft, @kailums, @ke1337, @khspear, @kit1980, @KnightYao, @kpkbandi, @krahenbuhl, @kunal-vaishnavi, @Lafi7e, @linkerzhang, @liqunfu, @liuziyue, @lukas-folle-snkeos, @mastryukov1990, @mc-nv, @micmelesse, @mika-fischer, @mindest, @mityada, @mszhanyi, @mtavenrath, @mwootton, @namgyu-youn, @naomiOvad, @natke, @neginraoof, @nenad1002, @oliviajain, @PatriceVignola, @PeixuanZuo, @pengwa, @petermcaughan, @poweiw, @pranavsharma, @praneshgo, @prathikr, @qti-jkilpatrick, @qti-kromero, @RandySheriffH, @RandyShuai, @raviskolli, @Rishi-Dave, @RyanUnderhill, @satyajandhyala, @selenayang888, @seungtaek94, @shaahji, @shahasad, @sheetalarkadam, @SherlockNoMad, @ShirasawaSama, @skottmckay, @skyline75489, @smk2007, @snnn, @sohaibiftikhar, @souptc, @stevenlix, @Stonesjtu, @suffiank, @thevishalagarwal, @thiagocrepaldi, @tiagoshibata, @tianleiwu, @titaiwangms, @Tixxx, @tpboudreau, @tracysh, @umangb-09, @viboga, @vraspar, @weixingzhang, @wejoncy, @wschin, @xadupre, @xiaofeihan1, @xiaomsft, @xinyazhang, @XiyinOSS, @xzhu1900, @yangchen-MS, @ybrnathan, @yf711, @yihonglyu, @yinli-systems, @ytaous, @yuanbyu, @yufenglee, @YUNQIUGUO, @yuslepukhin, @zhanghuanrong, @zhijxu-MS, @Zyrin

Assets 8
Loading