onnxruntime-gpu for NVIDIA Jetson Orin — JetPack 7 / sm_87

Target board: Jetson Orin Nano Super (8 GB). Developed and verified on it; works on any Jetson Orin with sm_87 (Nano / Nano Super / NX / AGX).

A reproducible kit to build onnxruntime-gpu (CUDA + TensorRT execution providers) from source for JetPack 7.2 (L4T r39.2, CUDA 13.2, TensorRT 10.16, Ubuntu 24.04, sm_87 — Orin family). No prebuilt wheel exists for this stack; this repo builds one and proves it runs on the GPU.

make all    # build base image → build the wheel → run the GPU smoke tests

Why this exists

pip install onnxruntime-gpu ships only x86_64 + Windows wheels. For aarch64 + CUDA 13 + sm_87 there is no prebuilt GPU wheel anywhere — every candidate was checked and fails on Orin:

Source	Wheel	Result on Jetson Orin sm_87
PyPI	onnxruntime-gpu	x86_64 / Windows only — no aarch64
jetson-ai-lab	sbsa/cu130 1.24.0	SBSA sm_90 (Grace), not Tegra sm_87
HuggingFace	…cuda13-sm121	sm_121 (Thor/Blackwell)
ultralytics/assets	1.24.0-cp312-…aarch64.whl	providers list looks right, but runtime: cudaErrorNoKernelImageForDevice

The lesson: get_available_providers() is a compile-time list — a non-sm_87 build still lists CUDAExecutionProvider but has no kernel image for the device. Only a real inference run tells the truth. This repo builds with CMAKE_CUDA_ARCHITECTURES=87 so the kernel is actually present, and ships a test that proves it.

What you get

• lab5/jp7-base:cuda13.2-trt10.16 — Ubuntu 24.04 + CUDA 13.2 + cuDNN 9 + TensorRT 10.16, assembled from the public Jetson apt repo (no NVIDIA login).
• onnxruntime_gpu-1.23.0-cp312-cp312-linux_aarch64.whl — CUDA + TensorRT EP, compiled for sm_87. Providers: [Tensorrt, CUDA, CPU].

Requirements

• NVIDIA Jetson Orin — developed on the Orin Nano Super (8 GB); any Orin sm_87 works (Nano / Nano Super / NX / AGX). JetPack 7.2 (L4T r39.2, CUDA 13.2, TRT 10.16).
• Docker with the NVIDIA runtime (--runtime nvidia)
• ~30 GB free disk, swap recommended (the onnxruntime build is memory-bound; 16 GB swap is enough on an Orin Nano)
• ~4 hours for the build (memory-bound, not CPU)

Build

All steps are plain docker build / docker run driven by a Makefile — no agent, no remote orchestration. Run on the Jetson host itself.

git clone <this-repo> jetson-jp7-onnxruntime
cd jetson-jp7-onnxruntime

make base    # 1. base image (CUDA 13.2 + TensorRT 10.16)
make ort     # 2. onnxruntime-gpu wheel from source (~hours) → dist/*.whl
make smoke   # 3. install the wheel + run tests/ on the GPU
make proof   # 4. (optional) TensorRT engine benchmark via trtexec

Or simply make all (= base + ort + smoke).

Targets

Target	What it does
make base	docker build the base image from docker/base/Dockerfile
make ort	runs scripts/build_onnxruntime.sh in the base image; wheel lands in dist/
make smoke	throwaway container: installs the wheel, runs every tests/test_*.py on the GPU
make proof	trtexec builds an FP16 engine from a tiny ONNX model and benchmarks it
make clean	removes dist/

Verify

make smoke passes when all three tests print OK:

• tests/test_providers.py — get_available_providers() returns [Tensorrt, CUDA, CPU].
• tests/test_cuda_ep.py — a tiny ONNX model runs on CUDAExecutionProvider. This is the test a non-sm_87 wheel fails with cudaErrorNoKernelImageForDevice.
• tests/test_trt_ep.py — the same model runs through the TensorRT provider.

The wheel is now in dist/ — copy it into any project that needs GPU ONNX inference on Jetson Orin (Frigate, a custom detector, etc.).

Troubleshooting

Build dies on a CUDA-deprecation warning (longlong4). CUDA 13 renames longlong4; ORT 1.23 still uses the old name and -Werror makes it fatal. The build script passes --compile_no_warning_as_error to downgrade it. (On ORT 1.27 with CUDA-13-native this is not needed.)

Build swaps for hours / OOM. cicc (the NVCC compiler) uses 5–10 GB per parallel job. The script caps this with --parallel 3 --nvcc_threads 1. If you still OOM, drop to --parallel 2. Add swap (16 GB is enough) before building.

Re-running the build restarts from scratch. Don't recreate the container — docker start <name> reuses the compiled tree. The clone in the script is idempotent ([ -d .git ] || git clone …).

CUDA error cudaErrorNoKernelImageForDevice at runtime. The wheel was not built for sm_87. Confirm the build used CMAKE_CUDA_ARCHITECTURES=87 — a prebuilt wheel from elsewhere (SBSA sm_90, sm_121, etc.) will list the providers but fail here. Use the wheel this repo builds.

Layout

docker/base/        base image: CUDA 13.2 + TensorRT 10.16 (public Jetson apt)
scripts/            build_onnxruntime.sh (source build), trt_proof.sh (TRT benchmark)
tests/              GPU smoke tests (providers, CUDA EP, TensorRT EP)
Makefile            reproducible build driver (base / ort / smoke / proof)

License

MIT, Copyright (c) 2026 dev. See LICENSE.