Is triton server images ready for Jetson AGX Thor

[jilongliao]

I was trying to run nvcr.io/nvidia/tritonserver:25.08-py3-igpu on the newly released Jetson AGX Thor which only supports CUDA 13 w/ Jetpack 7.

When I first pull the container image, I realize it's still on CUDA 12.x, so I went on to build my own container with the following Dockerfile.

# This Dockerfile builds a Triton Inference Server image with
# dependencies of our Python model backends installed.
ARG TRITON_VERSION=25.08-py3-igpu

# Base image for everything
FROM nvcr.io/nvidia/tritonserver:${TRITON_VERSION} AS base

WORKDIR /app
COPY requirements.txt /app

RUN pip install --no-cache-dir -r /app/requirements.txt
RUN pip install --no-cache-dir https://pypi.jetson-ai-lab.io/sbsa/cu130/+f/bbe/d2ec4262e5dbd/torch-2.9.0-cp312-cp312-linux_aarch64.whl#sha256=bbed2ec4262e5dbd4f9803b0895593f73d040d34df15ce79f9f1b53a476f6cc0
RUN pip install --no-cache-dir https://pypi.jetson-ai-lab.io/sbsa/cu130/+f/c26/fb4d05c0d694c/torchvision-0.24.0-cp312-cp312-linux_aarch64.whl#sha256=c26fb4d05c0d694cc24812fccd3b52555544744538a943ba78d07506f1ea79b6

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb \
    && dpkg -i cuda-keyring_1.1-1_all.deb \
    && apt-get update \
    && apt-get -y install cuda-toolkit-13-0 nvpl cusparselt-cuda-13 libnccl2 libnccl-dev libgl1 libnvinfer-plugin10 libopenblas-dev libpng-dev libjpeg-dev \
    && rm -rf /var/lib/apt/lists/*

ENV LD_LIBRARY_PATH=/usr/local/cuda-13.0/targets/sbsa-linux/lib/:$LD_LIBRARY_PATH
ENV PATH=/usr/local/cuda-13.0/bin:$PATH
ENV CUDA_HOME=/usr/local/cuda-13.0

CMD [ \
        "/opt/tritonserver/bin/tritonserver", \
        "--model-repository=/models", \
        "--model-control-mode=explicit", \
        "--load-model=*" \
    ]

I also built pytorch backends from Jetson AGX Thor with CUDA 13 and point triton to use the pytorch backends i built.

Afterwards, my pytorch models are loaded fine on Triton server. When I ran the inference, triton server crashes at runtime with the following error:

Invalid handle. Cannot load symbol cublasLtCreate Signal (6) received. Signal (6) received. Signal (6) received. Signal (6) received. 0# 0x0000AAAAC8540490 in /opt/tritonserver/bin/tritonserver 1# __kernel_rt_sigreturn in linux-vdso.so.1 2# 0x0000FFFFA3FD7608 in /usr/lib/aarch64-linux-gnu/libc.so.6 3# gsignal in /usr/lib/aarch64-linux-gnu/libc.so.6 4# abort in /usr/lib/aarch64-linux-gnu/libc.so.6 5# 0x0000FFFA371F3930 in /usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2 6# 0x0000FFFA37B18A5C in /usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2 7# 0x0000FFFA37202EDC in /usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.10.2 8# cudnn::backend::Engine::finalize_internal() in /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2 9# cudnn::backend::Descriptor::finalize() in /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2 10# cudnn::backend::EngineConfig::finalize_internal() in /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2 11# cudnn::backend::EngcfgTmpVar::Results::emplace_back_on_success(cudnn::backend::EngineConfig&&, cudnn::backend::EngcfgTmpVar::ErrorStack&) in /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2 12# cudnn::backend::EngineHeuristics::get_internal(cudnnBackendAttributeName_t, cudnnBackendAttributeType_t, long, long*, void*) const in /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2 13# cudnnBackendGetAttribute in /usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.10.2 14# cudnnBackendGetAttribute in /usr/lib/aarch64-linux-gnu/libcudnn.so.9 15# 0x0000FFFD24647100 in /backends/pytorch/libtorch_cuda.so 16# 0x0000FFFD2464A110 in /backends/pytorch/libtorch_cuda.so 17# 0x0000FFFD2464A3C4 in /backends/pytorch/libtorch_cuda.so 18# 0x0000FFFD2463DEA4 in /backends/pytorch/libtorch_cuda.so 19# 0x0000FFFD24657EB8 in /backends/pytorch/libtorch_cuda.so 20# 0x0000FFFD244089C8 in /backends/pytorch/libtorch_cuda.so 21# 0x0000FFFD244073FC in /backends/pytorch/libtorch_cuda.so 22# at::native::cudnn_convolution(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, bool) in /backends/pytorch/libtorch_cuda.so 23# 0x0000FFFD2679F124 in /backends/pytorch/libtorch_cuda.so 24# 0x0000FFFD24429EEC in /backends/pytorch/libtorch_cuda.so 25# at::_ops::cudnn_convolution::call(at::Tensor const&, at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::SymInt, bool, bool, bool) in /backends/pytorch/libtorch_cpu.so 26# at::native::_convolution(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long, bool, bool, bool, bool) in /backends/pytorch/libtorch_cpu.so 27# 0x0000FFFD4C496654 in /backends/pytorch/libtorch_cpu.so 28# 0x0000FFFD4C4969A0 in /backends/pytorch/libtorch_cpu.so 29# at::_ops::_convolution::call(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, bool, c10::ArrayRef<c10::SymInt>, c10::SymInt, bool, bool, bool, bool) in /backends/pytorch/libtorch_cpu.so 30# at::native::convolution(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, bool, c10::ArrayRef<long>, long) in /backends/pytorch/libtorch_cpu.so 31# 0x0000FFFD4C4964F4 in /backends/pytorch/libtorch_cpu.so 32# 0x0000FFFD4C4968B0 in /backends/pytorch/libtorch_cpu.so 33# at::_ops::convolution::call(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, bool, c10::ArrayRef<c10::SymInt>, c10::SymInt) in /backends/pytorch/libtorch_cpu.so 34# at::native::conv2d_symint(at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::SymInt) in /backends/pytorch/libtorch_cpu.so 35# 0x0000FFFD4DA44B1C in /backends/pytorch/libtorch_cpu.so 36# 0x0000FFFD50D0F0C8 in /backends/pytorch/libtorch_cpu.so 37# 0x0000FFFD50951734 in /backends/pytorch/libtorch_cpu.so 38# 0x0000FFFD50954E68 in /backends/pytorch/libtorch_cpu.so 39# 0x0000FFFD50936594 in /backends/pytorch/libtorch_cpu.so 40# torch::jit::Method::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, c10::IValue> > > const&) const in /backends/pytorch/libtorch_cpu.so 41# 0x0000FFFD53E65784 in /backends/pytorch/libtriton_pytorch.so 42# 0x0000FFFD53E59330 in /backends/pytorch/libtriton_pytorch.so 43# 0x0000FFFD53E5BA90 in /backends/pytorch/libtriton_pytorch.so 44# TRITONBACKEND_ModelInstanceExecute in /backends/pytorch/libtriton_pytorch.so 45# 0x0000FFFFA4A94D74 in /opt/tritonserver/bin/../lib/libtritonserver.so 46# 0x0000FFFFA4A95490 in /opt/tritonserver/bin/../lib/libtritonserver.so 47# 0x0000FFFFA4B70098 in /opt/tritonserver/bin/../lib/libtritonserver.so 48# 0x0000FFFFA4A9859C in /opt/tritonserver/bin/../lib/libtritonserver.so 49# 0x0000FFFFA42E1AE0 in /usr/lib/aarch64-linux-gnu/libstdc++.so.6 50# 0x0000FFFFA3FD595C in /usr/lib/aarch64-linux-gnu/libc.so.6 51# 0x0000FFFFA403B7DC in /usr/lib/aarch64-linux-gnu/libc.so.6

Can I get some help to resolve this issue?

[yinggeh]

I think it's now supported for the latest release. cc @mc-nv to confirm.