Can NVIDIA Jetson Orin/Thor be used as the host for model quantization and export?

[FABallemand]

The TensorRT Edge-LLM documenation states that "must run on an x86 Linux system with an NVIDIA GPU."

However, this Jetson AI Lab tutorial suggests that NVIDIA Jetson Thor is a supported alternative.

Can NVIDIA Jetson Orin or Jetson Thor be used as the host for model quantization and export? If so, are there any limitations or differences compared to using an x86 Linux host with an NVIDIA GPU?

[RenSolvyn]

Both of your sources are right, and the reconciliation is a Thor/Orin split. Short version: Jetson Thor yes, Jetson Orin no, and the docs sentence you quoted is the conservative supported matrix rather than a technical wall. I verified the mechanics from the repo and packaging before saying so.

The authoritative statement is NVIDIA's own Jetson AI Lab tutorial you referenced (https://www.jetson-ai-lab.com/tutorials/tensorrt-edge-llm/): its export step is labeled as running on x86 or Thor, and it states explicitly that the quantize-and-convert step needs significant GPU memory and Python, so it runs on an x86 workstation or Jetson Thor, and not on Orin devices. Orin remains fully supported as a deployment target in that same tutorial (Qwen3-4B INT4 on an Orin Nano 8GB), it just cannot host the export stage.

That is consistent with everything checkable in this repo, which is why I would trust it over the stricter installation.md line:

1. The x86 requirement is scoped, not global. installation.md splits the product into two components: quantization plus tensorrt_edgellm (the part carrying the "must run on an x86 Linux system with an NVIDIA GPU" sentence, with a CC 8.0+ GPU requirement), and the C++ runtime, whose supported list is Jetson Thor, NVIDIA DRIVE, DGX Spark, or an optional x86 developer build. Engine building is already on-device by design; the tts example says outright that engine builds run on the edge device, because TensorRT engines are hardware-specific. So your question only ever concerned the quantize/export stage.
2. Nothing in the code or packaging blocks aarch64. The pinned quantization dependency nvidia-modelopt==0.44.0 ships exactly one wheel on PyPI, py3-none-any, pure Python and platform-independent. requirements.txt is torch, transformers, onnx, onnxscript, safetensors, numpy, onnx-graphsurgeon, all available for aarch64. And a grep across tensorrt_edgellm/ finds no platform.machine or x86 gate anywhere; the quantize entry point just takes --device (default cuda). The "x86 host" line is a tested-matrix statement, and the Jetson AI Lab flow is NVIDIA extending that matrix to Thor.
3. Why Thor and not Orin, concretely: the tutorial's stated reason is GPU memory plus the Python stack. Thor (sm_110, CUDA 13 era, large unified memory) can hold the full-precision checkpoint during calibration and has a current PyTorch path, which the tutorial smooths further by providing a Thor Docker container for the export step. Orin technically meets the CC 8.0+ line (sm_87 is Ampere-class), so the exclusion is practical rather than compute-capability: full-precision calibration memory on 8 to 64 GB devices, and the JetPack-pinned Python/CUDA stack versus pins like torch==2.12.0.

So the practical answers to your two questions:

• Thor: yes, and it collapses the whole pipeline onto one box: quantize and export on Thor (use the tutorial's container path for the smoothest dependency story), build engines on the same device, run inference. Limitations versus an x86 host: you are on the Jetson AI Lab flow rather than the repo's tested matrix, so expect the container route to matter (getting the pinned CUDA torch natively on Thor outside it is the fiddly part), and calibration will be slower than a big discrete GPU, bounded by memory for larger models.
• Orin: no for hosting quantization/export; keep Orin as the build-and-inference side and produce the ONNX on x86 or a Thor.

Caveat for honesty: I verified the docs scoping, the wheel platforms, and the absence of code gates directly from the repo and PyPI, but I have not executed the export on Jetson hardware myself; the Thor-yes claim rests on NVIDIA's Jetson AI Lab tutorial doing exactly that. If a maintainer can confirm whether Thor-as-host is headed into the official installation.md matrix, that would resolve the doc contradiction you found at the source.