Skip to content

jetson‐inference container

Jump to bottom
Sergei Grichine edited this page Mar 9, 2026 · 2 revisions

Running the "jetson-inference" container on a Jetson Nano B01

This guide summarizes how to successfully run GPU-accelerated AI on a Jetson Nano (JetPack 4.6 / L4T R32.7.1) using Docker, bypassing the broken Python 3.6 scripts and missing manifests.

Note:

  • The machine, OS, and related experiments are described here.
  • There is also a way to use a different setup to serve requests from a ROS2 node.
  • Portions of this text are AI-generated, although carefully selected, reviewed, and edited by an inferior life form.

The Task

"AI inference is the operational phase where a trained machine learning model applies learned patterns to new, unseen data to make predictions, decisions, or generate content."

The goal is to find a way to run AI workloads on the now-outdated Jetson Nano. As a proof of concept, we attempt to run the following Docker image: dustynv/jetson-inference:r32.7.1 (see this repository).

If it works and uses the Nano's GPU (and it does), that proves a working AI stack still exists for this machine.

The process

The machine and OS installation process are described here.

Check JetPack Version: cat /etc/nv_tegra_release (Confirmed: R32.7.6):

cat /etc/nv_tegra_release
# R32 (release), REVISION: 7.6, GCID: 38171779, BOARD: t210ref, EABI: aarch64, DATE: Tue Nov  5 07:46:14 UTC 2024

Install prerequisites:

sudo -H pip3 install typing_extensions
sudo -H pip3 install wget

Clone jetson-containers:

cd; git clone https://github.com/dusty-nv/jetson-containers.git

Pull the specific image tagged for Nano's L4T version:

mkdir ~/my_data
mkdir ~/my_data_networks
docker run --runtime nvidia -it --rm \
  --network host \
  --volume ~/my_data:/data \
  --volume ~/my_data_networks:/jetson-inference/data \
  dustynv/jetson-inference:r32.7.1
...
 downloading and extracting 3.52 GB image
...
Status: Downloaded newer image for dustynv/jetson-inference:r32.7.1
(we are in container's shell now):
root@jetson:/jetson-inference#

We need to perform certain operations in the container shell:

  • Download models:
# create folders and copy files where the download script expects them:
root@jetson:/jetson-inference# mkdir -p /jetson-inference/data/networks
root@jetson:/jetson-inference# cp /jetson-inference/tools/bvlc_googlenet.caffemodel /jetson-inference/data/networks/.
root@jetson:/jetson-inference# cp /jetson-inference/tools/googlenet.prototxt /jetson-inference/data/networks/.
# we need to "fake" a labels file so the engine could initialize:
root@jetson:/jetson-inference# python3 -c "with open('/jetson-inference/data/networks/labels.txt', 'w') as f: [f.write(f'class_{i}\n') for i in range(1000)]"
root@jetson:/jetson-inference# ll /jetson-inference/data/networks/labels.txt
-rw-r--r-- 1 root root 9890 Feb 26 17:17 /jetson-inference/data/networks/labels.txt
root@jetson:/jetson-inference#

Now we can download models (press Enter to select ):

root@jetson:/jetson-inference# cd /jetson-inference/tools
root@jetson:/jetson-inference/tools# ./download-models.sh
Screenshot from 2026-02-26 11-05-57
  • Perform object recognition (inference):
cd /jetson-inference/build/aarch64/bin
./imagenet --model=/jetson-inference/data/networks/bvlc_googlenet.caffemodel \
           --prototxt=/jetson-inference/data/networks/googlenet.prototxt \
           --labels=/jetson-inference/data/networks/labels.txt \
           --input_blob=data \
           --output_blob=prob \
           /jetson-inference/examples/my-recognition/polar_bear.jpg \
           /data/output_bear.jpg

It takes a while the first time. Check jtop in another terminal. It should show some GPU usage, even in a "headless" (sudo init 3) mode.

Check the result:

root@jetson:/jetson-inference/build/aarch64/bin# ll /data
total 396
-rw-r--r-- 1 root root 396457 Feb 26 18:02 output_bear.jpg

The resulting image shows up in the mounted host's volume, you can view it (if in sudo init 5 state):

jetson@jetson:~$ ll ~/my_data
total 168
-rw-r--r--  1 root   root   161688 Feb 26 11:41 output_bear.jpg

You can repeat the command, using "brown_bear.jpg" or "black_bear.jpg" as input. It runs fast. You can place other images in host's ~/my_data and use those images as input.

output_bear

Note:

  • Once you exit the container shell, the Docker image will survive, but all work you did inside the container will be gone, except for the mounted volumes - the host's ~/my_data and ~/my_data_networks folders. So, the downloaded models and related configuration data will persist.
  • If via SSH - copy to your workstation: scp jetson@jetson.local:~/my_data/output_bear.jpg ~/Downloads/

Subsequent runs

Because you have all configuration and models saved on the host's volumes, repeated runs are easy:

# run the container with volumes mounted (comes up fast):
docker run --runtime nvidia -it --rm \
  --network host \
  --volume ~/my_data:/data \
  --volume ~/my_data_networks:/jetson-inference/data \
  dustynv/jetson-inference:r32.7.1
root@jetson:/jetson-inference#

# Run the inference:
root@jetson:/jetson-inference# cd /jetson-inference/build/aarch64/bin
root@jetson:/jetson-inference/build/aarch64/bin# ./imagenet --model=/jetson-inference/data/networks/bvlc_googlenet.caffemodel \
            --prototxt=/jetson-inference/data/networks/googlenet.prototxt \
            --labels=/jetson-inference/data/networks/labels.txt \
            --input_blob=data \
            --output_blob=prob \
            /jetson-inference/examples/my-recognition/polar_bear.jpg \
            /data/output_bear.jpg

Explanation: The Stack

here is the stack from the Inference Runner down to the Jetson Nano hardware.

1. The Application Layer (Inference Runner)

Component: imagenet (C++ / Python)

Role: This is the high-level code. It loads the image, talks to the camera, and handles the input/output logic. It doesn't "know" how to do math; it just asks the next layer to do it.

2. The Optimization Layer (TensorRT)

Component: NVidia TensorRT V8.2.1 (The "Engine Builder")

Role: This is the most important layer for Jetson. It takes a generic model (Caffe, ONNX, or TensorFlow) and re-writes it into a .engine file optimized specifically for the Nano’s Maxwell architecture. Why it's unique: It chooses the fastest "tactics" (math paths) for your specific GPU memory and core count.

3. The Math Libraries (CUDA / cuDNN)

Component: CUDA Toolkit V10.2.460 & cuDNN V8.2.1 (Deep Neural Network library)

Role: These are the "primitives." If the model says "multiply these two massive matrices," CUDA provides the thousands of small programs (kernels) that run in parallel to do that math.

4. The Bridge (NVIDIA Container Runtime)

Component: nvidia-container-runtime V1.1.4-0 (The Docker "Glue")

Role: Standard Docker containers cannot see hardware. This layer acts as a "portal," mapping the GPU device drivers from your Ubuntu host into the isolated environment of the container so imagenet can see the GPU.

When you run a container with --runtime nvidia, the Jetson automatically mounts these files from your Host into the Container at runtime:

  • CUDA 10.2 / cuDNN 8.2.1 / TensorRT 8.2.1: These are actually installed on your Jetson's local disk, but the container "links" to them so it doesn't have to carry 2GB of math libraries inside the image.

If this version is older than 3.0 (and it is), many modern Docker images (even those for Jetson) will fail to see the GPU entirely.

5. The Driver Layer (L4T R32.7.1 / Linux for Tegra)

Component: NVIDIA Display Driver & CUDA Driver (on host)

Role: This is part of your JetPack 4.6 install. It communicates directly with the Linux kernel to manage power, clocks (frequency), and memory allocation for the GPU.

6. The Hardware (The "Silicon")

Component: Tegra X1 SoC (Maxwell Architecture, 2019, up to 921 MHz, 472 Gigaflops of FP16 AI performance)

Role: 128 CUDA cores. This is where the actual electricity moves. It processes the instructions provided by the CUDA kernels to classify your polar bear.

Why this specific stack matters for you

  • The "Ollama" Problem: Most ollama Docker images are built for CUDA 11.x or 12.x. Jetson Nano hardware is physically capped at CUDA 10.2. This is why without extra agruments the manifest will be "unknown"—the image was built for a newer "language" than the Nano speaks.
  • The "Python" Problem: This stack (JetPack 4.6) is hard-coded to Python 3.6. Newer AI tools require Python 3.8+, creating the "dependency hell".

The Path Forward: To run a Large Language Model (LLM) on this stack, you need a runner that specifically supports CUDA 10.2.

llama.cpp is currently the best (and only) reliable way to do that on this specific version of the stack on Nano. See this guide for details.

Back to Home

Clone this wiki locally