Dependency Version Pinning for Reproducible Builds

[youtalk]

Motivation

Autoware's build reproducibility is currently not guaranteed. While some dependencies are already pinned (CUDA/TensorRT via .env, source repos via autoware.repos tags/SHAs, pre-commit hooks via rev:), many critical dependencies are resolved dynamically at build time:

Category	Current State	Pinned?
CUDA / TensorRT	Explicit in .env	Yes
Source repos	autoware.repos tags/SHAs	Yes
Pre-commit hooks	rev: in config	Yes
ML model artifacts	SHA256 checksum verification	Yes
ROS packages (APT)	Dynamic via rosdep	No
System APT packages	Ansible state: latest	No
Python packages	APT or pipx, mostly unversioned	Partial
Docker base images	:latest tag	No
ROS 2 APT source	GitHub API "latest" release	No
pip packages in Dockerfiles	Some unversioned (e.g., yamale)	No

This means that two builds from the same commit on different days can produce different results, making release management unreliable and debugging regressions difficult.

Goals

1. Reproducible builds: Given the same commit, produce identical build environments regardless of when the build runs
2. Backward compatibility: Default behavior (without opt-in flags) remains unchanged
3. Low operational overhead: Lock files are auto-generated and updated via CI
4. Flexibility: Developers use latest packages by default; locked versions are opt-in for releases

Non-Goals

• Bit-for-bit reproducibility (that would require Nix/Guix, which is out of scope)
• Pinning transitive APT dependencies (APT handles these; we pin the top-level packages)
• Replacing the existing build infrastructure (we extend it, not replace it)

Approach: Lock File Method

I evaluated three approaches for each setup type (Docker and native):

Docker Setup

Aspect	A: Lock Files	B: APT Snapshots	C: Custom APT Repo
Reproducibility	Good	Excellent	Excellent
Implementation complexity	Low	Medium-High	High
Compatibility with existing infra	Excellent	Fair	Poor
Operational cost	Low	Low	High
External dependency	None	snapshot.ubuntu.com	Self-hosted infra
Security updates	Easy	Complex	Medium
Recommendation	Selected	-	-

Native Setup

Aspect	A: Ansible Lock Files	B: System Images (Packer)	C: Nix/Guix
Reproducibility	Good	Excellent	Excellent
Implementation complexity	Low-Medium	Medium	High
Compatibility with existing infra	Excellent	Poor	Poor
Customizability	Excellent	Poor	Fair
ROS 2 integration	Excellent	Excellent	Poor
Learning curve	Low	Medium	High
Recommendation	Selected	-	-

Selected approach: Lock file method for both Docker and native setups. It provides the best balance of reproducibility, implementation simplicity, and compatibility with the existing Ansible + Docker infrastructure.

Design Overview

Architecture

Both native and Docker setups share the same dependency installation flow:

1. setup-dev-env.sh
2. Ansible
3. Base APT packages
4. vcs import
5. Source code (autoware.repos)
6. rosdep install
7. ROS dependency packages (from package.xml)
8. colcon build

Therefore, Ansible lock files and rosdep lock files are shared between native and Docker. Docker only adds image digest pinning and Dockerfile-specific pip pinning.

flowchart TB
    subgraph shared["Shared Lock Files"]
        ansible_lock["ansible/vars/<br/>locked-versions-{distro}-{arch}.yaml"]
        rosdep_lock["lockfiles/{arch}/<br/>rosdep-resolved-{distro}.lock"]
    end

    subgraph native["Native Setup"]
        native_cmd["setup-dev-env.sh<br/>--locked"]
    end

    subgraph docker["Docker Setup"]
        docker_cmd["setup-dev-env.sh<br/>--locked"]
        docker_extra["+ Base image digest<br/>+ pip version pins"]
    end

    shared --> native
    shared --> docker

    style shared fill:#ffffcc,stroke:#999
    style native fill:#e6f3ff,stroke:#999
    style docker fill:#e6ffe6,stroke:#999

Loading

Lock File Types

Lock File	Used By	Purpose
ansible/vars/locked-versions-{distro}-{arch}.yaml	Both	Pins APT and pipx packages installed by Ansible roles
lockfiles/{arch}/rosdep-*-{distro}.lock	Both	Pins rosdep-resolved APT packages
Base image digest in .env	Docker only	Pins Docker base images
pip version pins in tools Dockerfiles	Docker only	Pins pip packages in tool images

How It Works

Native Setup

# Development (default, unchanged behavior)
./setup-dev-env.sh -y

# Release (locked versions)
./setup-dev-env.sh -y --locked

When --locked is specified, setup-dev-env.sh:

1. Locates ansible/vars/locked-versions-{distro}-{arch}.yaml
2. Passes it to Ansible via --extra-vars @lockfile (highest precedence)
3. The lock file sets use_locked_versions: true, which triggers version-pinned installation in each Ansible role

Docker Setup

# Development (default, unchanged behavior)
docker build -t autoware:dev .

# Release (locked versions)
docker build --build-arg USE_LOCKFILE=true -t autoware:release .

When USE_LOCKFILE=true:

1. setup-dev-env.sh --locked is called inside the container
2. Pre-resolved rosdep lock files are used instead of dynamic rosdep resolve
3. Base images are referenced by digest (when configured in .env)

Ansible Role Modification Pattern

Each role that installs packages with state: latest gets a conditional branch:

# Locked: install exact version
- name: Install packages (locked)
  ansible.builtin.apt:
    name: "{{ locked_package_list }}"  # e.g., ["pkg1=1.2.3-1", "pkg2=4.5.6-1"]
    state: present
    allow_downgrade: true
    update_cache: true
  when: use_locked_versions | default(false) | bool

# Default: install latest (existing behavior, unchanged)
- name: Install packages
  ansible.builtin.apt:
    name:
      - pkg1
      - pkg2
    state: latest
    update_cache: true
  when: not (use_locked_versions | default(false) | bool)

Lock File Format

Ansible lock file (ansible/vars/locked-versions-humble-amd64.yaml):

# Auto-generated. Do not edit manually.
# Generated: 2026-02-26T00:00:00Z
# Platform: humble-amd64
use_locked_versions: true

locked_ros_apt_source_version: "0.5.1"
locked_ros2_package_version: "0.10.0-1jammy.20260220.000000"

locked_ros2_dev_tools_packages:
  - "python3-colcon-mixin=0.2.3-1"
  - "python3-flake8-docstrings=1.6.0-2"
  - "python3-rosdep=0.25.1-1"
  # ...

locked_ccache_version: "4.5.1-1"
locked_rmw_package_version: "0.25.6-1jammy.20260101.000000"
# ...

rosdep lock file (lockfiles/amd64/rosdep-core-common-depend-humble.lock):

# Auto-generated. Do not edit manually.
# Generated: 2026-02-26T00:00:00Z
# ROS Distro: humble
ros-humble-ament-cmake-auto=1.3.7-1jammy.20260110.123456
ros-humble-rclcpp=21.2.0-1jammy.20260112.234567
ros-humble-tf2-ros=0.31.3-1jammy.20260111.345678
...

New File Structure

autoware/
├── ansible/
│   └── vars/                                    # NEW
│       ├── locked-versions-humble-amd64.yaml
│       ├── locked-versions-humble-arm64.yaml
│       └── locked-versions-jazzy-amd64.yaml
├── lockfiles/                                   # NEW
│   ├── amd64/
│   │   ├── rosdep-core-common-depend-humble.lock
│   │   ├── rosdep-core-common-exec-depend-humble.lock
│   │   └── ...
│   └── arm64/
│       └── ...
├── scripts/                                     # NEW
│   ├── generate-lockfiles.sh
│   ├── generate-rosdep-lockfiles.sh
│   └── validate-lockfiles.sh
├── setup-dev-env.sh                             # MODIFIED: add --locked option
├── amd64.env                                    # MODIFIED: add ros_apt_source_version, digests
├── amd64_jazzy.env                              # MODIFIED: same
├── arm64.env                                    # MODIFIED: same
├── docker/
│   ├── Dockerfile.base                          # MODIFIED: USE_LOCKFILE support
│   ├── Dockerfile                               # MODIFIED: USE_LOCKFILE + rosdep lock
│   ├── build.sh                                 # MODIFIED: --locked flag
│   └── tools/
│       ├── visualizer/Dockerfile                # MODIFIED: pin yamale, xmlschema
│       └── scenario-simulator/Dockerfile        # MODIFIED: pin yamale
└── .github/workflows/
    ├── generate-lockfiles.yaml                  # NEW: monthly lock file generation
    └── validate-lockfiles.yaml                  # NEW: lock file validation on PR

Risks and Mitigations

Risk	Impact	Mitigation
Lock file becomes stale (new deps added to package.xml)	Locked build fails	Monthly auto-regeneration; fallback to latest with warning if locked version missing
PPA drops old package versions (kisak_mesa)	Locked build fails	Monthly updates keep versions fresh; PPA retention is typically long
Architecture name mismatch (x86_64 vs amd64)	Wrong lock file loaded	Normalize via dpkg --print-architecture consistently
Ansible set_fact overrides --extra-vars	Locked version ignored	Set fact conditionally (locked path first) in affected roles
Lock file merge conflicts	Developer friction	Lock files are auto-generated, so conflicts are resolved by regeneration

[youtalk]

I'm planning to proceed with the following phases.

Phase 1: Foundation

Task	Deliverable
Add --locked option to setup-dev-env.sh	setup-dev-env.sh
Add ros_apt_source_version to .env files	amd64.env, amd64_jazzy.env, arm64.env
Create lock file generation scripts	scripts/generate-lockfiles.sh, scripts/generate-rosdep-lockfiles.sh
Create lock file validation script	scripts/validate-lockfiles.sh

Phase 2: Ansible Role Modifications

Role	Complexity	Changes
ros2	High	Skip GitHub API call when locked; pin ros-apt-source and ROS 2 package versions
ros2_dev_tools	Low	Pin 16 packages via lock file list
build_tools	Low	Pin ccache version
dev_tools	Medium	Pin APT packages + pipx tools (pre-commit, clang-format)
geographiclib	Low	Pin geographiclib-tools version
rmw_implementation	Low	Pin RMW package version
gdown	Low	Pin pipx + gdown versions
kisak_mesa	Low	Pin 8 Mesa packages
docker_engine	Low	Pin docker-ce and related packages
nvidia_container_toolkit	Low	Pin toolkit version
telegraf	Low	Pin influxdb2, telegraf versions

No changes needed for already-pinned roles: tensorrt, cuda, agnocast, spconv, acados, artifacts, qt5ct_setup.

Phase 3: Docker Implementation

Task	Deliverable
Add base image digests to .env files	amd64.env, arm64.env
Add USE_LOCKFILE support to Dockerfile.base	docker/Dockerfile.base
Add rosdep lock file support to Dockerfile	docker/Dockerfile
Pin pip packages in tools Dockerfiles	docker/tools/visualizer/Dockerfile, docker/tools/scenario-simulator/Dockerfile
Add --locked flag to docker/build.sh	docker/build.sh

Phase 4: CI/CD

Task	Deliverable
Monthly lock file generation workflow	.github/workflows/generate-lockfiles.yaml
Lock file validation workflow	.github/workflows/validate-lockfiles.yaml
(Future) Add locked build to health-check	.github/workflows/health-check.yaml

Lock File Lifecycle

1. Monthly CI Schedule or manual trigger
2. Build in container for each platform (distro x arch)
3. Run generate scripts dpkg-query, rosdep resolve, pip freeze
4. Create PR with updated lock files
5. Review & merge (validate workflow runs automatically)
6. Release branch uses--locked / USE_LOCKFILE for pinned builds

Security Update Process

1. Monitor security advisories for affected packages
2. Update the specific package version in the lock file
3. Run validation workflow
4. Merge and tag release

[xmfcx]

Hi, thanks for coordinating this task!

• Right now I am working on this issue: Simplify the docker images and workflows #6852

With this I'll eliminate the .env files, setup-dev-env.sh file and update the docker build workflows in general.

Once my task is finished, this lock file integration could be more easily accomplished.

[xmfcx]

Great proposal overall! One concern I wanted to flag regarding the ROS APT package pinning. The ROS build farm only retains the latest version of each package in its repo. Unlike NVIDIA's CUDA repo which keeps multiple versions available, packages.ros.org replaces packages in-place:

# ROS: only one version available
$ apt-cache policy ros-jazzy-rviz2
ros-jazzy-rviz2:
  Installed: 14.1.19-1noble.20260126.201007
  Candidate: 14.1.19-1noble.20260126.201007
  Version table:
 *** 14.1.19-1noble.20260126.201007 500
        500 http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages

# CUDA: multiple versions retained
$ apt-cache policy cuda
cuda:
  Installed: (none)
  Candidate: 13.1.1-1
  Version table:
     13.1.1-1 600
     13.1.0-1 600
     ...

This means a rosdep lock file like ros-humble-rclcpp=21.2.0-1jammy.20260112.234567 would become un-"installable" as soon as the build farm publishes a newer version, potentially within days of generation.

This affects the rosdep lock files, locked_ros2_package_version, locked_ros2_dev_tools_packages, and anything else sourced from packages.ros.org.

You may already be aware of this, but wanted to make sure it's accounted for.

[mojomex]

Robostack solves this and can be used through Pixi!

[youtalk]

@xmfcx cc @mojomex
It's an important point. To address this, I plan to use snapshots.ros.org in locked mode. The ROS build farm maintains immutable snapshot repositories at http://snapshots.ros.org/{distro}/{YYYY-MM-DD}/ubuntu for example https://snapshots.ros.org/humble/2026-01-07/ubuntu/.
This is the same approach used by OSRF's official Docker images for reproducible builds.

[mojomex]

@youtalk That's nice, I haven't been able to find this previously.
Just a follow-up to my comment under the corresponding #6862 issue:

Have you had a chance to look at Pixi, which looks like it could solve version pinning with even less effort? What are your thoughts?

[mojomex]

FYI: https://github.com/orgs/autowarefoundation/discussions/7050 seems very promising, installing Autoware on top of a (now officially supported) Pixi workspace.