ATOM + Dynamo integration on AMD MI355X

Reproducible integration and evidence for running NVIDIA Dynamo v1.3.0 with AMD ATOM v0.1.6-rc0 on two AMD Instinct MI355X nodes.

The project validates both ordinary ATOM-backed Dynamo serving and true disaggregated prefill/decode through upstream AMD-capable NIXL. It also records a separate MORI dma-buf correctness investigation that produced ROCm/mori#509.

Result at a glance

Capability	Result
Exact Dynamo 1.3.0 + ATOM 0.1.6-rc0 container	PASS
ATOM platform/model plugin active on MI355X	PASS
Single-node Qwen3-8B with prefix caching	8/8 requests
Two-node aggregated Qwen3-8B	32/32 requests
Qwen3.5 397B MXFP4 HTTP smoke	PASS
Two-node Dynamo P/D through upstream nixl_rocm	PASS
External prefix-cache hit on decode	100%
Successful NIXL KV transfer	2.25 MB in 1.588 ms
MORI sub-allocated dma-buf fix	16/16 requests in the separate SGLang case study

The NIXL run used prefix caching, max_num_seqs=32, one MI355X per worker, etcd discovery, TCP requests, and UCX/verbs for KV transfer. Both workers initialized upstream nixl_rocm; each logged 720 successful dma-buf registrations, with no registration failures or NIXL_ERR records.

Read the high-level achievement summary for the full story and the precise distinction between the MORI and NIXL paths.

Repository map

Dockerfile                         pinned ATOM + Dynamo base image
Dockerfile.nixl-rocm               upstream NIXL/ROCm UCX overlay
release-lock.json                  exact versions, commits, and hashes
config/cluster.env.example         cluster-specific settings
docs/achievement-summary.md        project-level technical summary
docs/mori-path.md                  MORI dma-buf root cause and fix
docs/nixl-path.md                  Dynamo P/D architecture and procedure
evidence/                          compact, sanitized success evidence
examples/mori/                     ionic dma-buf loopback reproducer
patches/                           explicit NIXL PoC compatibility patch
scripts/                           build, validation, and Slurm launchers
results/RESULTS.md                 chronological experiment record

Binary wheels, model checkpoints, vendored upstream trees, container archives, and full raw cluster logs are deliberately excluded from Git history. Exact release wheels are available as checksummed GitHub release assets.

Pinned stack

Component	Pin
Dynamo	v1.3.0, 8ce9e22f11576402102ea9d8b8e46233f5430a0d
ATOM	v0.1.6-rc0, f9017b58b8369a55d14e153b51cadd81ab2ba65a
ATOM base	rocm/atom-dev@sha256:3d0c714b...53537b0
vLLM	ATOM-tested 0b3ba88f165976e77ca5e6a7a3f5bba4562b80af
ROCm UCX	b56982d8076ac54eb9a0d4a7dbea7852ede75508
upstream NIXL	53be1d43a415ff3a14b8bc942ca7242307adb70d

See release-lock.json for complete hashes. Dynamo advertises vLLM 0.23 for this release, but ATOM was tested against its own AMD-modified vLLM 0.22 snapshot. The image intentionally preserves ATOM's ROCm/AITER/Triton/vLLM stack and installs the exact Dynamo/ATOM release wheels with --no-deps.

Prerequisites

• x86-64 AMD MI355X compute nodes with /dev/kfd and /dev/dri;
• Docker access on the compute nodes;
• RDMA devices under /dev/infiniband for the NIXL path;
• a shared filesystem visible at the same path on both nodes;
• Slurm for the included launchers;
• an etcd 3.6.x server and client binary on the shared filesystem;
• a shared model checkpoint, such as Qwen3-8B.

The examples were validated on Crusoe SPUR with AMD Pensando ionic RoCE NICs. Cluster names, accounts, paths, and placement behavior are environment-specific.

Configure

cp config/cluster.env.example cluster.env
$EDITOR cluster.env
source cluster.env

The most important variables are PROJECT_DIR, SHARED_DIR, MODEL_PATH, Slurm account/partition/QoS, and the two node names. DYNAMO_ATOM_IMAGE names the aggregated base image; NIXL_IMAGE names the disaggregated overlay. Batch scripts retain the validated Crusoe #SBATCH defaults, while submission wrappers pass the environment-provided Slurm settings explicitly.

Build the ATOM + Dynamo image

Fetch and verify the exact wheels:

bash scripts/fetch_release_wheels.sh

Build locally on each target node because the validated Crusoe setup uses a node-local Docker image cache:

bash scripts/build_image.sh

The default tag is:

dynamo-atom-mi355x:1.3.0-0.1.6rc0

Run the low-cost GPU/plugin validation before loading a model:

sbatch \
  --account="$SLURM_ACCOUNT" \
  --partition="$SLURM_PARTITION" \
  --qos="$SLURM_QOS" \
  scripts/gpu_validate_on_spur.sbatch

The check fails unless Dynamo and ATOM versions match, Torch is a ROCm build, ATOM is the active vLLM platform, ATOM owns the Qwen3 model registration, and Dynamo's frontend/vLLM adapter imports.

Aggregated smoke tests

Single node:

sbatch \
  --account="$SLURM_ACCOUNT" \
  --partition="$SLURM_PARTITION" \
  --qos="$SLURM_QOS" \
  --nodelist="$PRIMARY_NODE" \
  scripts/single_node_smoke.sbatch

Two separately pinned workers:

bash scripts/submit_two_node_poc.sh

The two-job layout is intentional. On the validation cluster, a nominal two-node/multi-task srun did not reliably place one role on each node.

Build and run upstream NIXL P/D

The complete procedure and success criteria are in docs/nixl-path.md. In outline:

# Build pinned ROCm UCX + upstream NIXL.
sbatch \
  --account="$SLURM_ACCOUNT" \
  --partition="$SLURM_PARTITION" \
  --qos="$SLURM_QOS" \
  --nodelist="$PREFILL_NODE" \
  scripts/nixl_rocm_source_build.sbatch

# Set SOURCE_RUN to that job's result directory, then build the overlay on
# every node that may host a worker.
sbatch --export=ALL,SOURCE_RUN="$SOURCE_RUN" \
  --nodelist="$PREFILL_NODE" scripts/build_nixl_overlay.sbatch
sbatch --export=ALL,SOURCE_RUN="$SOURCE_RUN" \
  --nodelist="$DECODE_NODE" scripts/build_nixl_overlay.sbatch

# Build the ionic dma-buf helper into the shared filesystem.
sbatch --nodelist="$PREFILL_NODE" scripts/build_dmabuf_shim.sbatch

# Start one prefill and one decode job joined through etcd.
bash scripts/submit_disagg_two_job_etcd.sh

The pinned ATOM/vLLM connector may import the legacy module name rixl on ROCm. The overlay contains only a thin compatibility re-export; the actual implementation is upstream nixl_rocm. This repository does not use RIXL as the transfer implementation.

MORI case study

MORI is not the connector used by the final Dynamo P/D run. The parallel MORI investigation found a general correctness issue for sub-allocated GPU buffers on ionic: dma-buf registration must use the nonzero byte offset returned by hsa_amd_portable_export_dmabuf. See docs/mori-path.md and the standalone loopback in examples/mori/.

Scope and cautions

• This is a validated PoC, not a production deployment recipe.
• The NIXL result is one end-to-end smoke request, not a throughput sweep.
• The reported request rates are observations, not tuned benchmarks.
• The NIXL HOST-classification patch is narrow PoC compatibility glue and should disappear when UCX/NIXL recognizes this ROCm dma-buf path natively.
• The dma-buf preload helper is experimental. Prefer upstream native fixes.
• Qwen3-8B BF16 and one Qwen3.5 MXFP4 smoke were validated; broad model, quantization, TP, and concurrency coverage remains future work.
• The repository contains no model weights or credentials.

License

The integration code and documentation in this repository are licensed under Apache-2.0. Upstream components retain their own licenses; see THIRD_PARTY.md.