[AIDynamo] Runtime discovery

[podkidyshev]

Summary

Enable runtime environment discovery for AIDynamo for file-based config files

Test Plan

• Automated CI
• Manual runs

Additional Notes

[coderabbitai]

📝 Walkthrough

Walkthrough

AI Dynamo now supports optional per-node startup commands, including containerized execution, runtime configuration localization, and integration with Slurm workload and single-sbatch command generation.

Changes

AI Dynamo per-node startup

Layer / File(s)	Summary
Startup configuration and installables src/cloudai/workloads/ai_dynamo/ai_dynamo.py, doc/workloads/ai_dynamo.rst	Adds optional startup command and Docker image configuration, conditional image installation, and documentation for per-node startup and runtime replacements.
Per-node runtime localization src/cloudai/workloads/ai_dynamo/ai_dynamo.sh	Reads node runtime manifests, copies and replaces referenced configuration files, exports localized paths, and fails when localization errors occur.
Slurm startup execution src/cloudai/workloads/ai_dynamo/slurm_command_gen_strategy.py, tests/workloads/ai_dynamo/test_command_gen_strategy_slurm.py	Generates startup srun commands with optional containers, prepends them to workload execution, updates DCGM prefix handling, and tests normal and single-sbatch behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: jeffnvidia, srivatsankrishnan

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title check	❓ Inconclusive	It is related to the AIDynamo runtime-config changes, but "Runtime discovery" is too vague to convey the main change.	Use a more specific title such as "AIDynamo: add per-node startup command and runtime config localization".

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description clearly matches the AIDynamo runtime environment discovery changes described in the diff.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch ipod/dynamo-runtime-discovery

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/cloudai/workloads/ai_dynamo/ai_dynamo.sh`:
- Around line 635-656: Ensure the manifest-localization Python command fails the
surrounding shell flow when it cannot generate the environment file, before
executing source "$environment_file". Update the block around the embedded
Python script and its caller to use the existing failure-handling path
(including the Line 1410 handler), preventing an empty or partial generated file
from being sourced.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: cc5c2aca-e92f-414e-93d0-e55da2432380

📥 Commits

Reviewing files that changed from the base of the PR and between c30e75c and 2012c6a.

📒 Files selected for processing (5)

• doc/workloads/ai_dynamo.rst
• src/cloudai/workloads/ai_dynamo/ai_dynamo.py
• src/cloudai/workloads/ai_dynamo/ai_dynamo.sh
• src/cloudai/workloads/ai_dynamo/slurm_command_gen_strategy.py
• tests/workloads/ai_dynamo/test_command_gen_strategy_slurm.py

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

src/cloudai/workloads/ai_dynamo/ai_dynamo.sh (3)

646-656: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Validate manifest keys before generating shell code.

config_var is inserted directly into a sourced shell script. A malformed manifest key can inject arbitrary commands into the workload shell. Require ^[A-Za-z_][A-Za-z0-9_]*$ (and reject invalid keys) before emitting the export statement.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cloudai/workloads/ai_dynamo/ai_dynamo.sh` around lines 646 - 656,
Validate each config_var in the manifest loop before using it to generate the
export statement, requiring the shell-variable pattern ^[A-Za-z_][A-Za-z0-9_]*$
and rejecting invalid keys. Keep valid-key processing unchanged, and ensure
validation occurs before emitting any sourced shell code.

---

648-653: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Preserve restrictive permissions on localized configs.

write_text() creates the copy using the process umask, which can broaden access from a source mode such as 0600 to commonly 0644. Preserve the source mode or explicitly create localized files with restrictive permissions, since these paths may reference credential-bearing configuration.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cloudai/workloads/ai_dynamo/ai_dynamo.sh` around lines 648 - 653, Update
the localized-config write flow around target.write_text in the replacement loop
to preserve the source file’s restrictive permissions on the newly created
target. Apply the source mode explicitly after creating the target, or create it
with equivalent restrictive permissions, ensuring credential-bearing configs are
not broadened by the process umask.

---

646-653: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Prevent localized-file name collisions.

The target name uses only the source basename components, so distinct files such as /a/config.yaml and /b/config.yaml overwrite each other when referenced by the same manifest. Detect duplicate targets and fail, or include a stable source-path discriminator in the filename before writing.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cloudai/workloads/ai_dynamo/ai_dynamo.sh` around lines 646 - 653, Update
the manifest-processing loop to prevent collisions in the target path derived
from source.stem and node_name. Track generated targets and fail clearly when
distinct source files map to the same target, or add a stable discriminator
based on the source path; preserve the existing content replacement and write
behavior for unique targets.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/cloudai/workloads/ai_dynamo/ai_dynamo.sh`:
- Around line 646-656: Validate each config_var in the manifest loop before
using it to generate the export statement, requiring the shell-variable pattern
^[A-Za-z_][A-Za-z0-9_]*$ and rejecting invalid keys. Keep valid-key processing
unchanged, and ensure validation occurs before emitting any sourced shell code.
- Around line 648-653: Update the localized-config write flow around
target.write_text in the replacement loop to preserve the source file’s
restrictive permissions on the newly created target. Apply the source mode
explicitly after creating the target, or create it with equivalent restrictive
permissions, ensuring credential-bearing configs are not broadened by the
process umask.
- Around line 646-653: Update the manifest-processing loop to prevent collisions
in the target path derived from source.stem and node_name. Track generated
targets and fail clearly when distinct source files map to the same target, or
add a stable discriminator based on the source path; preserve the existing
content replacement and write behavior for unique targets.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: f578d842-7b3b-4d37-bf29-ae08ab3f626a

📥 Commits

Reviewing files that changed from the base of the PR and between 2012c6a and aa36151.

📒 Files selected for processing (1)

• src/cloudai/workloads/ai_dynamo/ai_dynamo.sh