Skip to content

docs(examples): add Slurm launch examples - #54

Merged
ahmadki merged 2 commits into
mainfrom
rhewett/pr6-slurm-launch-examples
Sep 4, 2026
Merged

docs(examples): add Slurm launch examples#54
ahmadki merged 2 commits into
mainfrom
rhewett/pr6-slurm-launch-examples

Conversation

@rhewett-nv

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds self-contained Slurm launch examples that use the Lens CLI to set resource
attributes, emit launch spans, and write local collector output for plain Slurm
and NVRx workflows.

Issues

N/A

Usage

cd examples/slurm-launch

# Edit account, partition, qos, TEL, CONTAINER_IMAGE, and CONTAINER_MOUNTS first.
sbatch otel_minimal.sbatch
sbatch nvrx_bulk.sbatch
sbatch nvrx_bulk_spare.sbatch
sbatch nvrx_array.sbatch
sbatch nvrx_array_spare.sbatch

Before your PR is "Ready for review"

Pre checks:

  • I read the Contributor guidelines.
  • All commits are signed off (git commit -s) per the DCO.
  • I added or updated tests under tests/ for any behavior change.
  • I ran pytest and pre-commit run --all-files locally; both pass.
  • I updated docs/ and README.md where behavior, configuration, or public API changed.
  • If this changes a lens public API symbol, I updated the cross-repo _fallbacks.py files in Megatron-LM / RL / Gym.

Additional information

Validation on pr6-slurm-launch-examples:

  • ../.venv/bin/pre-commit run --all-files: passed
  • PYTHONPATH=src ../.venv/bin/python -m pytest -q: 265 passed, 7 warnings
  • ../.venv/bin/python -m compileall -q src tests examples/slurm-launch/mock_workload.py: passed
  • ../.venv/bin/ruff check examples/slurm-launch/mock_workload.py: passed
  • bash -n on all five .sbatch examples: passed
  • git diff --check pr5-slurm-resource-attrs-cli..pr6-slurm-launch-examples: passed
  • npm --prefix docs/fern run check: passed with 0 errors; one warning because the unauthenticated redirects check is skipped without fern login / FERN_TOKEN

Notes:

  • PR6 is one signed commit: 03f9b54 docs(examples): add Slurm launch examples.
  • pr6-slurm-launch-examples and schema-v01-resource-attrs point at the same
    commit.
  • The PR adds only examples/slurm-launch/ files; it does not change Lens
    runtime code, tests, packaging, docs-site pages, or fallback surfaces.
  • Example scripts use nemo-lens set-slurm-resource-attrs --stage sbatch in the
    sbatch context and --stage task inside srun.
  • SLUID lookup is done once in the sbatch/script context with sacct; the task
    context inherits LENS_SLURM_SLUID and LENS_SLURM_ARRAY_SLUID.
  • The examples use placeholder scheduler/path/image/mount values and the local
    task collector endpoint http://localhost:4317.
  • The NVRx examples assume ft_launcher is already installed and on PATH in
    the container.
  • The array-spare example may cancel a cold spare before it emits telemetry; that
    is expected for this test shape.

@rhewett-nv
rhewett-nv requested a review from ahmadki as a code owner August 30, 2026 23:57
@copy-pr-bot

copy-pr-bot Bot commented Aug 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch from d8642d3 to b42e6b9 Compare September 3, 2026 18:31
@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch 2 times, most recently from 67a1ac9 to 7f85b18 Compare September 3, 2026 19:05
@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch from 7f85b18 to 1313c39 Compare September 3, 2026 19:28
@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch from 1313c39 to 1c7679f Compare September 3, 2026 20:05
ahmadki
ahmadki previously approved these changes Sep 3, 2026
@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch from 1c7679f to ec56816 Compare September 4, 2026 00:30
@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch from ec56816 to 85cc56d Compare September 4, 2026 03:58
@ahmadki

ahmadki commented Sep 4, 2026

Copy link
Copy Markdown
Member

/ok to test 85cc56d

@ahmadki
ahmadki force-pushed the rhewett/pr6-slurm-launch-examples branch from 85cc56d to a998266 Compare September 4, 2026 14:17
Signed-off-by: Russell Hewett <rhewett@nvidia.com>
Signed-off-by: Russell Hewett <rhewett@nvidia.com>
@rhewett-nv
rhewett-nv force-pushed the rhewett/pr6-slurm-launch-examples branch from a998266 to 91225cb Compare September 4, 2026 14:44
Base automatically changed from rhewett/pr5-slurm-resource-attrs-cli to main September 4, 2026 14:45
@ahmadki
ahmadki merged commit 6d15644 into main Sep 4, 2026
8 checks passed
@ahmadki
ahmadki deleted the rhewett/pr6-slurm-launch-examples branch September 4, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants