feat: Dockerfile & dependencies by pablo-garay · Pull Request #35 · NVIDIA-NeMo/DFM

pablo-garay · 2025-11-11T08:39:41Z

feat: Dockerfile & dependencies

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

… dependency group) Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

…ath, no fail on coverage errors) Signed-off-by: Pablo Garay <pagaray@nvidia.com>

…un coverage' Signed-off-by: Pablo Garay <pagaray@nvidia.com>

pablo-garay · 2025-11-12T07:43:50Z

/ok to test 862f486

…rage, fix paths to /opt/DFM Signed-off-by: Pablo Garay <pagaray@nvidia.com>

pablo-garay · 2025-11-12T08:14:17Z

/ok to test fe4181f

- Add 'submodules: recursive' to test-template checkout (matches Megatron-Bridge) - Remove error-swallowing (|| echo) pattern from test script - Add proper timeout handling and exit code capture - Fixes: nemo-automodel missing pyproject.toml error Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

pablo-garay · 2025-11-12T10:12:37Z

/ok to test 3a56c39

abhinavg4

Looks good. Just a minor comment. Let's also get a review from @ko3n1g too before merging

abhinavg4 · 2025-11-12T14:38:29Z

-]
+nemo-automodel = { path = "3rdparty/Automodel" }
+megatron-bridge = { path = "3rdparty/Megatron-Bridge" }
+transformer-engine = { git = "https://github.com/NVIDIA/TransformerEngine.git", rev = "release_v2.9" }


Why do we have TE here? TE would be part of Mbrige right ?

both options exist, if you want bump TE more aggressively you can override it otherwise you can inherit from MBridge. Maybe for simplicity you want to inherit by default

* dockerfile install dependencies Signed-off-by: Pablo Garay <pagaray@nvidia.com> * update uv lock Signed-off-by: Pablo Garay <pagaray@nvidia.com> * docker adjustment for current dfm status Signed-off-by: Pablo Garay <pagaray@nvidia.com> * version being loaded dynamically Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Add megatron-bridge dependency (megatron-core as transitive) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * allow certain packages access existing env deps at build time Signed-off-by: Pablo Garay <pagaray@nvidia.com> * necessary build deps for pkgs when build isolation disabled (as build dependency group) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * use cuda version installed in pytorch container Signed-off-by: Pablo Garay <pagaray@nvidia.com> * build dep group first, then all other deps Signed-off-by: Pablo Garay <pagaray@nvidia.com> * generate lockfile in CI Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * generate lockfile update Signed-off-by: Pablo Garay <pagaray@nvidia.com> * generate lockfile update Signed-off-by: Pablo Garay <pagaray@nvidia.com> * generate lockfile update Signed-off-by: Pablo Garay <pagaray@nvidia.com> * generate lockfile update Signed-off-by: Pablo Garay <pagaray@nvidia.com> * need add wheel Signed-off-by: Pablo Garay <pagaray@nvidia.com> * need add wheel Signed-off-by: Pablo Garay <pagaray@nvidia.com> * uv lock Signed-off-by: Pablo Garay <pagaray@nvidia.com> * uv lock Signed-off-by: Pablo Garay <pagaray@nvidia.com> * uv lock 2 Signed-off-by: Pablo Garay <pagaray@nvidia.com> * try 2 Signed-off-by: Pablo Garay <pagaray@nvidia.com> * try 3 Signed-off-by: Pablo Garay <pagaray@nvidia.com> * generate lockfile another attempt Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Add generated uv.lock file Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix lockfile generation to match runtime configuration Signed-off-by: Pablo Garay <pagaray@nvidia.com> * uv.lock update Signed-off-by: Pablo Garay <pagaray@nvidia.com> * lintfix Signed-off-by: Pablo Garay <pagaray@nvidia.com> * run generate uv.lock workflow on manual trigger only Signed-off-by: Pablo Garay <pagaray@nvidia.com> * move to /opt/DFM Signed-off-by: Pablo Garay <pagaray@nvidia.com> * automodel & mbridge as submodules Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Initialize submodules in lockfile generation workflow Signed-off-by: Pablo Garay <pagaray@nvidia.com> * build-docs adapt for 2-step installation Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Pre-install build dependencies to avoid metadata extraction errors Signed-off-by: Pablo Garay <pagaray@nvidia.com> * adjust core dep due to Megatron-Energon requiring cuda resolution Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Shared template uses uv sync internally. energon as core dep Signed-off-by: Pablo Garay <pagaray@nvidia.com> * build-docs adapt Signed-off-by: Pablo Garay <pagaray@nvidia.com> * adjust: same git source for nvidia-resiliency-ext as Mbridge Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix: use poetry-core instead of poetry for nvidia-resiliency-ext build Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Override non-existent extras from megatron-bridge dependencies Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Add transformer-engine git source matching megatron-bridge Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Match megatron-bridge override-dependencies exactly (keep [pytorch] extra) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Remove extra-build-dependencies, add nvidia-resiliency-ext git source like megatron-bridge Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix lockfile generation: pre-install poetry-core and fail on error Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Install poetry (not poetry-core) for nvidia-resiliency-ext build Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix venv activation: source activate script instead of env var Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Use PyPI nvidia-resiliency-ext (pre-built wheels, no poetry needed) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Override nvidia-resiliency-ext source to PyPI (overrides megatron-bridge git source) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix syntax: use 'index' not 'registry' for nvidia-resiliency-ext Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Copy Megatron-Bridge env var approach for venv (UV_PROJECT_ENVIRONMENT + VIRTUAL_ENV) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Update uv.lock [skip ci] Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix trailing whitespace in generate-lockfile.yml Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Match Megatron-Bridge Dockerfile exactly: add cache mounts, remove --no-install-project, add cache prune Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Add nvidia-mathdx to build dependencies (required by transformer-engine, matching megatron-bridge) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * update uv.lock Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Add --no-install-project back for CI test workflow compatibility Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Mount CI test code at /opt/DFM to match Dockerfile WORKDIR (like Megatron-Bridge) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Activate venv before installing project in CI tests Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix coverage handling to match Megatron-Bridge (use exit code, full path, no fail on coverage errors) Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Align test scripts with Megatron-Bridge: use /opt/DFM paths and 'uv run coverage' Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix CI to match Megatron-Bridge: checkout submodules, use uv run coverage, fix paths to /opt/DFM Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix test execution: add submodules checkout + proper exit code handling - Add 'submodules: recursive' to test-template checkout (matches Megatron-Bridge) - Remove error-swallowing (|| echo) pattern from test script - Add proper timeout handling and exit code capture - Fixes: nemo-automodel missing pyproject.toml error Signed-off-by: Pablo Garay <pagaray@nvidia.com> * Fix trailing whitespace in test-template action Signed-off-by: Pablo Garay <pagaray@nvidia.com> --------- Signed-off-by: Pablo Garay <pagaray@nvidia.com>

pablo-garay added 30 commits November 6, 2025 00:13

dockerfile install dependencies

537b82e

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

update uv lock

1577f2c

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

docker adjustment for current dfm status

19197ef

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

version being loaded dynamically

6a6c427

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Add megatron-bridge dependency (megatron-core as transitive)

0547378

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

allow certain packages access existing env deps at build time

29f204f

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

necessary build deps for pkgs when build isolation disabled (as build…

f8e3440

… dependency group) Signed-off-by: Pablo Garay <pagaray@nvidia.com>

use cuda version installed in pytorch container

85babff

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

build dep group first, then all other deps

b6c7a9e

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

generate lockfile in CI

7edaaab

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

lintfix

b3d2fb4

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

lintfix

5bf51b3

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

generate lockfile update

de72a5b

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

generate lockfile update

b75984f

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

generate lockfile update

069c536

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

generate lockfile update

89f2866

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

need add wheel

1fcb8e0

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

need add wheel

162377d

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

uv lock

6af1db1

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

uv lock

2e071ab

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

uv lock 2

98a00d9

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

try 2

b9e5653

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

try 3

741bcd2

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

generate lockfile another attempt

7d245da

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Add generated uv.lock file

6c08f5e

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Fix lockfile generation to match runtime configuration

90f32c8

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

uv.lock update

8e0c998

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Merge branch 'main' into pablo-garay/dockerfile_install_dependencies2

e8c7efb

lintfix

793825a

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

run generate uv.lock workflow on manual trigger only

8b8adff

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

copy-pr-bot Bot had a problem deploying to nemo-ci November 11, 2025 22:53 Failure

pablo-garay added 3 commits November 11, 2025 23:30

Activate venv before installing project in CI tests

d795db9

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Fix coverage handling to match Megatron-Bridge (use exit code, full p…

d29065d

…ath, no fail on coverage errors) Signed-off-by: Pablo Garay <pagaray@nvidia.com>

Align test scripts with Megatron-Bridge: use /opt/DFM paths and 'uv r…

862f486

…un coverage' Signed-off-by: Pablo Garay <pagaray@nvidia.com>

copy-pr-bot Bot temporarily deployed to test November 12, 2025 07:43 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 12, 2025 07:44 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci November 12, 2025 07:49 Failure

Fix CI to match Megatron-Bridge: checkout submodules, use uv run cove…

fe4181f

…rage, fix paths to /opt/DFM Signed-off-by: Pablo Garay <pagaray@nvidia.com>

copy-pr-bot Bot temporarily deployed to test November 12, 2025 08:14 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 12, 2025 08:14 Inactive

copy-pr-bot Bot had a problem deploying to nemo-ci November 12, 2025 08:20 Failure

copy-pr-bot Bot had a problem deploying to nemo-ci November 12, 2025 08:20 Error

copy-pr-bot Bot had a problem deploying to nemo-ci November 12, 2025 09:20 Failure

pablo-garay added 2 commits November 12, 2025 02:09

Fix trailing whitespace in test-template action

3a56c39

Signed-off-by: Pablo Garay <pagaray@nvidia.com>

copy-pr-bot Bot temporarily deployed to test November 12, 2025 10:12 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 12, 2025 10:12 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 12, 2025 10:19 Inactive

copy-pr-bot Bot temporarily deployed to nemo-ci November 12, 2025 10:35 Inactive

abhinavg4 approved these changes Nov 12, 2025

View reviewed changes

ko3n1g approved these changes Nov 12, 2025

View reviewed changes

pablo-garay merged commit 3cb35cc into main Nov 12, 2025
15 checks passed

chtruong814 deleted the pablo-garay/dockerfile_install_dependencies4 branch January 29, 2026 20:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Dockerfile & dependencies#35

feat: Dockerfile & dependencies#35
pablo-garay merged 67 commits into
mainfrom
pablo-garay/dockerfile_install_dependencies4

pablo-garay commented Nov 11, 2025

Uh oh!

pablo-garay commented Nov 12, 2025

Uh oh!

pablo-garay commented Nov 12, 2025

Uh oh!

pablo-garay commented Nov 12, 2025

Uh oh!

abhinavg4 left a comment

Uh oh!

abhinavg4 Nov 12, 2025

Uh oh!

ko3n1g Nov 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

pablo-garay commented Nov 11, 2025

Uh oh!

pablo-garay commented Nov 12, 2025

Uh oh!

pablo-garay commented Nov 12, 2025

Uh oh!

pablo-garay commented Nov 12, 2025

Uh oh!

abhinavg4 left a comment

Choose a reason for hiding this comment

Uh oh!

abhinavg4 Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

ko3n1g Nov 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants