GB200 LLM performance scripts tuning by guyueh1 · Pull Request #12791 · NVIDIA-NeMo/NeMo

guyueh1 · 2025-03-26T18:34:46Z

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Tune the performance benchmarks of LLMs for GB200.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

scripts/performance/argument_parser.py

erhoo82 · 2025-04-03T21:24:24Z

scripts/performance/argument_parser.py

+    parser.add_argument(
+        "-fsdp",
+        "--use_mcore_fsdp",
+        help="Enable Mcore FSDP. Disabled by default",


Fix to Enable Megatron Core (Mcore) FSDP. Disabled by default

scripts/performance/utils.py

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

ko3n1g · 2025-04-06T21:18:10Z

stopping pipeline since merge-conflict in the context of limited time before code-freeze.

Signed-off-by: guyueh1 <140554423+guyueh1@users.noreply.github.com>

guyueh1 · 2025-04-07T18:33:47Z

@erhoo82 all checks passed, we can merge it if you approve

erhoo82

LGTM

* Add gb200 config, support FSDP and recompute in perf script Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Support offloading; change recipes Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix tp overlap config for gb200 Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix csv column name Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix ckpt layers in gb200 config Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Make Nemotron FP8 B200 use ring-exchange for GEMM+RS Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Tune configs for 405b bf16 and gpt Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Use pp config for nemotron 340b Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix tp overlap error by aggregate=False Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Make nemotron-340b bf16 use fsdp Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Handle segment for GB200 Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Disable CG when using FSDP Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Rm dp_size from B200 csv Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Address comments Signed-off-by: Guyue Huang <guyueh@nvidia.com> --------- Signed-off-by: Guyue Huang <guyueh@nvidia.com> Signed-off-by: guyueh1 <140554423+guyueh1@users.noreply.github.com>

* Add gb200 config, support FSDP and recompute in perf script Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Support offloading; change recipes Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix tp overlap config for gb200 Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix csv column name Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix ckpt layers in gb200 config Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Make Nemotron FP8 B200 use ring-exchange for GEMM+RS Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Tune configs for 405b bf16 and gpt Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Use pp config for nemotron 340b Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix tp overlap error by aggregate=False Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Make nemotron-340b bf16 use fsdp Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Handle segment for GB200 Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Disable CG when using FSDP Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Rm dp_size from B200 csv Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Address comments Signed-off-by: Guyue Huang <guyueh@nvidia.com> --------- Signed-off-by: Guyue Huang <guyueh@nvidia.com> Signed-off-by: guyueh1 <140554423+guyueh1@users.noreply.github.com> Signed-off-by: Jonathan Mitchell <jomitchell@nvidia.com>

* Add gb200 config, support FSDP and recompute in perf script * Support offloading; change recipes * Fix tp overlap config for gb200 * Fix csv column name * Fix ckpt layers in gb200 config * Make Nemotron FP8 B200 use ring-exchange for GEMM+RS * Tune configs for 405b bf16 and gpt * Use pp config for nemotron 340b * Fix * Fix tp overlap error by aggregate=False * Make nemotron-340b bf16 use fsdp * Handle segment for GB200 * Disable CG when using FSDP * Rm dp_size from B200 csv * Address comments --------- Signed-off-by: Guyue Huang <guyueh@nvidia.com> Signed-off-by: guyueh1 <140554423+guyueh1@users.noreply.github.com> Co-authored-by: guyueh1 <140554423+guyueh1@users.noreply.github.com>

* Add gb200 config, support FSDP and recompute in perf script Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Support offloading; change recipes Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix tp overlap config for gb200 Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix csv column name Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix ckpt layers in gb200 config Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Make Nemotron FP8 B200 use ring-exchange for GEMM+RS Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Tune configs for 405b bf16 and gpt Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Use pp config for nemotron 340b Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Fix tp overlap error by aggregate=False Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Make nemotron-340b bf16 use fsdp Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Handle segment for GB200 Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Disable CG when using FSDP Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Rm dp_size from B200 csv Signed-off-by: Guyue Huang <guyueh@nvidia.com> * Address comments Signed-off-by: Guyue Huang <guyueh@nvidia.com> --------- Signed-off-by: Guyue Huang <guyueh@nvidia.com> Signed-off-by: guyueh1 <140554423+guyueh1@users.noreply.github.com> Signed-off-by: Ritvik Bale <rbale@nvidia.com>

guyueh1 added 11 commits March 26, 2025 11:33

Add gb200 config, support FSDP and recompute in perf script

071a3d9

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Support offloading; change recipes

a27d4a8

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Fix tp overlap config for gb200

51a540f

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Fix csv column name

5ef4240

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Fix ckpt layers in gb200 config

a4d44b2

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Make Nemotron FP8 B200 use ring-exchange for GEMM+RS

ddaeafe

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Tune configs for 405b bf16 and gpt

e460faa

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Use pp config for nemotron 340b

c75ff5c

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Fix

b5b431b

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Fix tp overlap error by aggregate=False

99fda16

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Make nemotron-340b bf16 use fsdp

95728d3

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

guyueh1 marked this pull request as ready for review April 1, 2025 05:31

guyueh1 added 4 commits April 1, 2025 16:38

Handle segment for GB200

e68ffa1

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Disable CG when using FSDP

5a74ecc

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Merge branch 'main' into gb200_perf_scripts

c3c74d2

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

Rm dp_size from B200 csv

07602ab

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

erhoo82 reviewed Apr 3, 2025

View reviewed changes

Address comments

65cdb8a

Signed-off-by: Guyue Huang <guyueh@nvidia.com>

ko3n1g added the Run CICD label Apr 4, 2025

chtruong814 added Run CICD and removed Run CICD labels Apr 5, 2025

Merge branch 'main' into gb200_perf_scripts

07a643a

Signed-off-by: guyueh1 <140554423+guyueh1@users.noreply.github.com>

ko3n1g added Run CICD and removed Run CICD labels Apr 6, 2025

erhoo82 approved these changes Apr 7, 2025

View reviewed changes

erhoo82 merged commit 9657892 into NVIDIA-NeMo:main Apr 7, 2025
243 of 244 checks passed

erhoo82 added the r2.3.0 Pick this label for auto-cherrypicking into v2.3.0 label Apr 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GB200 LLM performance scripts tuning#12791

GB200 LLM performance scripts tuning#12791
erhoo82 merged 17 commits intoNVIDIA-NeMo:mainfrom
guyueh1:gb200_perf_scripts

guyueh1 commented Mar 26, 2025

Uh oh!

Uh oh!

Uh oh!

erhoo82 Apr 3, 2025

Uh oh!

Uh oh!

Uh oh!

ko3n1g commented Apr 6, 2025

Uh oh!

guyueh1 commented Apr 7, 2025

Uh oh!

erhoo82 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

Conversation

guyueh1 commented Mar 26, 2025

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Uh oh!

Uh oh!

erhoo82 Apr 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ko3n1g commented Apr 6, 2025

Uh oh!

guyueh1 commented Apr 7, 2025

Uh oh!

erhoo82 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments