Skip to content

Akoumparouli/fix sd train 2#8883

Merged
akoumpa merged 13 commits intomainfrom
akoumparouli/fix_sd_train_2
Apr 17, 2024
Merged

Akoumparouli/fix sd train 2#8883
akoumpa merged 13 commits intomainfrom
akoumparouli/fix_sd_train_2

Conversation

@akoumpa
Copy link
Collaborator

@akoumpa akoumpa commented Apr 11, 2024

What does this PR do ?

pass fp16 precision to UNetModel constructor if user specifies fp16 precision via CLI.

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

akoumpa added 2 commits April 11, 2024 05:22
@akoumpa
Use fp16 in unet if cfg.precision is 16bit
c794ca0 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
remove temp fix
9197533 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
guard unet_precision value setting
cc88a16 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 11, 2024

jenkins

@akoumpa
formatting
deb52ed 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa akoumpa force-pushed the akoumparouli/fix_sd_train_2 branch from 94ef985 to deb52ed Compare April 11, 2024 05:32
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 11, 2024

jenkins

akoumpa and others added 2 commits April 11, 2024 05:46
@akoumpa
throw a warning if unet_config.unet_precision is not set but trainer.…
6705b5a 
…precision is set

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
fefefae 
for more information, see https://pre-commit.ci
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 11, 2024

jenkins

@akoumpa akoumpa requested a review from dimapihtar April 11, 2024 05:51
@Dido0o0
Copy link

Dido0o0 commented Apr 11, 2024

Set unet_precision when user not set it.

@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 11, 2024

jenkins

@akoumpa
use model.optim.name=megatron_fused_adam in sd_train
4f04f69 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@github-actions github-actions bot added the CI label Apr 11, 2024
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 11, 2024

jenkins

@ericharper ericharper requested review from ShriyaRishab and removed request for dimapihtar April 11, 2024 21:23
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 12, 2024

jenkins

@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 12, 2024

jenkins

ShriyaRishab
ShriyaRishab previously approved these changes Apr 12, 2024
View reviewed changes
@ShriyaRishab
Copy link
Collaborator

ShriyaRishab commented Apr 12, 2024

jenkins

1 similar comment
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 15, 2024

jenkins

@akoumpa
use bf16 in sd_train
81db925 
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 15, 2024

jenkins

@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 16, 2024

jenkins

@akoumpa
Copy link
Collaborator Author

akoumpa commented Apr 16, 2024

jenkins

@akoumpa akoumpa marked this pull request as ready for review April 17, 2024 19:53
@akoumpa akoumpa requested a review from ShriyaRishab April 17, 2024 19:54
@akoumpa akoumpa merged commit 7c562b4 into main Apr 17, 2024
@akoumpa akoumpa deleted the akoumparouli/fix_sd_train_2 branch April 17, 2024 19:59
xingyaoww pushed a commit to xingyaoww/NeMo that referenced this pull request Apr 23, 2024
@akoumpa @pre-commit-ci @xingyaoww
Akoumparouli/fix sd train 2 (NVIDIA-NeMo#8883)
0238229 
* Use fp16 in unet if cfg.precision is 16bit

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove temp fix

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* guard unet_precision value setting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* formatting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* throw a warning if unet_config.unet_precision is not set but trainer.precision is set

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use model.optim.name=megatron_fused_adam in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* use bf16 in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
alxzhang-amazon pushed a commit to alxzhang-amazon/NeMo that referenced this pull request Apr 26, 2024
@akoumpa @pre-commit-ci @alxzhang-amazon
Akoumparouli/fix sd train 2 (NVIDIA-NeMo#8883)
bab0833 
* Use fp16 in unet if cfg.precision is 16bit

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove temp fix

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* guard unet_precision value setting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* formatting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* throw a warning if unet_config.unet_precision is not set but trainer.precision is set

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use model.optim.name=megatron_fused_adam in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* use bf16 in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
suiyoubi pushed a commit that referenced this pull request May 2, 2024
@akoumpa @pre-commit-ci @suiyoubi
Akoumparouli/fix sd train 2 (#8883)
572a8c5 
* Use fp16 in unet if cfg.precision is 16bit

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove temp fix

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* guard unet_precision value setting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* formatting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* throw a warning if unet_config.unet_precision is not set but trainer.precision is set

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use model.optim.name=megatron_fused_adam in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* use bf16 in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Ao Tang <aot@nvidia.com>
rohitrango pushed a commit to rohitrango/NeMo that referenced this pull request Jun 25, 2024
@akoumpa @pre-commit-ci
Akoumparouli/fix sd train 2 (NVIDIA-NeMo#8883)
75b0645 
* Use fp16 in unet if cfg.precision is 16bit

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* remove temp fix

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* guard unet_precision value setting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* formatting

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* throw a warning if unet_config.unet_precision is not set but trainer.precision is set

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use model.optim.name=megatron_fused_adam in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* use bf16 in sd_train

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ShriyaRishab ShriyaRishab ShriyaRishab approved these changes

Assignees

No one assigned

Labels

CI Multi Modal

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@akoumpa @Dido0o0 @ShriyaRishab