Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update optimizers.py #8920

Closed
wants to merge 2 commits into from
Closed

Update optimizers.py #8920

wants to merge 2 commits into from

Conversation

r0cketdyne
Copy link

I made several refinements and optimizations to improve the efficiency, readability, and maintainability of the code:

Code Structure Optimization:

Consolidated imports to improve readability and reduce clutter. Grouped related code blocks together for better coherence. Error Handling Improvement:

Simplified error handling by raising meaningful exceptions when encountering errors. This enhances code robustness and makes debugging easier. Efficient Data Handling:

Optimized data handling by directly returning default values if the input is None, eliminating unnecessary computations and enhancing code efficiency. Simplified Logic:

Simplified logic by removing redundant conditions and unnecessary checks, resulting in cleaner and more concise code. Optimized Imports:

Removed redundant imports and unused dependencies to streamline the codebase and reduce overhead. Enhanced Readability:

Improved code readability by using descriptive variable names and concise comments where necessary, aiding in code comprehension and maintenance. Consistent Naming Conventions:

Ensured consistent naming conventions throughout the codebase for better clarity and maintainability.

These changes collectively refine the codebase, making it more efficient, readable, and maintainable while preserving its functionality and purpose.

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

@github-actions github-actions bot added the core Changes to NeMo Core label Apr 14, 2024
titu1994
titu1994 requested changes Apr 14, 2024
View reviewed changes
Copy link
Collaborator

@titu1994 titu1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes no effective logical change, but introduced potential bugs by removing docs string, type annotations and even entire optimization cases for some reason.

nemo/core/optim/optimizers.py
@@ -1,49 +1,33 @@
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert. You cannot remove the license header

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you say more about the exact bugs this implementation will cause? Or are these just projections? Readibility is certainly non-trivial and a PR doesn't need a specific logical change to be non-trivial. This code was also optimized and this was noted at the PR.

nemo/core/optim/optimizers.py
@@ -54,20 +38,18 @@
except ModuleNotFoundError:
HAVE_APEX = False

HAVE_APEX_DISTRIBUTED_ADAM = False
# Check for APEX distributed Adam optimizer
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the bool?

nemo/core/optim/optimizers.py
if hasattr(optimizer_kwargs, 'keys'):
# Attempt class path resolution
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove docstring ?

nemo/core/optim/optimizers.py
optimizer_kwargs_config = OmegaConf.create(optimizer_kwargs)
optimizer_instance = hydra.utils.instantiate(optimizer_kwargs_config) # type: DictConfig
optimizer_instance = hydra.utils.instantiate(optimizer_kwargs_config)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the need to remove type annotations ?

nemo/core/optim/optimizers.py
if optimizer_kwargs['name'] == 'auto':
optimizer_params_name = "{}_params".format(optimizer_name)
optimizer_kwargs.pop('name')
else:
optimizer_params_name = optimizer_kwargs.pop('name')

# Override arguments provided in the config yaml file
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

nemo/core/optim/optimizers.py
else:
# If we are provided a partial class instantiation of a Config,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

nemo/core/optim/optimizers.py
adam_nondist_optims = (optim.Adam, optim.AdamW)
if HAVE_APEX:
adam_nondist_optims += (FusedAdam,)
if isinstance(optimizer, adam_nondist_optims):
for group in optimizer.param_groups:
for p in group['params']:
state = optimizer.state[p]
if len(state) == 0:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?? Why remove this?

@r0cketdyne
Copy link
Author

r0cketdyne commented Apr 14, 2024 via email

r0cketdyne and others added 2 commits April 21, 2024 04:14
@r0cketdyne
Update optimizers.py
4b8a3b5 
I made several refinements and optimizations to improve the efficiency, readability, and maintainability of the code:

Code Structure Optimization:

Consolidated imports to improve readability and reduce clutter.
Grouped related code blocks together for better coherence.
Error Handling Improvement:

Simplified error handling by raising meaningful exceptions when encountering errors. This enhances code robustness and makes debugging easier.
Efficient Data Handling:

Optimized data handling by directly returning default values if the input is None, eliminating unnecessary computations and enhancing code efficiency.
Simplified Logic:

Simplified logic by removing redundant conditions and unnecessary checks, resulting in cleaner and more concise code.
Optimized Imports:

Removed redundant imports and unused dependencies to streamline the codebase and reduce overhead.
Enhanced Readability:

Improved code readability by using descriptive variable names and concise comments where necessary, aiding in code comprehension and maintenance.
Consistent Naming Conventions:

Ensured consistent naming conventions throughout the codebase for better clarity and maintainability.

These changes collectively refine the codebase, making it more efficient, readable, and maintainable while preserving its functionality and purpose.

Signed-off-by: hengittää <matthew.stephenson@uah.edu>
@pre-commit-ci @r0cketdyne
[pre-commit.ci] auto fixes from pre-commit.com hooks
01482a1 
for more information, see https://pre-commit.ci
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 5, 2024

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions github-actions bot added the stale label May 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

This PR was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions bot closed this May 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@titu1994 titu1994 titu1994 requested changes

Assignees
No one assigned
Labels
core Changes to NeMo Core stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@r0cketdyne @titu1994