-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add EMA Docs, fix common collection documentation (#5757)
* Add EMA docs, fix docs due to incorrect import, fix doc format for common collection Signed-off-by: SeanNaren <snarenthiran@nvidia.com> * Address feedback Signed-off-by: SeanNaren <snarenthiran@nvidia.com> Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
- Loading branch information
Showing
6 changed files
with
89 additions
and
34 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
********* | ||
Callbacks | ||
********* | ||
|
||
Exponential Moving Average (EMA) | ||
================================ | ||
|
||
During training, EMA maintains a moving average of the trained parameters. | ||
EMA parameters can produce significantly better results and faster convergence for a variety of different domains and models. | ||
|
||
EMA is a simple calculation. EMA Weights are pre-initialized with the model weights at the start of training. | ||
|
||
Every training update, the EMA weights are updated based on the new model weights. | ||
|
||
.. math:: | ||
ema_w = ema_w * decay + model_w * (1-decay) | ||
Enabling EMA is straightforward. We can pass the additional argument to the experiment manager at runtime. | ||
|
||
.. code-block:: bash | ||
python examples/asr/asr_ctc/speech_to_text_ctc.py \ | ||
model.train_ds.manifest_filepath=/path/to/my/train/manifest.json \ | ||
model.validation_ds.manifest_filepath=/path/to/my/validation/manifest.json \ | ||
trainer.devices=2 \ | ||
trainer.accelerator='gpu' \ | ||
trainer.max_epochs=50 \ | ||
exp_manager.ema.enable=True # pass this additional argument to enable EMA | ||
To change the decay rate, pass the additional argument. | ||
|
||
.. code-block:: bash | ||
python examples/asr/asr_ctc/speech_to_text_ctc.py \ | ||
... | ||
exp_manager.ema.enable=True \ | ||
exp_manager.ema.decay=0.999 | ||
We also offer other helpful arguments. | ||
|
||
.. list-table:: | ||
:header-rows: 1 | ||
|
||
* - Argument | ||
- Description | ||
* - `exp_manager.ema.validate_original_weights=True` | ||
- Validate the original weights instead of EMA weights. | ||
* - `exp_manager.ema.every_n_steps=2` | ||
- Apply EMA every N steps instead of every step. | ||
* - `exp_manager.ema.cpu_offload=True` | ||
- Offload EMA weights to CPU. May introduce significant slow-downs. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
Losses | ||
------ | ||
.. autoclass:: nemo.collections.common.losses.AggregatorLoss | ||
:special-members: __init__ | ||
|
||
.. autoclass:: nemo.collections.common.losses.CrossEntropyLoss | ||
:special-members: __init__ | ||
|
||
.. autoclass:: nemo.collections.common.losses.MSELoss | ||
:special-members: __init__ | ||
|
||
.. autoclass:: nemo.collections.common.losses.SmoothedCrossEntropyLoss | ||
:special-members: __init__ | ||
|
||
.. autoclass:: nemo.collections.common.losses.SpanningLoss | ||
:special-members: __init__ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
Metrics | ||
------- | ||
|
||
.. autoclass:: nemo.collections.common.metrics.Perplexity | ||
:show-inheritance: | ||
:members: | ||
:undoc-members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
Tokenizers | ||
---------- | ||
.. autoclass:: nemo.collections.common.tokenizers.AutoTokenizer | ||
:special-members: __init__ | ||
.. autoclass:: nemo.collections.common.tokenizers.SentencePieceTokenizer | ||
:special-members: __init__ | ||
.. autoclass:: nemo.collections.common.tokenizers.TokenizerSpec | ||
:special-members: __init__ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters