Pytorch Lightning Integration #569

SeanNaren · 2020-08-18T17:50:52Z

Closes #568

SeanNaren · 2020-08-18T17:51:38Z

Next steps are to integrate saving/loading of the model from training to eval.

README.md

stale · 2020-11-23T16:04:06Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

…ity, integrated checkpoint support, fixed validation support

Added trains viz logging Precision

…x for adam Trains support, removed autocast since this is handled via lightning

… an4 config

…orrect lightning version

… the epochs used for previous release

… release

…ew ModelCheckpoint class for model saving

…ath, refactor path name and test

…bject for manifest to modify root path

piraka9011 · 2020-12-16T16:53:17Z

deepspeech_pytorch/model.py

        )
        self.fc = nn.Sequential(
            SequenceWise(fully_connected),
        )
        self.inference_softmax = InferenceBatchSoftmax()
+        self.criterion = CTCLoss(reduction='sum', zero_infinity=True)


I'm currently using reduction='mean' and zero_infinity=False (the defaults)
What's the reason for these parameters?

Reduction sum means do not average the loss/gradients. This is reflecting the behaviour of warp-ctc. Zero infinity means if we get an inf gradient, we zero instead of returning infinite. Another behaviour copying warp-ctc!

deepspeech_pytorch/data/utils.py

Co-authored-by: Anas Abou Allaban <aabouallaban@protonmail.com>

SeanNaren · 2020-12-27T13:07:35Z

I've run into some strange convergence issues with librispeech on this branch. Going to investigate and have a look (might just be the batch size needs to be small).

piraka9011 · 2020-12-28T20:55:11Z

FYI, I had to install omegaconf from source (specifically from this commit) in order to get DeepSpeech.load_from_checkpoint() to work when performing inference.

Otherwise, I would get an error when the model tried to parse the spect_cfg.sample_rate (or any spect_cfg) from the checkpoint.

TypeError: issubclass() arg 1 must be a class

where the type of sample_rate for example is inferred as Any.

My assumption is that this is the relevant issue.

Maybe this is also an upstream issue w/ PTL?

deepspeech_pytorch/inference.py

Specify blank index explicitly

…-model Add blank index to ctc loss explicitly

Fix Syntax Warning

Fix CI

* Use torchaudio * Add torchaudio to reqs

SeanNaren · 2021-01-29T20:16:22Z

Builds will fail due to lack of pre-trained models, but that should be solved once i make the release! I think we're there

ritwikmishra · 2021-03-09T12:33:51Z

TypeError: issubclass() arg 1 must be a class

I solved this by reinstalling omegaconf to this version omegaconf==2.1.0.dev9.

SeanNaren force-pushed the feature/lightning branch from fe6583e to b6f6572 Compare August 29, 2020 17:41

This was referenced Aug 31, 2020

Deprecate storing RNN Type, swap to using Enum #548

Closed

Code breaking changes upon installing pytorch version 1.6.0 #566

Closed

igormq reviewed Sep 1, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

SeanNaren mentioned this pull request Sep 9, 2020

New Pytorch Lightning Pre-trained Models Release #576

Closed

3 tasks

stale bot added the stale label Oct 3, 2020

Repository owner deleted a comment from stale bot Oct 3, 2020

stale bot removed the stale label Oct 7, 2020

stale bot added stale and removed stale labels Nov 23, 2020

Sean Narenthiran and others added 18 commits December 12, 2020 14:49

Added minimal code to integrate Pytorch Lightning into training.py

165f464

Added autocast support, removed intra epoch checkpointing for simplic…

5a6bfe4

…ity, integrated checkpoint support, fixed validation support

Fixed multi-gpu support

79eb233

Fixed smoke test, pretrained tests will be broken till new model release

5524938

Added trains viz logging Precision

Updated README, fixed server class, updated k8s config file, added fi…

7b2b460

…x for adam Trains support, removed autocast since this is handled via lightning

Swapped to using tqdm write for readability when checkpointing, added…

90d6d05

… an4 config

Added base script for each dataset, updated default params

52fbcd3

Swapped to using native CTC, updated common voice script, removed inc…

bef4c06

…orrect lightning version

Updated cv params and output manifest location, set default epochs to…

5c86984

… the epochs used for previous release

Disable trains logger for now, simplified checkpointing logic for new…

1b01171

… release

Added new metrics class, removed save_output/verbose for now, using n…

2e8cfab

…ew ModelCheckpoint class for model saving

multiprocess duration collection for speed, allow loading from file p…

00989e6

…ath, refactor path name and test

Swap to latest release candidate, fixed flag reference

f65f3ac

Format smoke test, update path to best save k model

2211540

Update to latest RC

eb64d3e

Removed trains logging, rely on PL tensorboard. swap to saving json o…

c0a0cc2

…bject for manifest to modify root path

Ensure abs path for manifest root path

c95f889

Use absolute paths for manifest

0d13d9c

SeanNaren force-pushed the feature/lightning branch from 0ba0fb7 to 51fc8fb Compare December 12, 2020 14:50

SeanNaren added 4 commits December 12, 2020 14:53

Enable checkpoint callback

d56fcef

Enable checkpoint callback, add verbosity

01bcce0

Add sharded as a dependency for better memory use

ce88f28

Set num workers, add spec augment

d7dc1ed

piraka9011 reviewed Dec 16, 2020

View reviewed changes

piraka9011 reviewed Dec 24, 2020

View reviewed changes

deepspeech_pytorch/data/utils.py Outdated Show resolved Hide resolved

Update deepspeech_pytorch/data/utils.py

fc5632d

Co-authored-by: Anas Abou Allaban <aabouallaban@protonmail.com>

piraka9011 reviewed Dec 28, 2020

View reviewed changes

deepspeech_pytorch/inference.py Outdated Show resolved Hide resolved

piraka9011 and others added 12 commits December 31, 2020 10:24

Specify blank index explicitly

add905e

Add blank index to ctc loss

aaf06de

Fix CI

6942d75

Fix Syntax Warning

ae41e9f

Merge pull request #601 from piraka9011/feature/lightning-blank-index

45e23c7

Specify blank index explicitly

Merge pull request #603 from piraka9011/feature/lightning-blank-index…

2d6fce0

…-model Add blank index to ctc loss explicitly

Merge pull request #605 from piraka9011/feature/lightning-warning

844e6dc

Fix Syntax Warning

Merge pull request #604 from piraka9011/feature/lightning-ci

44dce2b

Fix CI

Merge branch 'master' into feature/lightning

f85c802

Fix install requirements

96d9143

Use torchaudio (#607)

175bfdc

* Use torchaudio * Add torchaudio to reqs

Fixes for testing, update AN4 config, update dockerfile base image

0eb90f7

SeanNaren added 3 commits January 29, 2021 21:21

Add noninteractive to remove stalling

f5a2a27

revert

bd3957e

Update API

810a82e

SeanNaren merged commit d9790d9 into master Jan 30, 2021

SeanNaren deleted the feature/lightning branch January 30, 2021 12:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytorch Lightning Integration #569

Pytorch Lightning Integration #569

SeanNaren commented Aug 18, 2020 •

edited

SeanNaren commented Aug 18, 2020

stale bot commented Nov 23, 2020

piraka9011 Dec 16, 2020

SeanNaren Dec 24, 2020

SeanNaren commented Dec 27, 2020

piraka9011 commented Dec 28, 2020 •

edited

SeanNaren commented Jan 29, 2021

ritwikmishra commented Mar 9, 2021 •

edited

Pytorch Lightning Integration #569

Pytorch Lightning Integration #569

Conversation

SeanNaren commented Aug 18, 2020 • edited

SeanNaren commented Aug 18, 2020

stale bot commented Nov 23, 2020

piraka9011 Dec 16, 2020

Choose a reason for hiding this comment

SeanNaren Dec 24, 2020

Choose a reason for hiding this comment

SeanNaren commented Dec 27, 2020

piraka9011 commented Dec 28, 2020 • edited

SeanNaren commented Jan 29, 2021

ritwikmishra commented Mar 9, 2021 • edited

SeanNaren commented Aug 18, 2020 •

edited

piraka9011 commented Dec 28, 2020 •

edited

ritwikmishra commented Mar 9, 2021 •

edited