Support Canary parallel inference #9517
Conversation
Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>
Signed-off-by: karpnv <karpnv@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -109,7 +112,9 @@ class ParallelTranscriptionConfig: | |||
| # att_context_size can be set for cache-aware streaming models with multiple look-aheads | |||
| att_context_size: Optional[list] = None | |||
|
|
|||
| trainer: TrainerConfig = TrainerConfig(devices=-1, accelerator="gpu", strategy="ddp") | |||
| trainer: TrainerConfig = TrainerConfig( | |||
| devices=-1, accelerator="gpu", strategy="ddp", use_distributed_sampler=False | |||
There was a problem hiding this comment.
be careful with distributed sampler setting here: non-lhotse datasets still likely require True. it might be better to just override this for EncDec model?
There was a problem hiding this comment.
ok, rm use_distributed_sampler=False
| @@ -72,7 +72,7 @@ def __getitem__(self, cuts: CutSet) -> tuple[torch.Tensor, torch.Tensor, torch.T | |||
| prompts = None | |||
| prompts_lens = None | |||
|
|
|||
| return audio, audio_lens, prompts_with_answers, prompts_with_answers_lens, prompts, prompts_lens | |||
| return audio, audio_lens, prompts_with_answers, prompts_with_answers_lens, prompts, prompts_lens, cuts | |||
There was a problem hiding this comment.
this is not ideal as returning cuts here will transfer the data held in-memory across dataloading worker subprocesses to the main training/inference loop process. we should return cuts.drop_recordings() instead.
|
|
|
trainer.global_rank is apriori set by PTL in slurm environment. It does not require model to be built or it's functions to be called c Is this a case where PTL cannot apriori detect global rank ? |
Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>
Signed-off-by: karpnv <karpnv@users.noreply.github.com>
Signed-off-by: Nikolay Karpov <nkarpov@nvidia.com>
…karpnv/canary_parallel
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
What does this PR do ?
Support Canary at transcribe_speech_parallel.py script
Collection: ASR
#python3 ./examples/asr/transcribe_speech_parallel.py model=./canary-1b.nemo predict_ds.manifest_filepath=./manifest.json output_path=/tmp trainer.devices=-1PR Type:
Who can review?
@pzelasko