Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WER.update doesn't work #8585

Closed
yuntang opened this issue Mar 5, 2024 · 2 comments
Closed

WER.update doesn't work #8585

yuntang opened this issue Mar 5, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@yuntang
Copy link

yuntang commented Mar 5, 2024

Describe the bug
As shown in
(https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/metrics/wer.py#L349-L350), the new scores and words will be assigned to the object and the pervious scores and words are dropped.

  1. we might rename this function to WER.set
    or
  2. we might update the code as

self.scores += torch.tensor(scores, device=self.scores.device, dtype=self.scores.dtype)
self.words += torch.tensor(words, device=self.words.device, dtype=self.words.dtype)

The current code could lead to WER report inconsistent during training and inference if we use fuse_loss_wer in the Transducer model training, i.e., model.joint.fuse_loss_wer=True and model.joint. fused_batch_size > 1. In this setting, only the last sub-mini-batch WER is accumulated during validation stage.

@yuntang yuntang added the bug Something isn't working label Mar 5, 2024
@titu1994
Copy link
Collaborator

titu1994 commented Mar 5, 2024

Thank you very much for raising this! We have fixed it in this PR - #8587
It occurred due to a large refactor and unification of metrics in ASR to make it simpler to extend in the long run.

The patch will be there in the next NeMo release, and we have added a release note in the 1.23 release page https://github.com/NVIDIA/NeMo/releases/tag/v1.23.0 so that users are aware and can utilize correct metrics during evaluation by using the speech to text eval script (or disabling fused batch explicitly)

@titu1994
Copy link
Collaborator

Fixed via #8587

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants