Skip to content
This repository was archived by the owner on Jan 7, 2025. It is now read-only.

Conversation

@thisismatu
Copy link

@thisismatu thisismatu commented Nov 8, 2022

  • Update progress bar characters and width and clear it on finish
  • Group both ASR and NLU output better
  • Use our fork of Needleman-Wunsch alignment for ASR and NLU output. This is a rather simple tool that shows the where the ground truth and prediction differs.

Progress bar

Transcribing  50%  █████████▌░░░░░░░░░░  (2/4, 36 utt/min) [5s:3s]

NLU output

speechly evaluate nlu <app_id> test.txt

Line: 2
└─ Ground truth: *add_to_cart i want [two|2](amount) of those
└─ Prediction:   *add_to_cart i want *two*********** of those

Line: 4
└─ Ground truth: *set_delivery_date delivery [tomorrow|2022-01-*01](delivery_date)
└─ Prediction:   *set_delivery_date delivery [tomorrow|2022-11-10*](delivery_date)

Accuracy: 0.50 (2/4)

ASR output

speechly evaluate asr <app_id> test.jsonl 
                                                                  
Audio: podcast1.wav
└─ Ground truth: WELCOME TO ANOTHER EPISODE OF THE SPEECHLY PODCAST
└─ Prediction:   WELCOME TO ANOTHER EPISODE OF THE SPEECH** PODCAST

Audio: podcast3.wav
└─ Ground truth: THIS CONCEPT OF VOICE BEING AN EXPERT UI COULD YOU MAY***BE UNPACK THAT CONCEPT
└─ Prediction:   THIS CONCEPT OF VOICE BEING AN EXPERT UI COULD YOU MIGHT BE UNPACK THAT CONCEPT

Word Error Rate (WER): 0.04 (3/68)

Known issues

Long lines will wrap, but at least with these changes they are easier to distinguish. This PR does not try to make changes to that since it's a rabbit hole, believe me. I was in that hole for a brief moment of time before i managed to pull myself out of it...

Mathias Lindholm added 2 commits November 8, 2022 22:37
Group items better and use Needleman-Wunsch alignment for asr evaluation output
@bigdatabaracus
Copy link
Contributor

bigdatabaracus commented Nov 9, 2022

Thanks @thisismatu Nice! One comment right of the bat. I think using the Needleman-Wunsch alignment also for NLU to highlight differences would make a lot of sense. As an example.

Line: 2
└─ Ground truth: *add_to_cart i want -two----------- of those
└─ Prediction:   *add_to_cart i want [two|2](amount) of those

@bigdatabaracus
Copy link
Contributor

Thanks @thisismatu for the update. Out of curiosity does the second NLU example in the PR description look now like this now?

Line: 4
└─ Ground truth: *set_delivery_date delivery [tomorrow|2022-10--09](delivery_date)
└─ Prediction:   *set_delivery_date delivery [tomorrow|2022-1-1-09](delivery_date)  

@thisismatu
Copy link
Author

thisismatu commented Nov 9, 2022

@bigdatabaracus it varies, hence i didn't add it in the first place. As there's no customization, our options are to remove it for nlu or live with this.

Few example outputs with ground truth on top and prediction on bottom:

[tomorrow|2022-10-11]
[tomorrow|2022-11-10]

[tomorrow|2021-01--01]
[tomorrow|2022-11-10-]

[tomorrow|1998-12-21-]
[tomorrow|2022-11--10]

@bigdatabaracus
Copy link
Contributor

The hyphen symbol is an unfortunate choice of alignment character for the go library. We are just about to add that character to our ASR output as well. 🤔

teelisyys
teelisyys previously approved these changes Nov 9, 2022
@thisismatu thisismatu merged commit bb8c46b into master Nov 9, 2022
@thisismatu thisismatu deleted the format-evaluate-output branch November 9, 2022 11:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants