Pronunciation Assessment phoneme accuracy score are wrong.

[riiight]

Describe the bug
If you do a pronunciation assessment on a word and make a clear mistake one of the phonemes or syllables, the scores for all phonemes decreases instead of just the one you made the mistake on.

To Reproduce
Steps to reproduce the behavior:

1. Go to the Pronunciation Assessment tool in Speech Studio (https://speech.microsoft.com/portal/_____/pronunciationassessmenttool)
2. Click on the "Enter your own script" tab
3. Enter "automobile" in the text box to be the word you want to test.
4. Click "Record audio with microphone" and say "automobile" correctly.
5. In "Assessment Result", hover over the word "automobile" and see that its scores are nearly 100 across all phonemes.
6. Try again but this time deliberately say "kertomobile".
7. In "Assessment Result", hover over the word "automobile" and see that its all its scores are lower, not just the score for "ao" (the first phoneme that you deliberately said wrong).

Expected behavior
I believe that if only one phoneme is wrong, only its score should lower. It appears as though all phoneme scores lower together making them almost useless.

Please correct this behavior as it is vital for true pronunciation assessment. Thanks.

[pankopon]

@yulin-li Can you please check.

[riiight]

Here's a screenshot from when I said, "kertomobile". This is actually not the worst result I've seen. Sometimes, I see the same score across all phonemes. I'd expect to see 0-20 for the first phoneme and then 95+ for the rest of them.

[yulin-li]

@yinhew @wangkenpu could you please help to check?

[wangkenpu]

Thanks for reporting the issue. We are working on it and will let you know once it's fixed.

[chrisrollins65]

I came here to report this same thing but it looks like you've already reported it. I posted this issue in the Azure community Q&A but didn't get much of a response there. Here's a video I took of the problem: https://youtu.be/RUwA_ZhWpVU

This is happening both in the Speech Studio as well as in my own application where I integrated the API. For the time being I've had to switch to another pronunciation service since this one isn't working correctly.

[bobir01]

switch to another pronunciation service since this one isn't working corre

hi @chrisrollins65 i am this issue too and some other bugs with STT services can tell which pronunciation APIs you found help and reliable Thanks, you can drop it mardonovbobir9@gmail.com

[chrisrollins65]

@bobir01 I'm using an API by Language Confidence: https://rapidapi.com/language-confidence-language-confidence-default/api/pronunciation-assessment1

It's not perfect, it's less robust and it's more expensive, but at least it differentiates between phonemes, unlike Azure's solution. That was the main feature I needed for my purposes.

[bobir01]

@bobir01 I'm using an API by Language Confidence: https://rapidapi.com/language-confidence-language-confidence-default/api/pronunciation-assessment1

It's not perfect, it's less robust and it's more expensive, but at least it differentiates between phonemes, unlike Azure's solution. That was the main feature I needed for my purposes.

@chrisrollins65 Thanks

[wangkenpu]

The issue has been fixed. @riiight @chrisrollins65 @bobir01

We can close this issue. @pankopon

[bobir01]

The issue has been fixed. @riiight @chrisrollins65 @bobir01

We can close this issue. @pankopon

wow thanks,
how about this one Azure/azure-sdk-for-python#31102 (comment)

[wangkenpu]

The issue has been fixed. @riiight @chrisrollins65 @bobir01
We can close this issue. @pankopon

wow thanks, how about this one Azure/azure-sdk-for-python#31102 (comment)

Hello @bobir01. Please discuss this issue in the corresponding thread. It is beyond our team's scope.

[glecaros]

closing the issue as resolved