oneCore synth: Fix lag between utterances introduced in NVDA 2019.3. #11023
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When pushing the last chunk of audio for an utterance, use WavePlayer.sync() instead of WavePlayer.idle(). We do this because idle closes the audio device. If there's something in the queue after playing the final chunk, that will result in WaveOutOpen being called in the callback when we push the next chunk of audio. We *really* don't want this because calling WaveOutOpen blocks for ~100 ms if called from the callback when the SSML includes marks, resulting in lag between utterances. If the queue is empty after sync returns, only then do we call idle and allow the device to close.
This was referenced Apr 17, 2020
Closed
feerrenrut
approved these changes
Apr 17, 2020
Contributor
feerrenrut
left a comment
feerrenrut
left a comment
There was a problem hiding this comment.
Thanks @jcsteh, I have tested this build and can confirm.
I made recordings of alpha vs this PR using One Core - David at rate 40 with rate boost enabled.
With "speak key presses" enabled, on the desktop I pressed a key and waited for the file name to be announced.
I measured the time between the key press announcement and the icon name on the desktop. On average this PR is was about 150ms faster on my machine.
Collaborator
|
This might solve also #10971. |
feerrenrut
pushed a commit
that referenced
this pull request
Aug 10, 2020
…io played for 10 seconds. (PR #11024) Fixes #5172 Fixes #10721 ## Summary of the issue: - As described in #5172, some audio drivers/hardware take a long time to open the device and/or truncate the start/end of the audio. - As described in #10721 (comment), calling WaveOutOpen in the OneCore synth callback mysteriously blocks (and thus lags) for ~100 ms. Because we close the audio device on idle, we can trigger this problem. Although #11023 mostly fixes this, it's impossible (or at least very difficult) to resolve this completely from within the OneCore driver. - Aside from all of this, closing and opening the audio device for rapid short utterances (e.g. rapid movement with the cursor keys or typing) doesn't seem particularly optimal. It's difficult to measure this, but I'd say mitigating this is likely to make audio performance faster/smoother. ## Description of how this pull request fixes the issue: When WavePlayer.idle() is called and closeWhenIdle is set, instead of closing the device immediately after the audio finishes, set a timer. If audio is played before the timer elapses, stop the timer. Close the device when the timer expires.
feerrenrut
added a commit
that referenced
this pull request
Sep 1, 2020
Fixes #5172 Fixes #10721 Fixes #11482 Fixes #11490 ### History #5172 Some audio drivers/hardware take a long time to open the device and/or truncate the start/end of the audio. #10721 Calling WaveOutOpen in the OneCore synth callback mysteriously blocks (and thus lags) for ~100 ms. Because we close the audio device on idle, we can trigger this problem. Although PR #11023 mostly fixed this, it's impossible (or at least very difficult) to resolve this completely from within the OneCore driver. PR #11024 attempted to fix these issues by waiting 10 seconds before closing the audio device" ### Problem to solve Subsequent to #11024, there are occasional exceptions from nvwave, particularly when switching synthesisers. In particular the following cases need to be handled smoothly: - When using Microsoft Sound Mapper, NVDA should use the Windows default device (even if it changes) - When the NVDA configured devices becomes invalid, nvWave should fall back to Microsoft Sound Mapper - When the NVDA configured device becomes available again, NVDA should switch back to using it. - Handle no audio device at all. Since these issues needed to be fixed, and also: - Closing and opening the audio device was originally introduced to support Remote Desktop audio, this is now better served by other solutions. - Performance is improved by keeping it open, using a timeout means that the lag is more rare, but will still occur. ### How it is solved Instead, now don't close the device at all. As per the discussion: #11505 (comment) To fix issues with current / preferred device: - nvWave now saves the preferred device when it is constructed. - If using a device fails, it is considered unavailable, nvWave attempts to fall back to "Microsoft Sound Mapper". - From my testing, using "Microsoft Sound Mapper" correctly handled changing the default device (Win 10 2004). It would be helpful if others could confirm, especially on different OS versions. - During `_idleUnbuffered` - The current device is checked to see if it matches the preferred device. - The available devices are polled to see if the preferred device is available, if so it switches.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Link to issue number:
Fixes #10721. Mostly.
Summary of the issue:
When using Windows OneCore speech, there is a significant lag between utterances which was not present in NVDA <= 2019.2.1.
Description of how this pull request fixes the issue:
See #10721 (comment) for additional background.
When pushing the last chunk of audio for an utterance, use WavePlayer.sync() instead of WavePlayer.idle().
We do this because idle closes the audio device.
If there's something in the queue after playing the final chunk, that will result in WaveOutOpen being called in the callback when we push the next chunk of audio.
We really don't want this because calling WaveOutOpen blocks for ~100 ms if called from the callback when the SSML includes marks, resulting in lag between utterances.
If the queue is empty after sync returns, only then do we call idle and allow the device to close.
Testing performed:
Tested as per STR in #10721 (comment).
Known issues with pull request:
Unfortunately, there's still a rare race condition where as we're closing the device, something can arrive in the queue, so we hit the same issue again. I think a much broader solution is needed to fix this. See #10721 (comment).
Change log entry:
Bug fixes:
- The Windows OneCore speech synthesizer no longer lags between utterances. (#10721)