-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Assertion: "About to add the priority listener twice and only one should exist on the Utterance...." #46
Comments
OK I identified the case, it is actually pretty simple. We are not handling what should happen when you add an utterance to the queue while the Announcer is announcing that Utterance. const testUtterance = new phet.utteranceQueue.Utterance( { alert: 'This is a test utterance.', announcerOptions: { cancelSelf: false } } );
phet.scenery.voicingUtteranceQueue.addToBack( testUtterance );
window.setTimeout( () => {
phet.scenery.voicingUtteranceQueue.addToBack( testUtterance );
}, 500 ); |
For behavior, it seems important that if you add the Utterance to the queue while the Announcer is still speaking it it should not interrupt. We cannot simply remove the assertion. If this ran without error, the listener would be removed when the Announcer was done speaking the Utterance the first time. So changing the priorityProperty would have no effect while it was speaking the Utterance the second time. |
Brainstorming fixes:
|
I feel like this is the way to go, but I would make it a list of functions, so you can remove the one that applies to you. Sorta like the "count" features of scenery node structures. We just said that a constraint of the system (Announcer + UtteranceQueue) is that one Utterance can be in multiple stages of the system at the same time. If duplicates are allowed, then changing the Map to be more tolerant feels great to me. Thoughts? |
I like that, but I am struggling with how to find the right listener to remove from the list. |
|
I liked this idea, this morning I tried an approach where we use counting variables to decide whether to add or remove a priorityProperty listener. It is basically a way to implement
without looking at the Announcer. UtteranceWrapper has a counting variable
If counting variable is greater than zero when it is time to add or remove a priorityProperty listener we do nothing. In that case, the Utterance exists in the queue or is still being spoken by the Announcer. So far it is working well. I am not seeing this problem anymore and unit tests are passing but it feels pretty complicated and I suspect there is an easier way. Here is the patch with this solution though: Index: js/UtteranceQueue.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/UtteranceQueue.js b/js/UtteranceQueue.js
--- a/js/UtteranceQueue.js (revision 439ba41f11aca54bd94e0878ccc0ec83518f2730)
+++ b/js/UtteranceQueue.js (date 1642785722172)
@@ -89,10 +89,18 @@
// utteranceToPriorityListenerMap.
this.announcer.announcementCompleteEmitter.addListener( utterance => {
+ const utteranceWrapper = Array.from( this.utteranceToPriorityListenerMap.keys() ).find( utteranceWrapperInMap => utteranceWrapperInMap.utterance === utterance );
+
// TODO: Can we replace this with an assertion to enforce that it exists? It breaks when using voicingManager.speakIgnoringEnabled but it shouldn't. See https://github.com/phetsims/joist/issues/752
// assert && assert( this.utteranceToPriorityListenerMap.has( utterance ), 'Utterance missing from utteranceToPriorityListenerMap' );
- if ( this.utteranceToPriorityListenerMap.has( utterance ) ) {
- this.removePriorityListener( utterance );
+ if ( this.utteranceToPriorityListenerMap.has( utteranceWrapper ) ) {
+ assert && assert( utteranceWrapper.usageCount > 0, 'Utterance was used by Announcer, it should have a usage count' );
+ utteranceWrapper.usageCount--;
+ if ( assert && utteranceWrapper.usageCount > 0 ) {
+ assert( this.queue.includes( utteranceWrapper ), 'Done speaking, only way utteranceWrapper can have usage is if it exists in the queue' );
+ }
+
+ this.removePriorityListener( utteranceWrapper );
}
} );
@@ -137,6 +145,8 @@
// Remove identical Utterances from the queue and wrap with a class that will manage timing variables.
const utteranceWrapper = this.prepareUtterance( utterance );
+ assert && assert( utteranceWrapper.usageCount <= 1, 'At this point the Utterance cannot be in the queue because of prepareUtterance but may be in the Announcer, there is at most one usage.' );
+ utteranceWrapper.usageCount++;
// Add to the queue before prioritizing so that we know which Utterances to prioritize against
this.queue.push( utteranceWrapper );
@@ -179,13 +189,17 @@
* @param utteranceWrapper {UtteranceWrapper}
*/
addPriorityListenerAndPrioritizeQueue( utteranceWrapper ) {
- assert && assert( !this.utteranceToPriorityListenerMap.has( utteranceWrapper.utterance ),
- 'About to add the priority listener twice and only one should exist on the Utterance. The listener should have been removed by removeOthersAndUpdateUtteranceWrapper.' );
- const priorityListener = () => {
- this.prioritizeUtterances( utteranceWrapper );
- };
- utteranceWrapper.utterance.priorityProperty.lazyLink( priorityListener );
- this.utteranceToPriorityListenerMap.set( utteranceWrapper.utterance, priorityListener );
+
+ if ( utteranceWrapper.usageCount <= 1 ) {
+ assert && assert( !this.utteranceToPriorityListenerMap.has( utteranceWrapper ),
+ 'About to add the priority listener twice and only one should exist on the Utterance. The listener should have been removed by removeOthersAndUpdateUtteranceWrapper.' );
+ const priorityListener = () => {
+ this.prioritizeUtterances( utteranceWrapper );
+ };
+
+ utteranceWrapper.utterance.priorityProperty.lazyLink( priorityListener );
+ this.utteranceToPriorityListenerMap.set( utteranceWrapper, priorityListener );
+ }
this.prioritizeUtterances( utteranceWrapper );
}
@@ -245,6 +259,13 @@
// remove all occurrences, if applicable
const removedUtteranceWrappers = _.remove( this.queue, utteranceWrapperToUtteranceMapper );
+ removedUtteranceWrappers.forEach( utteranceWrapper => {
+ utteranceWrapper.usageCount -= removedUtteranceWrappers.length;
+
+ assert && assert( utteranceWrapper.usageCount <= 1, 'The Utterance is only used by the Announcer if at all, should be one or less usages' );
+ assert && assert( utteranceWrapper.usageCount >= 0, 'negative usages??' );
+ } );
+
if ( options.removePriorityListener ) {
this.removePriorityListeners( removedUtteranceWrappers );
}
@@ -331,12 +352,14 @@
assert && assert( utteranceWrapper instanceof UtteranceWrapper );
const times = [];
+ const usageCounts = [];
// we need all the times, in case there are more than one wrapper instance already in the Queue.
for ( let i = 0; i < this.queue.length; i++ ) {
const currentUtteranceWrapper = this.queue[ i ];
if ( currentUtteranceWrapper.utterance === utteranceWrapper.utterance ) {
times.push( currentUtteranceWrapper.timeInQueue );
+ usageCounts.push( currentUtteranceWrapper.usageCount );
}
}
@@ -344,8 +367,18 @@
utteranceWrapper.timeInQueue = Math.max( ...times );
}
+ // Make sure that the usageCount from the utterances that were removed is propagated to the new utteranceWrapper
+ // so that we know if announcer is still announcing this Utterance.
+ if ( usageCounts.length >= 1 ) {
+ utteranceWrapper.usageCount = Math.max( ...usageCounts );
+ }
+
// remove all occurrences, if applicable. This side effect is to make sure that the timeInQueue is transferred between adding the same Utterance.
const removedWrappers = _.remove( this.queue, currentUtteranceWrapper => currentUtteranceWrapper.utterance === utteranceWrapper.utterance );
+ utteranceWrapper.usageCount -= removedWrappers.length;
+ assert && assert( utteranceWrapper.usageCount <= 1, 'Only usage could be for the Announcer, should be one or less Usages' );
+ assert && assert( utteranceWrapper.usageCount >= 0, 'negative usages??' );
+
this.removePriorityListeners( removedWrappers );
}
@@ -425,21 +458,21 @@
* @param utteranceWrappers
*/
removePriorityListeners( utteranceWrappers ) {
- utteranceWrappers.forEach( utteranceWrapper => this.removePriorityListener( utteranceWrapper.utterance ) );
+ utteranceWrappers.forEach( utteranceWrapper => this.removePriorityListener( utteranceWrapper ) );
}
/**
* @private
- * @param utterance
+ * @param utteranceWrapper
*/
- removePriorityListener( utterance ) {
- const listener = this.utteranceToPriorityListenerMap.get( utterance );
+ removePriorityListener( utteranceWrapper ) {
+ const listener = this.utteranceToPriorityListenerMap.get( utteranceWrapper );
// The same Utterance may exist multiple times in the queue if we are removing duplicates from the array,
// so the listener may have already been removed.
- if ( listener ) {
- utterance.priorityProperty.unlink( listener );
- this.utteranceToPriorityListenerMap.delete( utterance );
+ if ( listener && utteranceWrapper.usageCount === 0 ) {
+ utteranceWrapper.utterance.priorityProperty.unlink( listener );
+ this.utteranceToPriorityListenerMap.delete( utteranceWrapper );
}
}
@@ -564,6 +597,9 @@
utteranceWrapper.stableTime = Number.POSITIVE_INFINITY;
utteranceWrapper.timeInQueue = Number.POSITIVE_INFINITY;
+ assert && assert( utteranceWrapper.usageCount <= 1, 'At this point the Utterance cannot be in the queue because of prepareUtterance but may be in the Announcer, there is at most one usage.' );
+ utteranceWrapper.usageCount++;
+
// addPriorityListenerAndPrioritizeQueue assumes the UtteranceWrapper is in the queue, add first
this.queue.unshift( utteranceWrapper );
this.addPriorityListenerAndPrioritizeQueue( utteranceWrapper );
@@ -591,6 +627,10 @@
// only announce the utterance if not muted and the Utterance predicate returns true
if ( !this._muted && utterance.predicate() && utterance.getAlertText( this.announcer.respectResponseCollectorProperties ) !== '' ) {
+
+ utteranceWrapper.usageCount++;
+ assert && assert( utteranceWrapper.usageCount === 2, 'Utterance in both queue and announcer at this point, should have two usages' );
+
this.announcer.announce( utterance, utterance.announcerOptions );
sentToAnnouncer = true;
}
@@ -604,7 +644,7 @@
// only remove the priority listener if it has not been received by the Announcer, otherwise the Announcer
// will let us know when it is finished with it and we will remove the listener then
- removePriorityListener: !sentToAnnouncer
+ removePriorityListener: true
} );
}
}
@@ -672,6 +712,8 @@
// in this case the time will be since the first time the utterance was added to the queue.
this.timeInQueue = 0;
+ this.usageCount = 0;
+
// @public {number} - in ms, how long this utterance has been "stable", which
// is the amount of time since this utterance has been added to the utteranceQueue.
this.stableTime = 0;
|
FYI, with this patch, I can test voicing code that doesn't use priority: Index: js/UtteranceQueue.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/UtteranceQueue.js b/js/UtteranceQueue.js
--- a/js/UtteranceQueue.js (revision 439ba41f11aca54bd94e0878ccc0ec83518f2730)
+++ b/js/UtteranceQueue.js (date 1642787271019)
@@ -181,11 +181,11 @@
addPriorityListenerAndPrioritizeQueue( utteranceWrapper ) {
assert && assert( !this.utteranceToPriorityListenerMap.has( utteranceWrapper.utterance ),
'About to add the priority listener twice and only one should exist on the Utterance. The listener should have been removed by removeOthersAndUpdateUtteranceWrapper.' );
- const priorityListener = () => {
- this.prioritizeUtterances( utteranceWrapper );
- };
- utteranceWrapper.utterance.priorityProperty.lazyLink( priorityListener );
- this.utteranceToPriorityListenerMap.set( utteranceWrapper.utterance, priorityListener );
+ // const priorityListener = () => {
+ // this.prioritizeUtterances( utteranceWrapper );
+ // };
+ // utteranceWrapper.utterance.priorityProperty.lazyLink( priorityListener );
+ // this.utteranceToPriorityListenerMap.set( utteranceWrapper.utterance, priorityListener );
this.prioritizeUtterances( utteranceWrapper );
} |
Here is another solution that is more simple. It is also passing unit tests. It works by saving a reference to the UtteranceWrapper and priorityPropertyListener when we call this.announcer.announcementCompleteEmitter.addListener( utterance => {
// It is possible that this.announcer is used by a different UtteranceQueue. When the announcementCompleteEmitter
// announces, it may not be for this queue. Would love the following assertions though.
// TODO: This would break if both UtteranceQueues that share the same announcer both have the same Utterance at this.announcingUtteranceWrapper.utterance, https://github.com/phetsims/utterance-queue/issues/46
// assert && assert( this.announcingUtteranceWrapper, 'no announcingUtteranceWrapper' );
// assert && assert( this.announcingUtterancePriorityListener, 'no announcingUtterancePriorityListener' );
if ( this.announcingUtteranceWrapper && utterance === this.announcingUtteranceWrapper.utterance ) {
assert && assert( this.announcingUtteranceWrapper.utterance.priorityProperty.hasListener( this.announcingUtterancePriorityListener ) );
this.announcingUtteranceWrapper.utterance.priorityProperty.unlink( this.announcingUtterancePriorityListener );
this.announcingUtteranceWrapper = null;
this.announcingUtterancePriorityListener = null;
}
} ); Full patch: Index: js/UtteranceQueue.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/UtteranceQueue.js b/js/UtteranceQueue.js
--- a/js/UtteranceQueue.js (revision 439ba41f11aca54bd94e0878ccc0ec83518f2730)
+++ b/js/UtteranceQueue.js (date 1642788732997)
@@ -85,14 +85,24 @@
// removed from the queue.
this.utteranceToPriorityListenerMap = new Map();
+ this.announcingUtteranceWrapper = null;
+ this.announcingUtterancePriorityListener = null;
+
// When the Announcer is done with an Utterance, remove priority listeners and remove from the
// utteranceToPriorityListenerMap.
this.announcer.announcementCompleteEmitter.addListener( utterance => {
- // TODO: Can we replace this with an assertion to enforce that it exists? It breaks when using voicingManager.speakIgnoringEnabled but it shouldn't. See https://github.com/phetsims/joist/issues/752
- // assert && assert( this.utteranceToPriorityListenerMap.has( utterance ), 'Utterance missing from utteranceToPriorityListenerMap' );
- if ( this.utteranceToPriorityListenerMap.has( utterance ) ) {
- this.removePriorityListener( utterance );
+ // It is possible that this.announcer is used by a different UtteranceQueue. When the announcementCompleteEmitter
+ // announces, it may not be for this queue. Would love the following assertions though.
+ // TODO: This would break if both UtteranceQueues that share the same announcer both have the same Utterance at this.announcingUtteranceWrapper.utterance, https://github.com/phetsims/utterance-queue/issues/46
+ // assert && assert( this.announcingUtteranceWrapper, 'no announcingUtteranceWrapper' );
+ // assert && assert( this.announcingUtterancePriorityListener, 'no announcingUtterancePriorityListener' );
+ if ( this.announcingUtteranceWrapper && utterance === this.announcingUtteranceWrapper.utterance ) {
+ assert && assert( this.announcingUtteranceWrapper.utterance.priorityProperty.hasListener( this.announcingUtterancePriorityListener ) );
+ this.announcingUtteranceWrapper.utterance.priorityProperty.unlink( this.announcingUtterancePriorityListener );
+
+ this.announcingUtteranceWrapper = null;
+ this.announcingUtterancePriorityListener = null;
}
} );
@@ -591,6 +601,17 @@
// only announce the utterance if not muted and the Utterance predicate returns true
if ( !this._muted && utterance.predicate() && utterance.getAlertText( this.announcer.respectResponseCollectorProperties ) !== '' ) {
+
+ assert && assert( this.announcingUtteranceWrapper === null, 'This should have been cleared' );
+ assert && assert( this.announcingUtterancePriorityListener === null, 'This should have been cleared' );
+
+ this.announcingUtteranceWrapper = utteranceWrapper;
+ this.announcingUtterancePriorityListener = () => {
+ this.prioritizeUtterances( utteranceWrapper );
+ };
+
+ utteranceWrapper.utterance.priorityProperty.link( this.announcingUtterancePriorityListener );
+
this.announcer.announce( utterance, utterance.announcerOptions );
sentToAnnouncer = true;
}
@@ -604,7 +625,7 @@
// only remove the priority listener if it has not been received by the Announcer, otherwise the Announcer
// will let us know when it is finished with it and we will remove the listener then
- removePriorityListener: !sentToAnnouncer
+ removePriorityListener: true
} );
}
} EDIT: THe problem mentioned here is actually a problem for the patch in #46 (comment) as well. Confirmed that this breaks with patch in that comment. This is an issue with master as well. const testUtterance = new phet.utteranceQueue.Utterance( {
alert: 'This is a test utterance.',
alertStableDelay: 0,
announcerOptions: { cancelSelf: false }
} );
phet.scenery.voicingUtteranceQueue.addToBack( testUtterance )
phet.joist.joistVoicingUtteranceQueue.addToBack( testUtterance ) |
…nce is being announced, see #46
@zepumph and I discussed this during a11y dev meeting today and decided that the issue identified in #46 (comment) is a separate issue and also low priority (see #48). That freed up both solutions in mentioned above. The counting one seemed unnecessarily complicated so we went with #46 (comment). While pairing, we refined some of the checks in the I just committed the refined patch and verified that unit tests are passing. Next I would like to turn the case that identified this into a unit test. Then I think this issue can be closed. const testUtterance = new phet.utteranceQueue.Utterance( { alert: 'This is a test utterance.', announcerOptions: { cancelSelf: false } } );
phet.scenery.voicingUtteranceQueue.addToBack( testUtterance );
window.setTimeout( () => {
phet.scenery.voicingUtteranceQueue.addToBack( testUtterance );
}, 500 ); |
Unit test added above, it is passing in Chrome and Firefox. |
Options were removed from |
Id love to review the commits. I think there is one more assertion to add reopening. |
Please double check to make sure my changes are correct in your mind. This looks really really excellent. I want to note an important point of this work, just to make sure I understand it correctly. Let me know if you feel like this needs more explanation in the code. . . . This is a good pattern because it provides rigitity in how we ensure proper management of the priority listener add/remove calls, but flexibility as to whether or not the announcer actually cares about/handles priority (i.e. a lack of Feel free to close. |
I like that assertion a lot, and things are working well with it.
Yes, your description here is correct.
Sounds good, thank you for reviewing this. |
While working on review for #43 I hit this assertion
I think it is consistently happening in Chrome when pressing the voicing toolbar buttons rapidly.
The text was updated successfully, but these errors were encountered: