Alt input for rulers and toolbox.

[pixelzoom]

Related to #235 ...

Geometric Optics has a toolbox where the rulers live. There is 1 vertical ruler, and 1 horizontal ruler. It looks like this, and behaves like toolboxes in other PhET sims:

In #235 (comment), it was noted that alt input has not been done for a toolbox yet. @terracoda said:

@pixelzoom, we haven't explicitly done a grab/drag tools in toolbox, but we have done a variety of grab/drag interactions, most notably BASE, Friction. FL, GFL (regular). Each have their own specific sim-centric challenges more related to Interactive Description. All share the two-step grab then drag UX pattern. A learner lands on a "Grab X" interaction which is a native HTML button (very intuitive interaction), and then grabbing the object launches web application role allow the learner to use WASD or Arrow keys to move the object in any direction. There is a relatively standard looking visual keyboard hint that lets visual users know that they need to do the grab interaction with Space, though both Space and Enter work on a native button. The release interaction can be triggered by a variety of key presses Space, Enter, Tab to a new object, and possibly the Escape key.

And note that the grab/drag interaction still needs some improvements to be accessible on touch devices, see phetsims/balloons-and-static-electricity#458 and related issues. Currently the visual keyboard hint is showing up on touch devices. The hint should be platform specific. The interaction is fully working for the ruler in the published version of GFL regular. I hope that is helpful.

In #235 (comment), @arouinfar said:

Because of the toolbox interaction of putting in/taking out, I think we'll need to use the grab/drag interaction for rulers. While the grab/drag interaction has been done in several sims (e.g. balloons in BASE, magnet in Faraday's Law, ruler in GFL) it's never been done in a context with a toolbox. The full-sized item appears when a toolbox icon is clicked and it must then be dragged away from the toolbox to stay in the play area. Similarly, to put the ruler away, it will need to be released over the toolbox. I don't see how this could work with the Fourier pattern.

So before we get to implementation, there's some a11y design to be done here. Assigning to @arouinfar to discuss with the a11y team.

[pixelzoom]

In Slack, @kathy-phet said:

... Also, at some point this quarter or early next quarter, this work will need a bit of co-design time with Taliesin on the tool box patterns. TS and EM, if there is a time that might work well to layer in let me know. I don't think it would be too much time, but not sure how it will go.

Also in Slack, @emily-phet said:

Great! Glad to hear the toolbox will be getting some alternative input love.
...
Re toolbox design pattern: A few considerations that will likely be important for whoever is leading that:

• Would be good to know the span of behaviors in PhET sims for objects in the toolbox. Some "pop out", and become draggable objects. Some "pop out" and have multiple draggable pieces, etc. If someone could isolate all sims with toolboxes and list out all of the behaviors that happen, that would be helpful in making sure a design is robust across at least this set of toolbox situations. From my memory, one of the challenges with toolboxes is deciding what happens once you taken something out - does focus move to that object now in the play area, etc.
• Taliesin will likely have more thoughts on this, but how you design the alt input directly impacts the Interactive Description and Voicing experience. For example, the state of the simulation changes when you have meters out in use, which could change how parts of the PDOM are organized, etc. If the toolbox object is draggable, it might need to pop out and become a button to initiate grab and drag, etc. (What actually are the objects when they're in the toolbar, vs when they're out - i.e., are they buttons or toggles somehow in the toolbar to allow for them to be "activated" and something else outside of the toolbar... I don't know, but this will dictate how they're read out for interactive description.) You likely won't be able to fully explore all of these features together quite as deeply as you might like to finalize the alt input design, but would be good to have the needs of these features in mind while working on toolbox design.

[pixelzoom]

A few misc thoughts about this topic...

• PhET has a well-defined and widely-used pattern for implementing toolboxes, see Creator (with Drag Forwarding). Designers should familiarize themselves with this pattern, to inform the design.

• Buckets are another common UI element that behave much like toolboxes. You drag things out of buckets, and return things to buckets. Buckets and toolboxes both use the same "Creator" pattern mentioned above. Can the design work for both toolboxes and buckets?

• The item that is dragged out of a toolbox (or bucket) may be something simple like a ruler. Or it may be a complicated UI component with it's own interactive subcomponents - e.g. a measuring tape, a meter with probes and buttons.

• This design should not be based on the needs of Geometric Optics. Do a full survey of all PhET sims, to see how toolboxes (and buckets) are used.

[emily-phet]

Regarding:

Buckets are another common UI element that behave much like toolboxes. You drag things out of buckets, and return things to buckets. Buckets and toolboxes both use the same "Creator" pattern mentioned above. Can the design work for both toolboxes and buckets?"

It would be awesome if the design could address both buckets and toolboxes. Some related thoughts:

• My experience with toolboxes is the situation where the main focus for many objects is to provide access to the tool in the play area, and this may or may not require grab/dragging of the object. So emphasis when using a toolbox is on selecting the tool you want, and it appears in the play area. For example, "activating" something in a toolbox (e.g., clicking it) can result in it "popping" out into the play area, available potentially for dragging or other interaction. From an alternative input perspective, I would guess this is the most straightforward route to take, avoiding requiring that interaction with objects in a toolbox involve grab/drag interaction. So, interaction with an object in a toolbar is akin to selecting a checkbox to indicate you want X object in play area. Once in the play area, the object may be draggable, but doesn't have to be.
• My experience with buckets is the situation where often it's rather important that the object "pulled out" from the bucket is then dragged to a chosen location. In some cases clicking the bucket results in automatic placement of one item somewhere, but intent behind the use of a bucket is that the learner is selecting an object and placing the object. It may be more often that the bucket scenario requires the object to be draggable object from the onset of interaction. So interaction with a bucket would be akin to pushing the "grab yellow balloon" button in BASE.

In an oversimplification, to me, bucket item = draggable object, while toolbox item = object that can be added to play area (once object is in play area, it may be a draggable object).

[pixelzoom]

There was discussion about alternative input features at 11/11/21 dev meeting, and I identified this issue as a "gap" in alternative input support for Geometric Optics. @zepumph offered to check in with @arouinfar to verify whether this is a "must have" for Geometric Optics release, and (if so) how to get this issue moving forward so that it can be incorporated into the release.

[emily-phet]

@zepumph and I spoke yesterday about this issue and phetsims/a11y-research#166 (comment) - and I had a follow-up conversation this morning with @terracoda. Here's my recommendation and summary of outcomes from these discussions.

I recommend that we design an interaction pattern / implementation plan for the toolbox in Geometric Optics now, decoupling the more holistic design process of interaction pattern / implementation plan for all PhET toolboxes and tools from the publication goal for Geometric Optics. Both @zepumph and @terracoda agreed with this.

Reasoning: Because there are only two rulers in the toolboxes for Geometric Optics, and there is already an example of ruler interaction once out of the toolbox in Gravity Force Lab, the toolbox needs for Geometric Optics are a small subset of the total needs for all PhET toolboxes and tools. We think a reasonable solution for Geometric Optics that is likely very close to what would be implemented after a lengthier design process can be found.

On 11/12/2021, @zepumph created a demo of a potential toolbox interaction with Waves: Intro, and showed this to me. Here are recommendations for refinements to this that I think would be a good starting point for design discussions for Geometric Optics, and the remaining questions @terracoda and I thought would want to be considered by the Geometric Optics team. (I'll post a more complete description of design considerations for PhET toolboxes in general in phetsims/a11y-research#166 (comment) today).

• What kind of objects are the tools in the toolbox instrumented as? @zepumph demo had them as buttons. @terracoda and I agreed this made sense for Geometric Optics in the short term: simple push buttons to activate the rulers (jumping them out of the toolbox and into the play area). Once a tool is out of the toolbox, the button is not available to be interacted with again until the ruler has been returned.
• (Tab Order related) Are the tools grouped within the toolbox? @terracoda and I recommend that the tools in the toolbox are considered a group within the toolbox (so that you navigate to the toolbox with the tab key, and then arrow between the tools). For toolboxes with one object, this is probably not needed. For 3 or more tools it's likely very much needed. For 2 tools, it's more of a toss-up, but @terracoda and I thought that in the long-term it was likely that tools would be grouped, so we're recommending this for Geometric Optics.
• (Tab Order related) Where does focus go after a tool button has been activated? We recommend the focus go to the tool in the play area, as the "Grab" button for that ruler (in the case of Geometric Optics).
• How is the ruler tool moved? We recommend the grab + drag interaction, like the ruler in Gravity Force Lab. Note - this is different than @zepumph's 11/11/2021 Waves Intro demo.
• How does the tool return to the toolbox? In @zepumph's initial demo, tools could be returned by moving the object over the toolbox. I recommend this approach, as well as shortcut key. The shortcut key should not be "Esc" if the tool has a drag+grab interaction, since "Esc" already has an associated action, releasing the object in the play area.
• (Not really about the toolbox, but about the ruler when in play area): I recommend the design team consider shortcut keys (see shortcut keys for balloons in BASE, and for ruler in GFL) to help learners jump the rulers to the most useful places. You might also want to consider "snapping" behavior to help learners align the ruler with useful things in the sim.

[zepumph]

That sounds reasonable to me, though the PDOM will be different from "buttons" given the needs of cycling through tool items with arrow keys. Since we don't need to support description, the elements themselves can be anything, and we will add our own listeners to the toolbox in general to support the interaction. In fact we don't want each tool to be a button because then each tool will be in the focus order.

@pixelzoom, I think the best path forward would be to let me know if you would like to pair once #263 is finished. I foresee that changing the very code we will need to implement this. I'm happy to help!

Noting here my demo for public consumption: https://phet-dev.colorado.edu/html/waves-intro/1.2.0-altInputToolbox.1/phet/waves-intro_all_phet.html

[pixelzoom]

Noting here my demo for public consumption: https://phet-dev.colorado.edu/html/waves-intro/1.2.0-altInputToolbox.1/phet/waves-intro_all_phet.html

@zepumph can you please walk me through this demo? There are no rulers in this sim, so I'm guessing that you've implemented something for the toolbox in the upper-left corner of each screen. I also can't seem to tab into this sim. Was it published with alt input support enabled, or do I need to use some query parameter? A pointer to the relevant code would also be appreciated, thanks.

[pixelzoom]

@zepumph asking again - can you please walk me through this demo? I've revised the rulers implementation in #263, and I'm ready to try applying your demo.

[zepumph]

Yes thanks for the ping.

I'll frame the prototype, and then discuss how we should apply it to this sim.

I also can't seem to tab into this sim. Was it published with alt input support enabled, or do I need to use some query parameter?

That is confusing to me, I am able to use alt input on my computer. Perhaps a pair if you can't get it to work. To test the prototype launch the sim, go to the the "Water" screen, focus the stopwatch (2nd in focus order for me), and press space or enter to pull it out of the toolbox. Then you can drag it around freely with arrow keys.

See phetsims/wave-interference@6781e2c and phetsims/scenery-phet@2515980 from over in phetsims/a11y-research#166 to get the code that the prototype is using.

Taking the demo, and applying it to GO now (using the design from #258 (comment)). You will want to do the following:

• Make the rulers (not the ruler icons) a GrabDragInteraction, see ISLCRulerNode as an example, and please create issues for fine tuning or confusion along the way. This involves creating a KeyboardDragListener as well.
• Make the toolbox container a button element (button is likely acceptable).
• Add a listener on keydown (or perhaps keyup depending on what feels best) to the toolbox that changes the ruler selection with the arrow keys (ignoring any tools that have already been taken out).
• Add a listener that on the click that removes the icon, shows the ruler, and focuses the ruler, it should be in "grab" mode here, with a banner that says "space to grab" etc.
• Add code (likely on end or drag of the KeyboardDragListener) such that when it is back over the toolbox the ruler is hidden.
• Add a listenersForDragState to GrabDragInteraction that is a keyup so that when escape is pressed, it is returns the ruler to the toolbox.
• Consider adding hotkeys to your drag listener (see KeyboardDrayListener.addHotkey and usages).
• Create keyboard help content to support these interactions.

Let me know if you want further assistance. Upon reflection, I would make all the keydown listeners keyup to start with. I think that will be a better UX.

Good luck!

[kathy-phet]

@zepumph @pixelzoom @arouinfar @terracoda @emily-phet -

EM and I touched on the design behavior for this feature for Geometric Optics at one of our regular meetings shortly after the Nov 12 comments add above. Sorry I didn't summarize our discussion at that time, but since this is getting some activity again, it reminded me that I should add in the discussion content here.

Basically, we discussed the "grab" feature some. I understand from EM that the "grab" behavior was found pedagogically necessary for working with the balloon in BASE during interviews with screen reader users. The "grab" behavior is different then what we have in Fourier, where you just use tab to tab to the wavelength measures and then arrows to move around without an extra "grab" step.

EM mentioned in our discussion that it is an open-question that TS and she have discussed - as to whether the "grab" is needed for all sim components that have this drag interaction. And, in particular, whether "grab" is needed for components that are more secondary to the learning (e.g. not the central feature that you need students to do early on) and also for components that have a more natural cuing that you would want to move them (e.g. rulers for measuring things). EM expressed that it would be desirable to remove the "grab" interaction in cases where it is an extra interaction that is not needed, but that the design research needed to know under what conditions a "grab" step is needed just hasn't been done yet, because of other grant priorities.

So, as we think about the interactions for the GO sim where we have rulers and for building out a general alt-input pattern for draggable things and tool boxes, I think we would want to design in flexibility into the code structure. Where the "grab" functionality is an option that can be opted into or out of, depending on the design needs of the sim. We can use interviews done as part of the GO sim to add to our design understandings here, on when its needed or not, although this would only be for a sim without interactive description so far.

[pixelzoom]

@kathy-phet That all sounds great. But unfortunately not totally in-sync with how I've been proceeding for many weeks now.

For my part... I have almost totally rewritten the rulers in Geometric Optics, because they did not conform to the pattern that PhET uses with toolboxes. Now I'm in the process of adapting @zepumph's prototype to Geometric Optics. Adapting it involves taking sim-specific code that is currently in a branch of Wave Interference, and turning it into sim-specific code for Geometric Optics; there is no common-code support for toolboxes, and there is none planned at the moment. Therefore there is currently no option for "grab" vs some other behavior.

@kathy-phet Assuming that you want me to continue on this path... When I'm done, who would you like me to assign for evaluating the implementation, and doing additional design work?

[emily-phet]

If I'm understanding @kathy-phet 's comment correctly, I think it's important to point out the distinction between getting the rulers into/out of the toolbox, and how they behave once in the Play Area. The grab+drag interaction is for the ruler once it's out in the play area - see the pattern in GFL. Navigating to, into, and "activating" the ruler in the toolbox is different. Please see my comment here:, where I list out the interaction at the toolbox ("activating" the object to appear in the play area) and then it's behavior once in the play area (grab+drag interaction).

[pixelzoom]

What @emily-phet has described is not how @zepumph's prototype works. Once a ruler is out of the toolbox, there is no grab+drag interaction. The ruler is selected using focus traveral. And when it has focus, you can use the arrow keys to move it.

Since it seems like we're not all on the same page, I'm going to stop working on this integration, and assign to @zepumph for comment.

And again, I urge those involved to please test-drive the prototype. I've been proceeding under the assumption that designers have tested the prototype, and that it is indeed the desired behavior (or a good starting point). But it sounds like a test-drive has not been done.

To test:

1 . Run https://phet-dev.colorado.edu/html/waves-intro/1.2.0-altInputToolbox.1/phet/waves-intro_all_phet.html
2. Go to the Waves screen
3. Tab to the stopwatch in the toolbox
4. Press spacebar to pop the stopwatch out of the toolbox. Note that the stopwatch has focus when it comes out of the toolbox.
5. Use the arrow keys to move the stopwatch around using the arrow keys.
6. Tab around in the sim, and eventually tab back to the stopwatch. Note that as soon as the stopwatch has focus, it can be moved using the arrow keys. There is no "grab" interaction required.

[emily-phet]

@pixelzoom Seems reasonable to me.

To be clear, @zepumph 's prototype was (in my understanding), an initial idea. He followed up with a discussion with me, and I followed up with a discussion with @terracoda, and then listed our recommendations in this comment above. I specifically state in that comment that my recommendations (after discussion with @terracoda) are refinements to that prototype, trying to make it clear that the prototype is not the exact desired behavior.

Apologies for any confusion there.

[pixelzoom]

From #258 (comment):

I also can't seem to tab into this sim. Was it published with alt input support enabled, or do I need to use some query parameter?

That is confusing to me, I am able to use alt input on my computer. Perhaps a pair if you can't get it to work.

Apparently one has to Option-Tab in Safari, or change Preferences. A Google search led me to:

As it turns out, Safari does not enable tab highlighting by default. To turn it on:

1. Go to “Preferences”
2. Select the “Advanced” tab
3. Check “Press Tab to highlight each item on a webpage”

This might be good info to include in some a11y developer/QA doc (if it's not already there.)

[zepumph]

The prototype was a starting point. Please follow #258 (comment) to implement the designed first pass for this sim. This means that instead of the ruler being a freely movable object directly out of the toolbox, it needs to be grabbed. This involves following the pattern in ISLCRulerNode for adding a KeyboardDragListener + GrabDragInteraction to a ruler.

@emily-phet and I are in agreement about the steps in #258 (comment), even when they differ from the original prototype.

[pixelzoom]

In 12/16/21 design check-in, I gave a demo of 1.1.0-dev.10, and consensus was that it was a good starting point. And since 1.0 will not have voicing, it's probably good enough to publish with (given a bit of tweaking and polishing). So @kathy-phet instructed me to wait until Q1 2022 before working with @zepumph on the GrabDragInteraction piece of this.

@zepumph and I did meet to review what I've done so far. He suggested adding support for a hotkey to return the ruler to the toolbox. In the process of doing that, we discovered a bug in KeyboardDragListener, see phetsims/scenery#1331. When that bug is fixed, pressing escape will return the ruler to the toolbox.

@arouinfar @kathy-phet (and anyone else interested) please test drive this. And let me know how you'd like to proceed when Q1 2022 rolls around.

[pixelzoom]

@zepumph pointed out that if we use GrabDragInteraction for the rulers, then we should do the same for the other draggable things - source object (framed pictures and arrow representation), second point of interest, light sources, and projection screen.

[arouinfar]

@pixelzoom @zepumph I've reviewed the alt input behavior in master, and I'm really happy with it. I agree with @zepumph about the consistency of interaction -- if we use grab/drag for the rulers, we should use it everywhere. However, I don't think this sim needs to use the GrabDragInteraction since we aren't supporting description/voicing. Grabbing items will add an extra keypress, and the benefit is a better screen reader experience, which isn't yet relevant.

The only thing that needs further refinement is how rulers are returned to the toolbox. The rulers are too eagerly returned to the toolbox. If any part of the ruler bounds overlaps the toolbox, it will be returned. Perhaps a better rule would be that the ruler is returned if its center overlaps the toolbox.

Currently, it would be impossible to use keyboard navigation to move the ruler into the proper position to measure magnification in this situation:

[pixelzoom]

The only thing that needs further refinement is how rulers are returned to the toolbox. The rulers are too eagerly returned to the toolbox. If any part of the ruler bounds overlaps the toolbox, it will be returned. Perhaps a better rule would be that the ruler is returned if its center overlaps the toolbox.

I agree, it's a little too easy to cause the rulers to (accidentally) return to the toolbox. I'll start with the "center" approach, using the center point and checking for "contains". If that doesn't feel good, then we can define a rectangle at the center and check for intersection.

[pixelzoom]

The above commit returns a ruler to the toolbox when its center point is inside the toolbox. This certainly addresses the problem of unintentionally putting the ruler back in the toolbox. But I'm concerned that it may create usability issues.

Questions:

(1) Will users discover this, or will they have problems returning rulers to the toolbox?

(2) Instead of just the center point, should we try a larger part of the ruler? (Middle third? middle quarter?) Would that address the first question?

(3) If we stick with a point, should there be some (standard?) visual cue about where the point is that will return the ruler to the toolbox?

(4) Should the a11y team be consulted about this?

Back to @arouinfar.

[arouinfar]

Thanks @pixelzoom the current behavior addresses my concerns about accidentally putting away a ruler.

(1) Will users discover this, or will they have problems returning rulers to the toolbox?

The far more convenient way to put away the ruler is to use the esc key (which is documented in the keyboard help dialog). I could be convinced that this should be the ONLY way to put away the ruler, but I won't push for it.

The ruler movement is pretty speedy, so it's more likely that users will accidentally put the ruler away rather than experience an unsuccessful attempt. However, with the current implementation, the ruler shouldn't ever be accidentally returned to the toolbox while in the neighborhood of a useful measurement.

(2) Instead of just the center point, should we try a larger part of the ruler? (Middle third? middle quarter?) Would that address the first question?

I think the current behavior (center point) is working nicely, so my vote would be to leave it as-is.

I don't know that using a portion of the ruler would work. The toolbox is rectangular, so the maximum overlap area for the two rulers would be different. The vertical ruler will almost certainly always have the maximum overlap with the toolbox when it is used to measure the magnification of the image.

(3) If we stick with a point, should there be some (standard?) visual cue about where the point is that will return the ruler to the toolbox?

I don't think we need a visual cue related to the return point. The training should really be with the hotkeys in the keyboard help dialog. With a keyboard, there is no need to physically drag the ruler back to the toolbox to put it away.

(4) Should the a11y team be consulted about this?

This feels like a fairly minor change that improves usability, so I am comfortable moving forward without a larger discussion. If anyone tagged in this issue feels strongly or would like to provide further input, they're certainly welcome.

[pixelzoom]

@arouinfar that all sounds good to me.

That completes the work for 1.1 release, so closing. If there are changes needed for future releases (GrabDragInteraction, voicing, etc.), new issue(s) can be created.