Mobile VO skips content in the PDOM #175

jessegreenberg · 2022-07-28T15:37:31Z

Identified in
phetsims/friction#292
phetsims/greenhouse-effect#195
phetsims/john-travoltage#449
phetsims/friction#308

Mobile VO sometimes skips PDOM content.

Notes from testing:

In the case of friction, after particles the second two paragraphs in the screen summary are no longer readable in the PDOM.

I thought it might have something to with aria-live, but it doesn't happen after pressing the ResetAllButton which triggers an alert. I tried changing the ResetAllButton alert to assertive but it made no difference.

In one test, I found that content disappeared in friction only after all particles broke away from the book.

I noticed that in Friction, the content that gets skipped is nested under a div:

I tried pulling the paragraphs out of that div, it did not fix it.

I noticed turning VO on and off again fixes the problem.

I removed the div mentioned in the image above. VoiceOver still skips the first paragraph that was under the div, but does read the second one.

I noticed that the paragraphs that gets skipped are the ones whose content changes.

Ive been creating a more and more complicated test case, trying to reproduce what we are seeing in the sim. The demo as an aria-live element that spams alerts and paragraphs of text content that update. Ive tried

Keeping static paragraph content
Updating paragraph innerHTML and innerText
Removing then adding new paragraphs with content to the DOM
Using CSS to have the PDOM styled the way we have it in the sim
Using polite and assertive alerts
Increasing/decreasing the rate of alerts.
Removing then adding the same paragraphs with content to the DOM

So far I haven't seen the bug. Heres the html

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>

  <style>

    .pdom-root {
      position: absolute;
      z-index: 1;
      opacity: 0.0001;
    }

    .pdom {
      top: 0px;
      left: 0px;
      width: 1px;
      height: 1px;

      position: fixed;
      pointer-events: none;
      transform-origin: left top 0px;
      border-width: 0px;
      padding: 1px 1px;
      margin: 0px;
      white-space: nowrap;
      font-size: 1px;
      clip: rect(1px, 1px, 1px, 1px);


    }

    .alerter {
      position: absolute;
      left: 0px;
      top: 0px;
      width: 0px;
      height: 0px;
      clip: rect(0px, 0px, 0px, 0px);
      user-select: none;
    }
  </style>
</head>
<body>

<p class="alerter" aria-live="assertive" id="alerter"></p>

<div class="pdom-root">
  <article class="pdom">
    <div id="parent-element">
      <h1 class="pdom">My Heading</h1>
      <p class="pdom">Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control
        Area.</p>
      <p id='summary-paragraph' class="pdom">Chemistry book has far fewer jiggling atoms as many have broken away.
        Chemistry book rests lightly
        on
        top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface
        temperature is cool, and atoms jiggle a tiny bit.</p>
      <p id="hint-paragraph" class="pdom">Reset sim to make more observations.</p>
      <p class="pdom">If needed, check out keyboard shortcuts under Sim Resources.</p>
    </div>
  </article>
</div>


<button id="alert-button">Toggle alerting</button>

<script>
  const alertButton = document.getElementById( 'alert-button' );
  const alerter = document.getElementById( 'alerter' );

  const parentElement = document.getElementById( 'parent-element' );

  let running = false;

  const summaryContent = [
    'Chemistry book is ready to move!',
    'Chemistry book is on top of physics book.',
    'Chemistry book has all of its particles.',
    'Chemistry book has no more particles.'
  ];

  const hintContent = [
    'Reset sim to make more observations',
    'Keep moving the book!'
  ];

  alertButton.addEventListener( 'click', event => {
    running = !running;
  } );

  let i = 0;
  window.setInterval( () => {
    console.log( 'here 2' );
    if ( running ) {
      alerter.innerHTML = `This is a new alert ${i++}`;

      let summaryParagraph = document.getElementById( 'summary-paragraph' );
      let hintParagraph = document.getElementById( 'hint-paragraph' );

      parentElement.removeChild( summaryParagraph );
      parentElement.removeChild( hintParagraph );

      summaryParagraph = document.createElement( 'p' );
      hintParagraph = document.createElement( 'p' );

      summaryParagraph.id = 'summary-paragraph';
      hintParagraph.id = 'hint-paragraph'

      summaryParagraph.innerText = summaryContent[ i % summaryContent.length ];
      hintParagraph.innerText = hintContent[ i % hintContent.length ];

      parentElement.appendChild( summaryParagraph );
      parentElement.appendChild( hintParagraph );

      window.setTimeout( () => {
        alerter.innerHTML = '';
      }, 100 )
    }
  }, 200 );

</script>
</body>
</html>

I have tethered my ipad to my Mac and I can maniuplate the Friction PDOM in real time while VoiceOver runs. Ive been changing tags and removing attributes to see if anything impacts things. So far nothing has made a difference.

I was changing attributes and innerHTML wildly without any changes like this:

You can see the paragraph that is not being read highlighted in the DOM. Also some new paratraphs that I added to the DOM, VoiceOver is ignoring those too. But then I toggled "All Audio" off and suddenly VoiceOver is reading all the content again! ~~But maybe that is because the sim crashed, I got a "cannot read null" error in the dev tools.~~ No, I verified that turning off all audio makes issue go away. Why?

Once VoiceOver starts reading the paragraphs again, it is more difficult for it to break again but it does happen eventually. I removed all particles from the book ~3 times before the screen summary content became undescoverable again.

I have noticed a couple of times in the sim that the paragraph becomes undiscoverable when reading it gets interrupted by an assertive alert. I tried but have not been able to observe this in my vanilla HTML.

I tried removing the work of the "All Audio" toggle button and its aria-pressed state and still found that pressing that button makes the issue go away.

I found that if I wait for all alerts to be spoken (waiting until the "reset sim to make observations" alert in Friction) I do not see this bug. I think that is an important trigger for the bug.

I tried disabling sound in case that was causing this problem because pressing the audio button seems to fix this. But the behavior is the same. For some reason pressing the "All Audio" button makes the problem go away, even when the button does nothing.

I am now seeing that pressing the "Reset All" button makes the problem go away. Is it an activation event that fixes it? I just produced the problem, then double tapped on a book to pick it up and release it. It made the problem go away.

I disabled all alerts and I still see the problem. So it has nothing to do with aria-live.

I tried forcing a re-render of the DOM element in the PDOM by toggling it back and forth between a div and a p. No impact.

I tried in my example HTML changing the DOM content with very long strings in the paragraph content (~8 seconds to read) like in the sim. No impact.

I tried in my example HTML having the paragraph content update while I doing a custom drag and drop action. I pressed the "Toggle Alerting" button and then double-tap and held the button to start drag/drop events while the paragraph text content updated. No impact.

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>

  <style>

    .pdom-root {
      position: absolute;
      z-index: 1;
      opacity: 0.0001;
    }

    .pdom {
      top: 0px;
      left: 0px;
      width: 1px;
      height: 1px;

      pointer-events: none;
      user-select: none;
      -webkit-touch-callout: none; /* iOS Safari */
      -webkit-user-select: none; /* Safari */

      position: fixed;
      pointer-events: none;
      transform-origin: left top 0px;
      border-width: 0px;
      padding: 1px 1px;
      margin: 0px;
      white-space: nowrap;
      font-size: 1px;
      clip: rect(1px, 1px, 1px, 1px);


    }

    .alerter {
      position: absolute;
      left: 0px;
      top: 0px;
      width: 0px;
      height: 0px;
      clip: rect(0px, 0px, 0px, 0px);
      user-select: none;
    }
  </style>
</head>
<body>

<p class="alerter" aria-live="assertive" id="alerter"></p>

<div class="pdom-root">
  <article class="pdom">
    <div id="parent-element">
      <h1 class="pdom">My Heading</h1>
      <p class="pdom">Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control
        Area.</p>
      <p id='summary-paragraph' class="pdom">Chemistry book has far fewer jiggling atoms as many have broken away.
        Chemistry book rests lightly
        on
        top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface
        temperature is cool, and atoms jiggle a tiny bit.</p>
      <p id="hint-paragraph" class="pdom">Reset sim to make more observations.</p>
      <p class="pdom">If needed, check out keyboard shortcuts under Sim Resources.</p>
    </div>
  </article>
</div>


<button class="pdom" id="alert-button">Toggle alerting</button>

<script>
  const alertButton = document.getElementById( 'alert-button' );
  const alerter = document.getElementById( 'alerter' );

  const parentElement = document.getElementById( 'parent-element' );

  let running = false;

  const summaryContent = [
    '1 Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.',
    '2 Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.',
    '3 Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.',
    '4 Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.',
  ];

  const hintContent = [
    'Reset sim to make more observations',
    'Keep moving the book!'
  ];

  alertButton.addEventListener( 'click', event => {
    running = !running;
  } );

  let i = 0;
  window.setInterval( () => {
    if ( running ) {
      // alerter.innerHTML = `This is a new alert ${i++}`;

      let summaryParagraph = document.getElementById( 'summary-paragraph' );
      let hintParagraph = document.getElementById( 'hint-paragraph' );

      // parentElement.removeChild( summaryParagraph );
      // parentElement.removeChild( hintParagraph );

      // summaryParagraph = document.createElement( 'p' );
      // hintParagraph = document.createElement( 'p' );
      //
      // summaryParagraph.id = 'summary-paragraph';
      // hintParagraph.id = 'hint-paragraph'
      //
      console.log( 'changing content' );
      i++;
      summaryParagraph.innerHTML = summaryContent[ i % summaryContent.length ];
      // hintParagraph.innerHTML = hintContent[ i % hintContent.length ];
      //
      // parentElement.appendChild( summaryParagraph );
      // parentElement.appendChild( hintParagraph );

      window.setTimeout( () => {
        // alerter.innerHTML = '';
      }, 100 )
    }
  }, 5000 );

</script>
</body>
</html>

I tried testing with a full copy of the PDOM in Friction, and adding a script to update the content of the summary paragraph every 5 seconds. I tested with drag/drop input. The bug did not appear.

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>
</head>
<body>
<div class="a11y-pdom-root" lang="en">
  <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-353" id="display1-primary-415-429-353"
       style="top: 0px; left: 0px; width: 1px; height: 1px;"><h2 data-focusable="false" class="a11y-pdom-element"
                                                                 data-unique-id="415-429-353-414-355"
                                                                 id="display1-label-415-429-353-414-355" hidden=""
                                                                 style="top: 0px; left: 0px; width: 1px; height: 1px;">
    Toolbar</h2>
    <section data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-353-414-355"
             id="display1-primary-415-429-353-414-355" hidden="" style="top: 0px; left: 0px; width: 1px; height: 1px;">
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-353-414-355-364-365"
              id="display1-primary-415-429-353-414-355-364-365" aria-pressed="true" aria-checked="true" role="switch"
              hidden="" style="top: 0px; left: 0px; width: 1px; height: 1px;">Sim Voicing
      </button>
      <p data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-353-414-355-357"
         id="display1-primary-415-429-353-414-355-357" hidden="" style="top: 0px; left: 0px; width: 1px; height: 1px;">
        Quick Info</p>
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-353-414-355-381-380-375"
              id="display1-primary-415-429-353-414-355-381-380-375" hidden=""
              style="top: 0px; left: 0px; width: 1px; height: 1px;">Play Overview
      </button>
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-353-414-355-395-394-389"
              id="display1-primary-415-429-353-414-355-395-394-389" hidden=""
              style="top: 0px; left: 0px; width: 1px; height: 1px;">Play Details
      </button>
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-353-414-355-409-408-403"
              id="display1-primary-415-429-353-414-355-409-408-403" hidden=""
              style="top: 0px; left: 0px; width: 1px; height: 1px;">Play Hint
      </button>
    </section>
    <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-353-414-412"
            id="display1-primary-415-429-353-414-412" hidden="" style="top: 0px; left: 0px; width: 1px; height: 1px;">
      Close Toolbar
    </button>
  </div>
  <article data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520"
           id="display1-container-415-429-520">
    <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520" id="display1-primary-415-429-520"
         style="top: 0px; left: 0px; width: 1px; height: 1px;"><h1 data-focusable="false" class="a11y-pdom-element"
                                                                   data-unique-id="415-429-520-527-521"
                                                                   id="display1-primary-415-429-520-527-521"
                                                                   style="top: 0px; left: 0px; width: 1px; height: 1px;">
      Friction</h1>
      <p data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-524-525"
         id="display1-primary-415-429-520-527-524-525" style="top: 0px; left: 0px; width: 1px; height: 1px;">Friction is
        an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.</p>
      <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-524-519-528"
           id="display1-primary-415-429-520-527-524-519-528" style="top: 0px; left: 0px; width: 1px; height: 1px;"><p
          data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-524-519-528-529"
          id="display1-primary-415-429-520-527-524-519-528-529" style="top: 0px; left: 0px; width: 1px; height: 1px;">
        Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a
        Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature
        is cool, and atoms jiggle a tiny bit.</p>
        <p data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-524-519-528-530"
           id="display1-primary-415-429-520-527-524-519-528-530" style="top: 0px; left: 0px; width: 1px; height: 1px;">
          Reset sim to make more observations.</p></div>
      <p data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-524-526"
         id="display1-primary-415-429-520-527-524-526" style="top: 0px; left: 0px; width: 1px; height: 1px;">If needed,
        check out keyboard shortcuts under Sim Resources.</p>
      <section data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-522"
               id="display1-container-415-429-520-527-522"><h2 data-focusable="false" class="a11y-pdom-element"
                                                               data-unique-id="415-429-520-527-522"
                                                               id="display1-label-415-429-520-527-522"
                                                               style="top: 0px; left: 0px; width: 1px; height: 1px;">
        Play Area</h2>
        <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-522"
             id="display1-primary-415-429-520-527-522" style="top: 0px; left: 0px; width: 1px; height: 1px;">
          <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-522-546"
               id="display1-container-415-429-520-527-522-546">
            <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-520-527-522-546"
                    id="display1-primary-415-429-520-527-522-546" aria-label="Grab Chemistry book"
                    style="top: 563.124px; left: 117.363px; width: 250.675px; height: 83.1797px;">Grab Chemistry book
            </button>
            <p data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-522-546"
               id="display1-description-415-429-520-527-522-546">Look for grab buttons. Once grabbed, use keyboard
              shortcuts to move book or zoomed-in book.</p></div>
          <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-522-595-596-639-641"
               id="display1-container-415-429-520-527-522-595-596-639-641">
            <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-520-527-522-595-596-639-641"
                    id="display1-primary-415-429-520-527-522-595-596-639-641" aria-label="Grab zoomed-in Chemistry book"
                    aria-roledescription="button" style="top: 3.38801px; left: 0px; width: 702px; height: 358.77px;">
              Grab zoomed-in Chemistry book
            </button>
          </div>
        </div>
      </section>
      <section data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-523"
               id="display1-container-415-429-520-527-523"><h2 data-focusable="false" class="a11y-pdom-element"
                                                               data-unique-id="415-429-520-527-523"
                                                               id="display1-label-415-429-520-527-523"
                                                               style="top: 0px; left: 0px; width: 1px; height: 1px;">
        Control Area</h2>
        <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-520-527-523"
             id="display1-primary-415-429-520-527-523" style="top: 0px; left: 0px; width: 1px; height: 1px;">
          <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-520-527-523-800"
                  id="display1-primary-415-429-520-527-523-800" style="top: 0px; left: 0px; width: 1px; height: 1px;">
            Reset All
          </button>
        </div>
      </section>
    </div>
  </article>
  <section data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-430-518"
           id="display1-container-415-429-430-518" aria-labelledby="display1-label-415-429-430-518"><h2
      data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-430-518"
      id="display1-label-415-429-430-518" style="top: 0px; left: 0px; width: 1px; height: 1px;">Sim Resources</h2>
    <div data-focusable="false" class="a11y-pdom-element" data-unique-id="415-429-430-518"
         id="display1-primary-415-429-430-518" style="top: 0px; left: 0px; width: 1px; height: 1px;">
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-430-518-517-493"
              id="display1-primary-415-429-430-518-517-493" aria-haspopup="true"
              style="top: 938.497px; left: 499.752px; width: 29.25px; height: 27.4238px;">Preferences
      </button>
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-430-518-517-507"
              id="display1-primary-415-429-430-518-517-507" aria-pressed="true"
              style="top: 0px; left: 0px; width: 1px; height: 1px;">All Audio
      </button>
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-430-518-517-515"
              id="display1-primary-415-429-430-518-517-515" style="top: 0px; left: 0px; width: 1px; height: 1px;">
        Keyboard Shortcuts
      </button>
      <button data-focusable="true" class="a11y-pdom-element" data-unique-id="415-429-430-518-485"
              id="display1-primary-415-429-430-518-485" aria-haspopup="true"
              style="top: 935.417px; left: 603.793px; width: 89.0662px; height: 33.5827px;">PhET Menu
      </button>
    </div>
  </section>
</div>
</body>

<script>
  const element = document.getElementById( 'display1-primary-415-429-520-527-524-519-528-529' );

  let i = 0;
  window.setInterval( () => {
    console.log( 'updadting' );
    const newContent = `${i++} Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.`

    element.innerHTML = newContent;
  }, 5000 );
</script>
</html>

The text was updated successfully, but these errors were encountered:

jessegreenberg · 2022-07-28T23:56:28Z

I have exhausted just about everything I can think of here and ready to recommend we list this as one of the known AT bugs.

jessegreenberg · 2022-07-29T14:04:44Z

@terracoda suggested running the HTML through a validator to see if it reports anything amiss.

EDIT: I ran the validator, no errors. Just a few warnings that seem unrelated.

EDIT: "View page source" gave me incomplete HTML, copying HTML from the dev tools showed me these errors from the validator". HTML copied was when the PDOM is in the state that VO skips content:

Both are attributes on the "Sim Voicing" button in the Toolbar, very unlikely to be causing this.

@terracoda also suggested that maybe unicode marks are causing this. We have seen unicode interfere with AT output in the past. I looked for unicode or other artifacts in the skipped content and didn't see any. My understanding is that unicode marks are added in built versions only and we are seeing this in both built and unbuilt versions

jessegreenberg · 2022-07-29T15:32:30Z

@terracoda recommended testing without any CSS, that was a good idea. Here is a test case that I set up by

Disabling all PDOM CSS.
Disabling all PDOM element positioning.
Using the sim to put it in the VO broken state.
Tethering my iPad to my Mac and removing all graphical elements with dev tools.
Reading through paragraph content with VoiceOver.

The problem STILL happens even when there is only the unstyled PDOM left.

Here is the result:

Untitled.video.-.Made.with.Clipchamp.mp4

Once graphics are removed, I swipe with VoiceOver to try to read all paragraphs. You can see content skipped. Near the end of the video I tap on the skipped paragraph with my finger and that makes the problem go away.

This could get us closer to identifying what is causing the issue if we want to continue.

jessegreenberg · 2022-07-29T20:17:02Z

@zepumph and I met to try to isolate the cause of this. We returned to the example HTML pages and found that this IS happening outside of PhET sims. I don't know why I wasn't seeing it when testing alone, maybe I failed to clear caches, or HTML updates happened to slowly.

Anyway, here is the most simple breaking case:

<!DOCTYPE html>
<html>
<body>

<h1>My Heading</h1>
<p>Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.</p>
<p id='summary-paragraph'>Summary HERE!</p>
<p>Reset sim to make more observations.</p>
<p>If needed, check out keyboard shortcuts under Sim Resources.</p>

<script>

  let i = 0;
  window.setInterval( () => {
    let summaryParagraph = document.getElementById( 'summary-paragraph' );
    summaryParagraph.textContent = ` ${i++} Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.`;
  }, 10000 );

</script>
</body>
</html>

We found that paragraph will become undiscoverable with iOS VoiceOver whenever its textContent changes. Maybe VoiceOver assumes that paragraphs are static? That doesn't explain why paragraphs become totally undiscoverable. But we though attributes role='status' aria-live="off" were the most correct way to tell the AT that the paragraph is going to change. But adding this did not fix the problem.

But we did find a workaround that DOES fix it: adding role="text" to the paragraph element. We got the idea from https://stackoverflow.com/questions/23526161/for-ios-safari-voiceover-how-do-i-get-voiceover-to-read-something-other-than.

text is not even a real role, it is not clear why this works. It feels very hacky. And we would need to build setting of this attribute into scenery/ParallelDOM.ts somehow so it is set for all paragraphs.

jessegreenberg · 2022-07-29T20:24:17Z

We wanted to run our findings by @terracoda and @emily-phet to help decide where to go from here. We did find a workaround but do not recommend using it. This is clearly an iOS problem and the workaround is totally hacky. It seems to work now but since it is outside of any DOM specification it could also stop working any time. If we proceed with it, we will do work in scenery to make sure that that all p elements get a role="text" attribute when they are rendered. Do you have any thoughts about this?

emily-phet · 2022-07-29T20:49:09Z

@jessegreenberg I think we shouldn't implement the workaround - but great that's it's more clear what the root of the issue is.

zepumph · 2022-07-29T21:02:41Z

But we though attributes role='status' aria-live="off" were the most correct way to tell the AT that the paragraph is going to change. But adding this did not fix the problem.

I think it would be nice to have a non-blocking conversation about adding these to our screen summary paragraphs. It would be a nice improvement in my mind to convey that this is not just a paragraph, but a dynamic text space that updates with the sim's updates. I defer to the A11y designers though.

terracoda · 2022-07-30T11:07:18Z

Excellent detective work.
@jessegreenberg, you won’t believe this, but I almost posted that the changing content could be the reason in slack. This was the first sim where I thought it would be nice to trace/track the rising and falling results of making or not making friction, even though it is difficult to catch live while changing.

This might also be an issue for MAL’s observation window, and GHE might have a dynamic Scene Summary similar in structure to the summary in Friction.

There are a couple of design things we should seriously discuss and consider for the long run:

Ask, is describing a changed state (clarification: in an actual State Description) as it happens super fast essential for learning? (At design time I thought it was, maybe it is not).
Can the change be captured differently. For example, in John Travoltage's scene summary, we describe the discharge event in the past tense and note the change in the amount of charge.
And I have one more technical question…Do list structures work any differently than paragraph structures?

I agree with @emily-phet that we should not implement the hack. It’s not a long term solution.

I do feel not having access to the scene summary in Friction - its single dynamic state description - is a bit of a problem which I think can be resolved with a small amount of re-wording and a reduction in parameters- the temperature is either dropping or already at cool.

In retrospect, apologies for designing such accurate state descriptions that screen reader software can’t handle them 😀

At least not as of July 2022.

jessegreenberg · 2022-08-01T13:26:37Z

Ok, thanks @emily-phet and @terracoda! I'll write a bug report to Apple about this just in case.

I'll also add this to the list of known AT problems.

jessegreenberg · 2022-08-30T21:16:40Z

Bug report submitted to apple, we will see if they respond.

jessegreenberg · 2022-08-31T19:03:54Z

@terracoda shared that there is a special address that Apple has for critical accessibility problems, Ill try and submit this issue there as well.

terracoda · 2023-09-01T14:53:38Z

I am running iOS 15.7.8 on my iPhone 7 and I still seem to experience this issue.

I hear the description for the Chemistry book on sim load, but then after interaction I can't seem to find that description with horizontal or vertical swipes using iOS VoiceOver.

terracoda · 2023-09-01T14:55:01Z

I can however grab and rub the books, so the custom grab is now working for Friction. What issue is related to that bug?

jessegreenberg self-assigned this Jul 28, 2022

jessegreenberg mentioned this issue Jul 28, 2022

Mobile VO isn't consistently reading alerts phetsims/friction#294

Closed

jessegreenberg mentioned this issue Jul 29, 2022

Full scene description isn't read through by swiping on Mobile VO phetsims/friction#292

Closed

zepumph self-assigned this Jul 29, 2022

jessegreenberg assigned emily-phet and terracoda Jul 29, 2022

emily-phet removed their assignment Jul 29, 2022

zepumph removed their assignment Jul 29, 2022

jessegreenberg added the a11y:platform-bug label Aug 1, 2022

This was referenced Aug 1, 2022

Mobile VO skips over part of scene description sometimes phetsims/john-travoltage#449

Closed

Alerts, Energy Balance section and other bullets may not be read in Mobile VO phetsims/greenhouse-effect#195

Closed

jessegreenberg unassigned terracoda Aug 1, 2022

jessegreenberg added a commit to phetsims/qa that referenced this issue Aug 1, 2022

Make note of phetsims/a11y-research#175

8c17733

This was referenced Aug 5, 2022

Investigate adding role='status' aria-live="off" to our screen summary content #176

Open

VO repeats descriptions in Wrappers phetsims/friction#286

Closed

jessegreenberg added the a11y:report-submitted label Aug 30, 2022

jessegreenberg removed their assignment Aug 30, 2022

jessegreenberg self-assigned this Aug 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mobile VO skips content in the PDOM #175

Mobile VO skips content in the PDOM #175

jessegreenberg commented Jul 28, 2022 •

edited

Loading

jessegreenberg commented Jul 28, 2022

jessegreenberg commented Jul 29, 2022 •

edited

Loading

jessegreenberg commented Jul 29, 2022 •

edited

Loading

jessegreenberg commented Jul 29, 2022

jessegreenberg commented Jul 29, 2022

emily-phet commented Jul 29, 2022

zepumph commented Jul 29, 2022

terracoda commented Jul 30, 2022 •

edited

Loading

jessegreenberg commented Aug 1, 2022 •

edited

Loading

jessegreenberg commented Aug 30, 2022

jessegreenberg commented Aug 31, 2022

terracoda commented Sep 1, 2023

terracoda commented Sep 1, 2023

Mobile VO skips content in the PDOM #175

Mobile VO skips content in the PDOM #175

Comments

jessegreenberg commented Jul 28, 2022 • edited Loading

jessegreenberg commented Jul 28, 2022

jessegreenberg commented Jul 29, 2022 • edited Loading

jessegreenberg commented Jul 29, 2022 • edited Loading

jessegreenberg commented Jul 29, 2022

jessegreenberg commented Jul 29, 2022

emily-phet commented Jul 29, 2022

zepumph commented Jul 29, 2022

terracoda commented Jul 30, 2022 • edited Loading

jessegreenberg commented Aug 1, 2022 • edited Loading

jessegreenberg commented Aug 30, 2022

jessegreenberg commented Aug 31, 2022

terracoda commented Sep 1, 2023

terracoda commented Sep 1, 2023

jessegreenberg commented Jul 28, 2022 •

edited

Loading

jessegreenberg commented Jul 29, 2022 •

edited

Loading

jessegreenberg commented Jul 29, 2022 •

edited

Loading

terracoda commented Jul 30, 2022 •

edited

Loading

jessegreenberg commented Aug 1, 2022 •

edited

Loading