Consistency report for ARIA AT App bots

[ccanash]

Bocoup is publishing consistency reports for the bots in the aria-at-app. The consistency reports can be found here: https://github.com/bocoup/aria-at-gh-actions-helper/wiki/20250403.consistency-report .

We are currently working on improving the interface.

[css-meeting-bot]

The ARIA-AT Community Group just discussed App issue 1365 - Bot reliability reporting.

The full IRC log of that discussion

<jugglinmike> Topic: App issue 1365 - Bot reliability reporting
<jugglinmike> github: https://github.com//issues/1365
<jugglinmike> Matt_King: Bocoup has come up with a testing methodology to test the reliability of the bots
<jugglinmike> Matt_King: I included a link to the report in the agenda
<jugglinmike> s/the report/a recent report/
<jugglinmike> Carmen: We are testing consistency by running each test five times and determining whether there were different responses in each "Trial"
<jugglinmike> Carmen: You will see that NVDA is over 99% consistent. Our focus this year is on VoiceOver--it is currently at 91%, and we would like to raise it to at least 95%
<jugglinmike> ChrisCuellar: Right now, it is triggered completely manually
<jugglinmike> ChrisCuellar: We would like to run it as a service on a regular schedule
<jugglinmike> Matt_King: I wonder if this will change with different versions of screen readers and bots
<jugglinmike> ChrisCuellar: Now that we're letting the CG know about the reports, they are almost like a feature that we can iterate on and that we can improve in response to external input
<jugglinmike> Matt_King: Okay, this is great. I'm really happy to have these metrics!
<jugglinmike> Zakim, end the meeting