Accessibility conformance results for React Spectrum, ahead of publication (all checks pass, and a thank you)

[amansoomro062]

This is not the usual issue, so thank you for reading this far. These are accessibility conformance results from Railing, an open source project that tests React component libraries against the W3C ARIA Authoring Practices Guide and publishes the results. React Spectrum is one of the seven libraries measured, and I wanted you to hear the results from me before anything is public.

The short version: React Spectrum passes every check we run. 64 assertions across five components, each citing an APG clause or WCAG success criterion, run three times and discarded if any result varies.

Component	Result	Against
Dialog	13/13	@adobe/react-spectrum@3.47.3
Combobox	14/14	@adobe/react-spectrum@3.47.3
Menu	13/13	@adobe/react-spectrum@3.47.3
Tabs	13/13	@adobe/react-spectrum@3.47.3
Accordion	11/11	@adobe/react-spectrum@3.47.3

There is something you should know beyond the numbers: React Spectrum is the calibration control for the whole index. Every new assertion is run against your library first, and a failure there is treated as evidence that our assertion is wrong until proven otherwise. That role exists because of the reputation your team's accessibility work has earned; several of our early runner bugs were caught precisely because "Spectrum failed" was more plausibly our fault than yours, and it always was. The full report and the adapter we used are below so you can check the role is deserved and the mounting is fair.

Components were mounted with default configuration, following your documentation, with no focus management, ARIA attributes or key handlers added by our adapter.

Why tell you at all? Three reasons:

1. You should hear about a public score from us rather than come across it. No score for React Spectrum is on our index yet, and none will be for at least fourteen days.
2. You may still think we measured something wrongly, and if so, please let us know. We publish corrections in the open (decision log).
3. When the index goes live, React Spectrum will be on it at 100% of detected checks, described as the calibration control, with a caveat we print on every page: a high score means no violations were detected by automated testing, not that a component is accessible. If you would rather phrase your own caveat beside it, we publish responses in full and unedited.

How the measurement works: railing.dev/method · scoring · disclosure policy · source

🔦 Context

This is a courtesy notice ahead of publication rather than a problem report. The full report, including the adapter used to mount your components, is below.

Full report

Accessibility conformance results for React Spectrum

You are hearing this before we publish anything. No score for React Spectrum is on our index, and none will be for at least fourteen days, whatever this report says.

Railing runs component libraries against the W3C ARIA Authoring Practices Guide and publishes the results. Every check cites the clause it measures, every score names an exact version, and every result is run repeatedly and discarded if the answer changes.

The short version

Component	Score	Failing checks
accordion	100%	none
combobox	100%	none
dialog	100%	none
menu	100%	none
tabs	100%	none

We found nothing to report. React Spectrum passes every check we run. We are telling you anyway, because you should hear about a public score from us rather than find it, and because you may still think we measured something wrongly.

How we tested it

• Versions: @adobe/react-spectrum@3.47.3
• Browser: chromium 151.0.7922.34
• Configuration: default. We use only what the library exports and never hand-write ARIA, even where the documentation instructs the developer to. If we did, the score would measure how carefully we copied your docs rather than what ships in the box.
• Traces: every run above has a replayable Playwright trace, captured from exactly the run that was scored. Ask and we will send them, or regenerate them with the command below.

Reproduce any of it from a clone of the repository:

pnpm --filter @railing-dev/adapter-react-spectrum run dev
pnpm railing run --target react-spectrum --component <component> \
  --base-url http://localhost:5181 --repeat 3

Where we might be wrong

The most likely cause of a wrong result is our adapter, not your library.

We have got this wrong repeatedly. One library's first run scored 27% and almost all of it was a selector of ours. Another was reported as having a broken focus trap when the trap worked and our test was reading focus too early.

So the adapter source is included below. Please tell us if we mounted your component in a way you would not recommend, or if a check misreads the specification. We will correct it and, if it has already been published, correct that too.

Where to check us

• How the measurement works, including why the runner never learns which library it is testing.
• How scoring works, with every weighting and what is deliberately left out.
• Our disclosure policy, covering notice, right of reply, and conflicts of interest.
• The decision log, which is largely a record of results we got wrong and corrected.
• The source. Every adapter, spec and score, and the code that produced this report.

What happens next

• You have fourteen days before anything about this library is published.
• If you reply, your response is published beside the score, in full and unedited.
• If you ship a fix in that window, we publish the fixed score. A finding that gets fixed before publication is the best outcome this project has, not a story we lost.
• If fourteen days is not enough for your release process, ask for more and you will get it.

The adapter we used

This is the whole of the library-specific code. There is nothing else.

accordion.tsx

import { useEffect } from "react";
import {
  Accordion,
  defaultTheme,
  Disclosure,
  DisclosurePanel,
  DisclosureTitle,
  Provider,
} from "@adobe/react-spectrum";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * Adobe React Spectrum accordion: calibration control for this pattern.
 *
 * Spectrum's DisclosureTitle renders the heading and the button itself, so the
 * harness ids are stamped onto the buttons by index and onto the panels by
 * content. Markers only, as always.
 */
export function AccordionHarness() {
  useEffect(
    () =>
      stampTestIds({
        "hr-header-1": { selector: "button[aria-expanded]", index: 0 },
        "hr-header-2": { selector: "button[aria-expanded]", index: 1 },
        "hr-header-3": { selector: "button[aria-expanded]", index: 2 },
        // Spectrum exposes disclosure panels as role=group. The APG lists
        // role=region for accordion panels as optional, so this is a legitimate
        // choice and not a defect, it was our selector that was wrong.
        "hr-panel-1": { selector: "[role=group]", textIncludes: TEXT.accordionPanel1 },
        "hr-panel-2": { selector: "[role=group]", textIncludes: TEXT.accordionPanel2 },
        "hr-panel-3": { selector: "[role=group]", textIncludes: TEXT.accordionPanel3 },
      }),
    [],
  );

  return (
    <Provider theme={defaultTheme}>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <Accordion>
        <Disclosure id="shipping">
          <DisclosureTitle>{TEXT.accordionHeader1}</DisclosureTitle>
          <DisclosurePanel>{TEXT.accordionPanel1}</DisclosurePanel>
        </Disclosure>
        <Disclosure id="payment">
          <DisclosureTitle>{TEXT.accordionHeader2}</DisclosureTitle>
          <DisclosurePanel>{TEXT.accordionPanel2}</DisclosurePanel>
        </Disclosure>
        <Disclosure id="review">
          <DisclosureTitle>{TEXT.accordionHeader3}</DisclosureTitle>
          <DisclosurePanel>{TEXT.accordionPanel3}</DisclosurePanel>
        </Disclosure>
      </Accordion>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </Provider>
  );
}

combobox.tsx

import { useEffect } from "react";
import { ComboBox, defaultTheme, Item, Provider } from "@adobe/react-spectrum";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * Adobe React Spectrum combobox: the calibration control for this pattern.
 *
 * React Spectrum's ComboBox forwards `data-testid` to its wrapper rather than to
 * the `input[role="combobox"]` inside it, and the listbox and options are
 * portalled in only when the popup opens. So the harness ids are stamped onto
 * the real semantic elements by structural selector, see the note on
 * `stampTestIds`. That places markers and nothing else: no ARIA, no handlers,
 * no focus management.
 */
export function ComboboxHarness() {
  useEffect(
    () =>
      stampTestIds({
        "hr-combobox": 'input[role="combobox"]',
        "hr-listbox": '[role="listbox"]',
        "hr-option-1": { selector: '[role="option"]', index: 0 },
        "hr-option-2": { selector: '[role="option"]', index: 1 },
        "hr-option-3": { selector: '[role="option"]', index: 2 },
      }),
    [],
  );

  return (
    <Provider theme={defaultTheme}>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <ComboBox label={TEXT.comboboxLabel}>
        <Item key="apple">{TEXT.comboboxOption1}</Item>
        <Item key="banana">{TEXT.comboboxOption2}</Item>
        <Item key="cherry">{TEXT.comboboxOption3}</Item>
      </ComboBox>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </Provider>
  );
}

dialog.tsx

import {
  ActionButton,
  Button,
  ButtonGroup,
  Content,
  defaultTheme,
  Dialog,
  DialogTrigger,
  Heading,
  Provider,
} from "@adobe/react-spectrum";
import { TEXT } from "@railing-dev/harness-kit";

/**
 * Adobe React Spectrum modal dialog: the calibration control.
 *
 * This adapter is not here to measure React Spectrum. It is here to measure
 * *us*. React Spectrum is widely regarded as the accessibility gold standard,
 * so any assertion it fails is presumed to be over-strict or wrong until
 * demonstrated otherwise, and no result for any subject library may be
 * published until this one scores at least 95%.
 *
 * As with every adapter: nothing here manages focus, adds ARIA, or handles
 * keys. The two plain inputs are harness furniture, deliberately plain rather
 * than Spectrum TextFields, so that both adapters traverse an identical set of
 * three focusable elements and the focus-trap counts stay comparable.
 */
export function DialogHarness() {
  return (
    <Provider theme={defaultTheme}>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <DialogTrigger type="modal">
        <ActionButton data-testid="hr-trigger">{TEXT.dialogTrigger}</ActionButton>
        {(close) => (
          <Dialog data-testid="hr-dialog">
            <Heading data-testid="hr-title">{TEXT.dialogTitle}</Heading>
            <Content>
              <input data-testid="hr-field-1" type="text" aria-label="Field one" />
              <input data-testid="hr-field-2" type="text" aria-label="Field two" />
            </Content>
            <ButtonGroup>
              <Button variant="secondary" onPress={close} data-testid="hr-close">
                {TEXT.close}
              </Button>
            </ButtonGroup>
          </Dialog>
        )}
      </DialogTrigger>

      <button data-testid="hr-outside-content" type="button">
        {TEXT.outsideContent}
      </button>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </Provider>
  );
}

menu.tsx

import { useEffect } from "react";
import { ActionButton, defaultTheme, Item, Menu, MenuTrigger, Provider } from "@adobe/react-spectrum";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * Adobe React Spectrum menu button: calibration control for this pattern.
 *
 * The menu and its items are portalled in only when opened, and Spectrum does
 * not forward attributes to the elements carrying the roles, so ids are stamped
 * by structural selector. Markers only, no ARIA, no handlers.
 */
export function MenuHarness() {
  useEffect(
    () =>
      stampTestIds({
        "hr-menu": '[role="menu"]',
        "hr-item-1": { selector: '[role="menuitem"]', index: 0 },
        "hr-item-2": { selector: '[role="menuitem"]', index: 1 },
        "hr-item-3": { selector: '[role="menuitem"]', index: 2 },
      }),
    [],
  );

  return (
    <Provider theme={defaultTheme}>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <MenuTrigger>
        <ActionButton data-testid="hr-trigger">{TEXT.menuTrigger}</ActionButton>
        <Menu>
          <Item key="cut">{TEXT.menuItem1}</Item>
          <Item key="duplicate">{TEXT.menuItem2}</Item>
          <Item key="paste">{TEXT.menuItem3}</Item>
        </Menu>
      </MenuTrigger>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </Provider>
  );
}

tabs.tsx

import { useEffect } from "react";
import { defaultTheme, Item, Provider, TabList, TabPanels, Tabs } from "@adobe/react-spectrum";
import { stampTestIds, TEXT } from "@railing-dev/harness-kit";

/**
 * Adobe React Spectrum tabs: calibration control for this pattern.
 *
 * Spectrum renders only the selected panel, so `hr-panel-2` and `hr-panel-3`
 * will not exist until their tab is chosen. The protocol allows that: the spec
 * marks unselected panels as not required at load, and treats absence from the
 * DOM as a perfectly good way of being hidden from assistive technology.
 *
 * The single stamped panel id therefore tracks whichever panel is currently
 * rendered, which is why it is indexed rather than matched per panel.
 */
export function TabsHarness() {
  useEffect(
    () =>
      stampTestIds({
        "hr-tablist": '[role="tablist"]',
        "hr-tab-1": { selector: '[role="tab"]', index: 0 },
        "hr-tab-2": { selector: '[role="tab"]', index: 1 },
        "hr-tab-3": { selector: '[role="tab"]', index: 2 },
        // Matched by content, not index: only one panel is ever rendered, and
        // Spectrum reuses the same node for whichever one that is.
        "hr-panel-1": { selector: '[role="tabpanel"]', textIncludes: TEXT.panel1 },
        "hr-panel-2": { selector: '[role="tabpanel"]', textIncludes: TEXT.panel2 },
        "hr-panel-3": { selector: '[role="tabpanel"]', textIncludes: TEXT.panel3 },
      }),
    [],
  );

  return (
    <Provider theme={defaultTheme}>
      <button data-testid="hr-before" type="button">
        Before
      </button>

      <Tabs>
        <TabList>
          <Item key="one">{TEXT.tab1}</Item>
          <Item key="two">{TEXT.tab2}</Item>
          <Item key="three">{TEXT.tab3}</Item>
        </TabList>
        <TabPanels>
          <Item key="one">{TEXT.panel1}</Item>
          <Item key="two">{TEXT.panel2}</Item>
          <Item key="three">{TEXT.panel3}</Item>
        </TabPanels>
      </Tabs>

      <button data-testid="hr-after" type="button">
        After
      </button>
    </Provider>
  );
}

Thanks for setting the standard the rest of the index is measured against. Literally.

Aman

💻 Code Sample

Everything is reproducible from a clone of https://github.com/amansoomro062/railing, and every scored run has a replayable Playwright trace:

pnpm --filter @railing-dev/adapter-react-spectrum run dev
npx @railing-dev/runner run --target react-spectrum --component dialog \
  --base-url http://localhost:5181 --repeat 3

Version

3.47.3

What browsers are you seeing the problem on?

Other

If other, please specify

Chromium 151.0.7922.34 (Playwright)

What operating system are you using?

macOS