How do you handle difficult-to-reproduce accessibility rule discrepancies in Alfa?

[Veljko157]

Hi,

I’m a developer and PhD student working in web accessibility, and I’m trying to better understand the debugging workflow used by accessibility testing engine maintainers.

Suppose Alfa produces one result for a piece of real-world web content, while another accessibility testing engine produces a different result for the same or apparently equivalent ACT rule.

How would you normally investigate such a discrepancy?

I’m particularly interested in two aspects:

• Do you typically need the reporter or maintainer to create a small/minimal HTML, CSS or ARIA reproduction before the issue can be investigated effectively?
• Is reducing a complex real-world page to such a reproduction a meaningful manual part of the debugging process?

I’m also exploring whether tooling that automatically removes unrelated page content while repeatedly verifying that the original cross-engine disagreement is still preserved could be useful.

For such an automatically reduced reproduction to be trustworthy, what would need to remain unchanged or be explicitly recorded? For example: ACT applicability, DOM/ARIA relationships, browser and Alfa versions, viewport, CSS state, or JavaScript-generated DOM state.

I’m not trying to determine automatically which engine is correct. I’m specifically interested in producing a small, stable and reproducible artifact that makes implementation differences easier to investigate.

If Alfa already has a workflow or tooling that solves this problem well, I would also be very interested in learning about it.

Thank you for any insight.

[jmrplens]

Not a maintainer, so I cannot speak for the project's workflow. I can answer the part your tooling question actually turns on, which is what an Alfa result is a function of, and I checked all of it against Alfa 0.119.0 rather than reasoning about it.

Alfa does not audit HTML, so HTML is the wrong thing to reduce

A rule in Alfa evaluates a Page, and a Page is {request, response, document, device} (alfa-web/src/page.ts at v0.119.0). It is json.Serializable and it round-trips: Page.toJSON() gives you the four keys, and Page.from(json) gives you back a page that evaluates identically. I ran the same rule on a page and on its serialized-then-deserialized copy and got the same outcome.

That matters for your design. If a reducer edits HTML and re-runs, each iteration re-scrapes, so the DOM and the computed styles are re-derived every time and you are not holding Alfa's input fixed. If it reduces the serialized Page instead, the input is fixed by construction and the artifact you ship is already the thing Alfa consumed.

The device half of that answers most of your explicit list on its own. Device.standard() (alfa-device/src/device.ts) serializes to:

{"type":"screen","viewport":{"width":1280,"height":720,"orientation":"landscape"},
 "display":{"resolution":1,"scan":"progressive"},"scripting":{"enabled":true},"preferences":[]}

so viewport, resolution, whether scripting was on, and media preferences such as prefers-reduced-motion or forced-colors are already recorded fields rather than things you have to remember to note. The DOM in that JSON is the post-JavaScript DOM, so "JS-generated DOM state" is captured too. What is not in the artifact is the version pair: the rules version is exported as alfaVersion (0.119.0 here), and the snapshot producer is versioned separately, @siteimprove/alfa-scraper is on 0.84.2 while @siteimprove/alfa-rules is on 0.119.0, so both belong in whatever you record.

Two ways an automatic reduction lies, measured

Both of these are the kind of content a reducer would drop as unrelated.

Name resolution reaches across the tree. A button whose aria-labelledby points at a <span> living in the footer, checked with R12:

R12, full page              -> ["passed"]
R12, footer removed         -> ["failed"]

Applicability, not just the verdict. A button inside a wrapper that is hidden from the accessibility tree. The wrapper looks like empty decoration, and removing it does not change a passed to a failed, it changes whether the rule applies at all:

R12, with the aria-hidden div   -> ["inapplicable"]
R12, div removed                -> ["failed"]

The part that bears on your invariant

I ran axe-core 4.13.0 on the same two documents. It flips exactly the same way: button-name passes on the full page and violates on the reduced one. So this is not a cross-engine artifact, the reduction changed the accessibility semantics and both engines correctly reported the new semantics.

That is the awkward case for "keep reducing while the disagreement is preserved". A reduction can preserve a disagreement and still have moved both engines off the thing you were investigating, so preservation of the disagreement is necessary but not sufficient. If you want an invariant that catches this, the ones worth pinning are per-target rather than per-page: the target element's identity, its computed role and accessible name, and the rule's applicability verdict on it. If applicability flips, or the accessible name changes, the reduction is not a smaller version of the same question even when both engines still disagree.

For what it is worth, the same reasoning suggests the reduced artifact should be checked against the original by re-evaluating the specific rule and comparing outcome plus applicability on the same target, not by comparing whole-page result sets, since a page-level diff hides a target that stopped being applicable.

[Veljko157]

Thank you, this is extremely helpful, particularly the distinction between preserving a cross-engine disagreement and preserving the actual accessibility question being evaluated.

Your point about reducing the serialized Alfa Page rather than repeatedly modifying and re-scraping HTML changes how I’m thinking about the design.

One follow-up question: if the goal is cross-engine differential debugging, would you consider a useful invariant to be something like:

• the same target identity,
• the same computed role,
• the same accessible name,
• unchanged rule applicability for that target,
• and the same normalized cross-engine outcome?

I’m also wondering about the boundary between the engine-specific representation and an engine-neutral reproduction artifact. Since Alfa can consume its serialized Page, while another engine such as axe-core normally consumes a browser DOM, would you preserve the Alfa Page as one engine-specific snapshot inside a broader reproduction bundle, rather than treating that snapshot itself as the common representation?

Thank you again, the examples you provided exposed an important flaw in a naive HTML-reduction approach.

[jmrplens]

Both good questions. The first one I would sharpen slightly, the second one I went and measured, because it turns out the symmetry is worse than it looks.

On the invariant set

Your list is the right shape, and adding the outcome is the right instinct, but I would pin it per engine rather than as a relation. "The disagreement is preserved" is satisfied when both engines flip together, which is exactly what my footer example did: removing the footer moved Alfa from passed to failed and moved axe-core the same way, so any relation-level invariant survived while the question changed underneath it. Pinning each engine's own outcome catches that; pinning the relation does not.

I would also keep applicability as its own axis rather than folding it into the outcome. The second example flipped inapplicable to failed, and a comparison that only diffs pass and fail sets sees "something changed" without distinguishing "this target stopped being in scope" from "this target now fails". For a reduction tool those two need different handling: the first means the reduction is invalid, the second might be the bug you are hunting.

One thing to add to the list, since it is free in Alfa's model: the device. A Page carries viewport, resolution, whether scripting was on, and media preferences, and a reduction that removes what looks like unrelated CSS can change computed styles under an unchanged DOM if it touches a media query the device matches. Holding the device fixed and recorded is cheap and removes a whole class of false reductions.

The limitation I would flag in whatever you write up: a per-target invariant cannot express rules whose applicability is defined over sets of elements. Uniqueness of an id, heading order, "the only main landmark", "duplicate accessible names among siblings". For those, "same target, same role, same name" is preserved by a reduction that nonetheless changes the answer, because the rule's subject is the set, not the element. Either exclude that family from the automated path or give it a structural invariant of its own.

On the boundary, which is the more interesting question

Yes, keep the Alfa Page as an engine-specific snapshot inside a bundle, and do not promote it to the common representation. But I would resist the symmetric assumption too, that HTML is the neutral artifact and only Alfa is the special case. It is not.

I ran this to check. A button with no accessible name inside an open shadow root, then the same DOM captured as outerHTML and reloaded without re-running the scripts, which is what a snapshot is for:

live page (button inside shadow DOM)                 violations=["button-name"]
post-JS outerHTML snapshot, scripts not re-run       violations=[]  inapplicable=["button-name"]

The host element serialises as <div id="host"></div> and the finding disappears. So the obvious artifact for axe-core silently drops results too. It is just less visible than Alfa's case, because nothing warns you that your snapshot is lossy, whereas with Alfa the need for a Page is explicit.

Worth adding a trap I hit while measuring, in case you build this: if you reload a captured DOM in a browser without stripping the inline scripts, they re-execute and rebuild exactly the state you were trying to capture, so the snapshot appears faithful when it is not. My first two attempts said "identical" for that reason.

My first instinct was to tell you to capture with getHTML({ serializableShadowRoots: true }) instead of outerHTML, so I checked that too, and it only half works:

attachShadow serializable:false   capture keeps the button: false   live=["button-name"] capture=[]              CHANGES
attachShadow serializable:true    capture keeps the button: true    live=["button-name"] capture=["button-name"] MATCHES

serializable defaults to false, and it is a choice the page author makes, not the person debugging it. So on somebody else's real-world page you generally cannot capture the shadow DOM by serialisation at all, whichever API you reach for. For those pages the honest artifact for axe-core is either the live page plus whatever builds it, or a capture that walks shadow roots explicitly rather than relying on the serialiser.

So the shape I would suggest is a bundle whose contract is per engine: for each engine, an artifact that engine can consume without re-deriving anything, plus the environment identity that engine's result actually depends on. For Alfa that is the Page JSON plus alfaVersion. For axe-core it is a DOM capture that genuinely preserves shadow content, with the caveat above, plus the browser build and viewport, since axe's answer is a function of those the same way Alfa's is a function of its device.

The neutral representation is then the bundle and its metadata, not any one serialization. Which is a less elegant answer than a single common format, but I think a single format is unreachable here: the engines do not consume the same thing, and the differences between what they consume are exactly where the disagreements you are studying come from.

[Veljko157]

Thank you, this was extremely useful, especially the distinction between preserving a relation between engines and preserving each engine’s own outcome and applicability.

Based on your feedback, I changed the prototype in a few concrete ways:

• the reducer now treats each engine’s outcome as a separate invariant rather than only preserving the disagreement relation;
• applicability is tracked separately;
• the target fingerprint includes role, accessible name and accessibility-tree inclusion;
• structural invariants can be added for rule families where the subject is a set rather than a single element;
• the reproduction format is moving toward a bundle-level contract with engine-specific artifacts, rather than trying to treat HTML or Alfa’s Page as a universal representation.

I also ran a small benchmark after the previous discussion. On one real Alfa/axe-core disagreement, both naive and semantic reduction preserved the same PASS | FAIL result. The naive reducer reduced the case from 28 to 13 elements but changed the target’s accessibility fingerprint. The semantic reducer stopped at 14 elements, preserved the fingerprint, and rejected two otherwise acceptable reductions.

Your Shadow DOM example also exposed another important gap in my current prototype. I had been treating browser DOM serialization as a much safer neutral artifact than it actually is. I am going to add a specific experiment comparing the live page, a normal DOM snapshot, and an explicit shadow-aware capture, including the script re-execution trap you mentioned.

I also agree with your point about set-based rules. I do not think it would be responsible to claim that one per-target invariant model can cover all ACT rule families. I am leaning toward explicitly classifying supported rule families and giving structural/set-based rules their own invariants.

Thank you again for taking the time to actually test these cases. It has materially changed the technical direction of the prototype.

[jmrplens]

That benchmark is the interesting part, and I would not bury it: your naive reducer got to 13 elements and your semantic one stopped at 14. By the only metric a reduction tool naturally reports, the naive one won, and it won while having broken the target's fingerprint. That asymmetry is worth stating explicitly wherever you write this up, because any tool that reports size alone will always look better than a correct one, and the single element of difference is what the correctness cost.

The two reductions your semantic reducer rejected are also worth keeping rather than discarding. They are the cheapest evidence you have of where the semantic boundary actually sits, and a rejected-reduction log will probably explain more about a rule family than the accepted ones do.

Good luck with it. If this answered what you came for, marking it as the answer would help, mostly so the thread does not read as unresolved to the next person arriving with the same question.

[Veljko157]

Thank you, that distinction between reduction size and reduction correctness is particularly useful.

I agree that the rejected reductions are evidence rather than noise. I’ll keep them in the benchmark output together with the semantic reason for rejection, rather than only reporting the final reduced size.

The 13-vs-14 result is also a useful way to frame the trade-off: the naive reducer achieved a slightly smaller case, but only by crossing the semantic boundary, while the semantic reducer paid a one-element cost to preserve the target fingerprint.

I’ll mark this as the answer. Thanks again for taking the time to test and explain all of this - it materially improved both the prototype and how I’m evaluating it.