Four Bootstrap PRs in a public study of accessibility review — corrections welcome

[ecogetaway]

We've been studying how accessibility changes get reviewed across open source, and four Bootstrap pull requests are among seven case studies in a public corpus: https://github.com/ecogetaway/oss-accessibility-inclusion

Posting here because we'd rather you heard this from us than came across it, and because two of the Bootstrap-specific findings surprised us enough that they're worth putting in front of you directly.

The method, briefly

Each case is scored against a six-criterion rubric that was published before any scoring. Every per-criterion justification is public, so any score can be disputed. The rubric deliberately does not judge whether a fix was technically correct — it asks whether the review process could have verified the accessibility claim. Two of our own scores have already changed after re-verification, and those changes are recorded.

The four cases: #42500 (6/12), #42539 (6/12), #42524 (7/12), #41607 (4/12).

What Bootstrap does better than anyone else in the corpus

#42500 names four WCAG success criteria and reasons explicitly about two of them. That is the strongest standards reasoning anywhere in our set — better than the case that scored highest overall.

The single best moment in the whole study is also yours

On #42524, @coliff found a keyboard lockout on iPadOS that no source-code review could have caught, and the response was: could you re-test once the preview rebuilds, I don't have a device to confirm. He did, on named platform versions, and reported what he observed. That exchange is exactly the practice we ended up recommending to everyone else — asking a reviewer with the right hardware rather than guessing.

Two findings we didn't expect

The first is about labels, and it changed our own methodology. #41607, from an outside contributor, was labelled accessibility within hours. None of #42500, #42539 or #42524 carries that label. We'd been searching by label to find accessibility work, which meant our own method would have found the one that was rejected and missed all three that shipped — so we now tell readers to search titles and bodies instead.

The second is that merging isn't shipping, which we'd been treating as an outcome measure without thinking about it. All four cases merged into v6-dev or were closed because v6-dev supersedes them. Issue #41606 is closed as completed, while anyone on 5.3.7 or later still has the focus-trap regression @DerLukas15 bisected to that release.

Corrections

If anything is factually wrong, or if there's context the public threads don't show, we'd like to know. We're working the corpus up into an article — nothing is scheduled yet, and any correction you send will be reflected in both the repo and whatever gets published.