[css-highlight-api] Figure out how highlights are exposed to the accessibility tree

[frivoal]

@atanassov recently reminded us that figure out how highlights are exposed to the accessibility tree. This had been know for a while, but we didn't have an issue I could find, so here's one.

[frivoal]

To sighted users, the highlights are only visible through the way they are styled. There is no other way to know what they're for. Maybe the same should be true of the highlights? Getting the highlight itself described through some text sounds like it might be more convenient, but I'm not sure where we'd pull that description from, how to deal with multiple languages, or the potential discrepancies between the descriptive text and what the highlight is actually used for.

[astearns]

And whatever gets decided here should likely match what is decided for #6417

[frivoal]

How are things like these exposed to the accessibility tree today?

• ::spelling-error
• ::grammar-error
• ::target-text

We don't necessarily have to do the same, but it'd be good to start with an understanding of how these are handled.

[ffiori]

#4601 seems to have some related remarks.

[dlibby-]

UserAgent detected spelling and grammar errors (i.e. ::spelling-error and ::grammar-error) in Chromium do have a mapping to various AT APIs. These are serialized as part of the accessibility tree and as far as I can tell, this is not part of any of the Aria-related standards. For example on UIA on Windows spelling errors are exposed as a text range with an AnnotationType_SpellingError. Not sure how this would fit into the spec, but perhaps is a path forward (following the UIA example, there is an AnnotationType_Highlight).

For completeness, spelling and grammar errors that are computed by the webpage can be annotated with aria-invalid. The corresponding concept for highlights would probably be the mark role (see https://w3c.github.io/aria/). However, these require additional DOM nodes, which is counter to the purpose of the HighlightAPI.

[dandclark]

I agree with @dlibby- that for the question of how to expose the highlight ranges to ATs we should be able to follow the lead of how ::spelling-error and ::grammar-error are exposed in Chromium, at least in part.

For expressing the semantics of the highlight ranges, the most flexible option seems to be to add a title string attribute to Highlight, which would be exposed to ATs. However, as pointed out in #4601 (comment), these strings wouldn't be localized. This approach also depends on the author to provide a string with the correct level of descriptiveness.

An alternative I'd like to consider is adding an enum to Highlight that captures the common use cases we expect for this API. These would be exposed as appropriate in different accessibility platforms, e.g. in UIA the most closely matching AnnotationType would be used.

enum HighlightType {
 "highlight",
 "spellingerror", // Exposed to ATs in the same way as ::spelling-error
 "grammarerror", // Exposed to ATs in the same way as ::grammar-error
 "textmatch", // e.g. for custom find-on-page results
// ...and any others for the common use cases we can think of.
};

[Exposed=Window]
interface Highlight {
  constructor(AbstractRange... initialRanges);
  setlike<AbstractRange>;
  attribute long priority;
  attribute HighlightType type; // Exposed to ATs. Default is "highlight".
};

The benefit of this is that we're not relying on author-specified strings, so it's easier for developers and it solves the issue of localization. An enumerated set of types could also be easier to expose to ATs.

The disadvantage is that use cases we don't anticipate won't have any good choices so authors will have to fall back to the generic "highlight". I don't think we'd be locked into the initial set of HighlightTypes though; we could probably add more if additional common uses cases arise.

[michael-n-cooper]

Tagging for a11y-tracker, since the dup that was closed was so tagged.

[dandclark]

Another thing I'll add here is that an advantage of an enum over a string is that when OSs have some first-class representation for any of the use cases in the enum, user agents could plug into them directly. For example, custom spell check implementations using Highlight API could be represented to accessibility tools in the exact same way as ranges marked by the native spell-checker. In UIA this might AnnotationType_SpellingError, or NSAccessibilityMarkedMisspelledTextAttribute on Mac. This would be a more consistent experience for AT users and would save the author the trouble of localizing a "spelling error" string.

It's also feasible that in the future ATs could become more powerful and offer first-class representations of more of these use cases, and it would be great if authors can provide metadata that provides a better experience in those cases.

If needed, in an L2 of the spec a string attribute could be added to Highlight for use cases that fall outside of those we've expressed in the enum. But for use cases that become common and have support in ATs across OSs, it seems more powerful to enable these as additions to the enum. Allowing arbitrary strings could miss opportunities to line up with things that OSs recognize.

[dandclark]

There's been some delay on agenda space opening up to discuss this during CSSWG calls, so I wanted try again to see if we could make some progress here in the issue.

Considering the enum approach considered here and looking more at the current capabilities of UIA's Annotation Type Identifiers and Mac's Attributed String Keys, I'd like to propose starting with an enum with just these:

enum HighlightType {
 // Exposed with AnnotationType_Highlight for UIA, accessibilityCustomText for Mac
 "highlight",
 // Exposed with AnnotationType_SpellingError for UIA, NSAccessibilityMisspelledTextAttribute for Mac
 "spellingerror",
 // Exposed with AnnotationType_GrammarError for UIA, NSAccessibilityMisspelledTextAttribute for Mac
 "grammarerror"
 // More can be added as use cases emerge and accessibility platforms add more text semantic types
};

[Exposed=Window]
interface Highlight {
  constructor(AbstractRange... initialRanges);
  setlike<AbstractRange>;
  attribute long priority;
  attribute HighlightType type; // Exposed to ATs. Default is "highlight".
};

This approach lets custom spellcheck achieve the same experience with accessibility tools as native spellcheck for UIA and Mac today, and leaves room to add more entries to the enum as use cases an support emerge. A custom string could be added in an L2 if the need for that level of flexibility emerges.

@frivoal @sanketj @dlibby- @lilles, any thoughts on this approach?

[css-meeting-bot]

The CSS Working Group just discussed [css-highlight-api] Figure out how highlights are exposed to the accessibility tree, and agreed to the following:

• RESOLVED: Add the new attribute of highlight type which is an enum
• RESOLVED: add spelling-error, grammar-error, and highlight

The full IRC log of that discussion

<dael> Topic: [css-highlight-api] Figure out how highlights are exposed to the accessibility tree
<dael> github: https://github.com//issues/6498
<BoCupp> https://github.com//issues/6498#issuecomment-930627914
<dael> Rossen_: Before we get going, we're starting with 2nd hour of topic. We'll box the first topic to 15 minutes. Right?
<dael> astearns: Sure. Not sure if more because topic is complex
<dael> BoCupp: I'm seeking maybe a few resolutions
<dael> BoCupp: I'll go through what hoping to resolve on and take questions
<dael> BoCupp: Issue comment I posted in IRC summerizes proposed API change
<Rossen_> q?
<dael> BoCupp: Would like to resolve on adding new type attribute to highlight api. Gives symantic meaning that can be used for highlight a11y mappings
<tantek> s/symantic/semantic
<BoCupp> https://drafts.csswg.org/css-pseudo-4/#highlight-selectors
<dael> BoCupp: What to resolve values should include generic highlight on enum that can be used in absense of more applicable value. Any other values should align with pseudo elements named ^
<dael> BoCupp: Last part would like to settle on is for L1 of spec with think in addition to generic is spelling-error and grammar-error. Open to additional ones. Right now have selection and target text in spec
<dael> BoCupp: Selection is complicated topic. In L1 missing support for pointer events, planned for L2. might defer until then. That's my preference. Additionally selection has semantics that make it more complicated
<dael> BoCupp: target-text is more of a question for anyone working on beforeMatch API. Might be good to have discussion on if highlight API can take over that roll
<florian> q+
<dael> BoCupp: If so might consider expanding enum. Hoping to get as far as we can. I'll take questions on it
<Rossen_> q+
<Rossen_> ack florian
<dael> florian: I think the approach is right. Had been mentioned to use descriptive strings but that has many more problems on translating and how a11y layer can use. Starting with spelling and grammar is good
<tantek> q+ to ask if there has been coordination with the Web Editing Working Group, feels like some overlap in scope
<dael> florian: Not sure I would limit to what's there. I think what's most important is a11y tree knows how to handle. If the platform knows how to handle we can add those.
<dael> florian: You have mentioned find-in-page, I think would make sense to expose even if it's not built in
<dael> florian: Agree with general approach. Can explore more values later
<dael> BoCupp: Thank you
<dael> Rossen_: Question. If we pull back, symantically what you desc makes sense for different types of ranges to be added
<dael> Rossen_: From a11y PoV, and this was motivation behind issue, can you explain how these different types of ranges are exposed to a11y?
<dael> Rossen_: Specific example: overlapping ranges that are all overlapping one string of text. What is the a11y tree going to look like? Will the highlight ranges have structural changes to a11y tree or simply exposed as property on existing nodes?
<Rossen_> ack Rossen
<dael> BoCupp: In terms of exposure; chromium a11y tree which translates to multiple platform a11y trees. Those trees have different attribution types. We have different UI constants assigned to range of text. These enums will map to constants in platform-specific way.
<dael> BoCupp: On mac see spelling not grammar so we'd map grammar to spelling. If we added find on page and there's no text-match enum we'd map to a generic highlight so the user knows something is highlighted
<dael> BoCupp: In chromium the format in tree has start and end ranges over text. Each paired with an enum value that annotates text with meaning. I htink they can overlap. Not 100% sure on overlap, good to followup
<dael> BoCupp: Do have priority on highlight interface. If we can't do something for a specific platofrm we can resolve based on priority of highlight. These custom highlights can sort below regular highlights
<dael> BoCupp: Does that answer?
<florian> q+
<dael> Rossen_: Sort of. My worry is we're still exposing something most tree structures are unsuited to deal with. B/c we're allow irregular overlapping. HTML went away from overlapping tags in past and this is allowing. Some worry.
<florian> q-
<dael> BoCupp: One thing to ease worry even though I don't have specific answer. We do ahve selection, find-on-page, spelling and grammar overlaps. Those exist today and our hope is to mirror native platform
<dael> tantek: I like the exploration of the area. Like Rossen_ unsure about implementability. Want to call out difference between delcaritive to style vs and API that could increase impl and privacy challanges.
<dael> tantek: find-in-page I wanted to point out browsers distinguish between all instances of text and the current found on text item which the user sees. Slightly different. Not sure if will expose in API
<Rossen_> ack tantek
<Zakim> tantek, you wanted to ask if there has been coordination with the Web Editing Working Group, feels like some overlap in scope
<dael> tantek: That is a privacy hole. giving access to what text is being searched has privacy implications. Not sure how to quantify but want to call it out
<dael> BoCupp: Response to that is highlight API does not expose that. It would only be if we have something like beforeMatch event where you would get what user is searching and then highlight you could use. It's not mech to discover what is searched for. Thanks for comments, though. Get distinction on find-on-page
<dael> tantek: this does seem to have overlap with web editing group. Is there coordination?
<dael> BoCupp: There is. Originally presented there but decided it was better developed in css
<dael> tantek: great
<dael> fantasai: I think having types of highlights makes sense, esp if can hook into platform APIs. Might be interesting to be able to spec with an extra field that's descriptive and l10n
<dael> fantasai: For all cases where we have or expect a highlight it would make sense to define enum up front. As people write code we want them to pick the appropriate enum even if that's not doing something just yet
<dael> BoCupp: Good point, thank you
<Rossen_> ack fantasai
<fantasai> s/and l10n/and allows l10n/
<dael> florian: I wouldn't rush to having a string. If we have it too early people would rush to that over enum
<dael> fantasai: I think you do that by making string more annoying. Have to put enum and the string and string through dictionary that maps lang tag. I don't think there's a reason not to do unless it would take a while
<dael> florian: I think it would take a while. How do you select what translation? UA but that's usually wrong so what do you use
<florian> s/that's usually wrong/that's often wrong/
<dael> BoCupp: My main concern with free string is that today b/c can't map free string into different platforms those platforms might evolve but in UIA we can't annotate te text.
<Rossen_> q?
<dael> BoCupp: In absense of author hearing the experience they don't know what to write and some people would write grammar and some would write "you shouldn't have a comma without at least 3 things in a string". How do we put some semantic to it. That was why constrained set of types
<dael> fantasai: Makes sense
<dael> BoCupp: Point I like is we can add more enums and communicating those semantics is straight forward. I would like to stay away from free string
<florian> +1
<fantasai> sgtm
<dael> Rossen_: I'm seeing +1 on IRC. Let's see if we can take resolutions
<dael> Rossen_: Prop: Add the new attribute of highlight type which is an enum
<dael> Rossen_: Before we decide on values, opinions or objections to adding?
<dael> RESOLVED: Add the new attribute of highlight type which is an enum
<dael> Rossen_: Highlight, spelling error, and grammar. Obj to adding those?
<dael> heycam: Is there a particular reason to have spellingerror instead of spelling-error?
<dael> BoCupp: Propose the hyphen to match the pseudo names
<dael> BoCupp: One of the things I said initially is prefer we align with css pseudo in terms of names
<dael> Rossen_: Not opposed. We're working on a design principle in TAG that would invalidate this soon, but won't stop now
<dael> Rossen_: Prop: add spelling-error, grammar-error, and highlight
<dael> RESOLVED: add spelling-error, grammar-error, and highlight
<GameMaker> +1 for matching
<florian> q+
<dael> Rossen_: Thank you for bringing this
<dael> florian: Quick question
<dael> florian: To make sense this needs to make to a11y tree. Is someone taking action to desc that somewhere? I suspect not in css spec
<dael> Rossen_: Suggest we add this to API joint meeting during TPAC
<dael> BoCupp: Sounds good

[SuzanneTaylor]

enum HighlightType {
 "highlight",
 "spellingerror", // Exposed to ATs in the same way as ::spelling-error
 "grammarerror", // Exposed to ATs in the same way as ::grammar-error
 "textmatch", // e.g. for custom find-on-page results
// ...and any others for the common use cases we can think of.
};

[Exposed=Window]
interface Highlight {
  constructor(AbstractRange... initialRanges);
  setlike<AbstractRange>;
  attribute long priority;
  attribute HighlightType type; // Exposed to ATs. Default is "highlight".
};

At the TPAC session, it was mentioned that arbitrary strings would have disadvantages. In lieu of this, I wonder if the generic highlight item in the enum could have a secondary field for color? In education, often a variety of highlight colors are made available and individual instructors and groups use those different colors for different meanings. For example, an instructor might tell a student they need to clarify the sections highlighted in yellow. If the color could be exposed with the information that the content is highlighted, this would allow AT users to participate in these informal meaning assignments.

[meyerweb]

Minutes from the CSS & APA joint meeting at TPAC: https://www.w3.org/2021/10/20-cssa11y-minutes.html#t02

[dandclark]

I wonder if the generic highlight item in the enum could have a secondary field for color?

@SuzanneTaylor If the highlight range is being used to change the text color or background color of a range of text, the author will have had to include a rule for that in the ::highlight() pseudo-element selector that the highlight is using. So if we want to include color in the information given to ATs about highlights, I think we could just reuse that style information. Allowing color to be specified separately on the Highlight interface could result in confusion if the color is changed in the CSS and the value on the interface becomes out-of-sync with the value in the ::highlight() pseudo.

[maryjom]

For the use case of highlights in application-specific highlighting. Using this CSS API can an author define multiple different highlights and provide the associated text meaning of a particular color of highlight that can be relayed to AT using the CSS attributes? As an example, an app that scans through large amounts of text to highlight certain aspects particular to the search (e.g. to point out sentiments or emotions in text, such as positive, negative, neutral, happy, angry, etc.)

[SuzanneTaylor]

I wonder if the generic highlight item in the enum could have a secondary field for color?

@SuzanneTaylor If the highlight range is being used to change the text color or background color of a range of text, the author will have had to include a rule for that in the ::highlight() pseudo-element selector that the highlight is using. So if we want to include color in the information given to ATs about highlights, I think we could just reuse that style information. Allowing color to be specified separately on the Highlight interface could result in confusion if the color is changed in the CSS and the value on the interface becomes out-of-sync with the value in the ::highlight() pseudo.

That makes sense, but how will colors specified as numbers become human-friendly words? With contrast requirements already restricting color choices for the highlights, background and text colors, I don't think being limited to the named colors will work well.

[dandclark]

For the use case of highlights in application-specific highlighting. Using this CSS API can an author define multiple different highlights and provide the associated text meaning of a particular color of highlight that can be relayed to AT using the CSS attributes? As an example, an app that scans through large amounts of text to highlight certain aspects particular to the search (e.g. to point out sentiments or emotions in text, such as positive, negative, neutral, happy, angry, etc.)

We’d explored adding a string property to Highlight to convey descriptive meanings, but have thus far avoided this in order to steer authors to built-in semantic types via the enum, which in the future can hopefully be expanded to encompass more use cases. These are preferable because some built-in types have direct representations on accessibility platform APIs, and won't suffer from localization issues. I want to stay open to the possibility of adding a string property in an L2 of the API if real-world usage demands it, but the group resolved here to start with just the enum for the initial version of the API.

That makes sense, but how will colors specified as numbers become human-friendly words? With contrast requirements already restricting color choices for the highlights, background and text colors, I don't think being limited to the named colors will work well.

It should be possible for accessibility tools to turn colors specified with numbers back into human-friendly names. I confirmed that Windows Narrator does this for any random RGB values I try. Here's a small example where the colors are randomly typed RGB values: https://codepen.io/daniec/pen/xxLrqqo
If I set Narrator to Verbosity Level 5 and read the sample text word-by-word I hear: "Background rosy-brown 'she', background dark-grey 'reads', background white 'the', background pale-green 'document". That seems like a pretty reasonable experience without any need for the author to specify the colors in two places that could potentially become out-of-sync.

[SuzanneTaylor]

Windows Narrator does this for any random RGB values I try

love it! (assuming, of course, highlights and their human-readable colors will be expected to be announced at default verbosity levels)

[SuzanneTaylor]

Would it be possible also to indicate in any documentation about browser accessibility support for this feature, that browsers should consider a setting (or maybe there should be a media query for this) that users could select to have the name of the color appear visually?

You would not want ALL CSS colors to appear visually of course, just those associated with highlights. One possible visual layout could be smaller text above the highlighted text, overlapping onto the highlight a bit.

[dandclark]

I think it's a good idea to at least add a note in the spec that browsers should ensure that the colors of the highlights are made available to accessibility.

I'm not yet sure whether there's a need for a setting to expose the highlight colors in a special way. In many (most?) cases, the specific color of the highlights won't matter semantically, e.g. with custom find-on-page, spelling/grammar error, and highlights used for generic emphasis of text. For the rarer cases where color matters, users would know from context on the page that the colors are meaningful and would be able to adjust their accessibility tool's settings accordingly to get that info.

That said I'd like to continue exploring an option like this in the L2 of the spec when we've got more experience with how the API ends up being used in the wild.