Improve the accessibility of figures shown with _repr_html_

[choldgraf]

I was running into some accessibility audit errors with Jupyter Book, and it made me realize that the _repr_html_ method of Figures doesn't output img elements that have alt tags (something that screen readers like).

It seems that the relevant code is here:

matplotlib/lib/matplotlib/figure.py

Line 378 in 59acd5c

def _repr_html_(self):

and then uses the template that's referenced here:

matplotlib/lib/matplotlib/backends/backend_webagg.py

Line 289 in 690fd71

tpl = Path(core.FigureManagerWebAgg.get_static_file_path(),

Maybe using this?

https://github.com/matplotlib/matplotlib/blob/be978a35932a9f303f3d60f14f52ed9df89334c5/lib/matplotlib/backends/web_backend/js/mpl.js

I wonder if this could be a relatively simple fix by making the figure title the alt tag by default, and if it doesn't exist, just making it something like Figure?

[timhoffm]

Adding a simple alt sounds reasonable.

[tonyfast]

So i'd love to raise this issue from the 💀

ipython/ipython#12864 adds alt text to Image displays. This begs the question "where could the alt text live?" I'd love to move this issue along and would appreciate some advice.

I'm not familiar with the matplotlib abstractions, but is there a way we could do figure.set_alt_text("a bar chart.."). This way authors could construct/derive their own alt text for a figure. Some sample code with pandas might look like:

plt = df.plot.bar()
plt.figure.set_alt_text(f"A bar chart showing features {', '.join(df.columns)}")

In the output image, the alt text would be assigned to the image with a change in the IPython/core/pylabtools.py

It seems like this could significantly impact a lot of documentation using notebooks or sphinx gallery, and provide extra surfaces for new-comers to contribute improvements.

Is there a certain matplotlib object this feature would live at? Or is something like this out of scope for matplotlib objects in general, like alt text ain't in scope for the abstractions and the display should handle it?

[QuLogic]

Could we grab it from the label? I think that the label is not used for anything on the Figures.

[tonyfast]

If that feature doesn't get used for anything then I think it would be a solid place to infer the alt text from. i don't know enough about the _label_s usage, if y'all think its reasonable to appropriate it for alt text then we'd have a nice existing solution.

[story645]

In #17893 there's a discussion about a colorblind friendly attribute & possibly a spec to go with it, and I think for similar reasons it'd be good to have an explicit .alt_text attribute that can start as a string and possibly grow into a spec as the community coalesces around standards.

I'm worried about using _label_s 'cause I think there's been some previous discussion on the calls about using the artist label attributes for some kind of artist management & now we have the subfigures too and I think it'd be better to keep that usage consistent.

[tonyfast]

if that is the case then maybe it is worth considering adding an aria attribute collection, alt is an alias for aria-label. this might be a more general approach that has a proper specification behind it.

plt = df.plot.bar()
alt = f"A bar chart showing features {', '.join(df.columns)}"
plt.figure.set_aria(label=alt)
# or
plt.figure.set_aria_label(alt)

i could imagine that with some thinking this abstraction could work its way into some of the interactive matplotlib views.

[tacaswell]

I think adding adding this as an Artist level state is probably the right place to put this. This will let us attach arbitrary aria data to anything in the draw tree. To start with we would probably only pull the one from the Figure to populate the data on the repr html, but I suspect that if we had information available we might be able to find a use for it.

Skimming https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques it looks like most of them are not relevant to us (as our scope is much more limited that "arbitrary document / UI").

I propose a straw-man implementation of:

def set_aria(self, aria: Dict[str, obj]):
    # TODO validation
    self._aria = aria

def update_aria(self, **aria):
    # TODO validation
    for k, v in aria.items():
        self._aria[k] = v

def get_aria(self):
    return dict(self._aria)

I think we would want to keep obj.get_aria()['label'] and obj.get_label() separate as the text you would want for a legend and the text you would want to generate a text-description of the object may not be the same text.

[psychemedia]

As well as alt tags, it would also be useful to be able to add longdesc descriptions to <img> tags.

[tonyfast]

as it stands now, longdesc is deprecated, but it could be provided as an alias to aria-details. i think we can capture most accessibility affordances through the different combinations of the aria properties. so adding a longdesc with the current proposal would look like:

plt = df.plot.bar()
alt = f"A bar chart showing features {', '.join(df.columns)}"
plt.figure.set_aria(label=alt, details="figure_description.html")

[psychemedia]

Ah.. I should probably read up on accessibility best practice. My knowledge is probably a bit dated...

[mpetroff]

Resolving this issue and defining a method for setting figure alt text will also allow for the Sphinx-Gallery Matplotlib scraper to be upgraded to use it. This will allow alt text to be used for plots in the examples gallery section of the documentation.

Based on how Sphinx-Gallery is structured, there's no clear way to include alt text unless it's exposed via Matplotlib. Once an alt text method is defined, it also might make sense to default the Description tag in the metadata used by the AGG/PNG, Agg/TIFF, and SVG backends to the alt text if not otherwise defined.