Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do an accessibility audit and make a blog post about it #10318

Closed
tofumatt opened this issue Oct 3, 2018 · 86 comments
Closed

Do an accessibility audit and make a blog post about it #10318

tofumatt opened this issue Oct 3, 2018 · 86 comments

Comments

@tofumatt
Copy link
Member

@tofumatt tofumatt commented Oct 3, 2018

Right now there is a perception that Gutenberg is quite inaccessible, both in general and compared to the Classic Editor. Below is more/less a copy+pasted comment I made in a recent Make Post about 5.0:

It should be pointed out that the existing (Classic) editor doesn’t include many components for plugin developers extending the editor. The core editor in WordPress is accessible by virtue of being quite basic (it's essentially a <textarea>), but it’s also at the mercy of plugin developers to ensure their code is accessible. This means the difference between a WordPress editor without and with plugins is wildly different–this should be less true for a Gutenberg install with 5 or 500 blocks.

Core Gutenberg components and blocks are built with accessibility in mind, and block author will be able to use these components in their own blocks and get a lot of accessibility built-in to their blocks for free. This is a big win for accessibility of the editor after it’s been extended at all.

There are certainly aspects (often third-party code like colour or date pickers) that are not accessible, though we’ve been working to improve those when there are better solutions.

The Classic Editor and Gutenberg’s editor aren’t an apples-to-apples comparison; one is extremely basic and one is very full-featured. I expect the latter to require more work to make fully accessible, but as we work toward that (and identify bugs), we can provide all users with a better experience.


I think there's a notion of Gutenberg being inaccessible because of older accessibility audits that identified a lot of issues in the very early versions. Things have changed a lot since the early days, and when the plugin was labeled "1.0" it was hardly a ready-to-ship product. I worry that many of those sentiments haven't been re-examined and updated, so there is a prevailing idea that Gutenberg is not accessible or is entirely less accessible than the Classic Editor.

What I'd venture is that Gutenberg is selectively less accessible, but overall more accessible feature-for-feature. Something like a date picker or a certain interaction being inaccessible does not make the entire editor inaccessible. Feature-for-feature, compared to a classic editor with similar capabilities (eg a bunch of plugins installed), I'd bet* Gutenberg is more accessible.

Anyway, I would like to see about doing a new round of accessibility testing with community a11y folks (and see about accessing some Automattic resources to help with it as well).

After that it'd be cool to do a blog post with data that could show Gutenberg's current state of accessibility. It'd be neat to highlight the built-in accessibility features that third-party blocks get for free as well, to show how blocks beat old editor plugins for built-in accessibility.

Inspired by: https://wordpress.slack.com/archives/C02RQBWTW/p1538599045000200


  • -One coffee bet, signed me.
@tofumatt tofumatt added this to the Merge: Accessibility milestone Oct 3, 2018
@tofumatt tofumatt self-assigned this Oct 3, 2018
@tofumatt
Copy link
Member Author

@tofumatt tofumatt commented Oct 4, 2018

It would be really cool to know that all blocks that replicate classic editor functionality are accessible in Gutenberg.

@mor10
Copy link
Contributor

@mor10 mor10 commented Oct 4, 2018

The right thing to do here is to get an independent accessibility audit. WordPress has a stated policy of meeting accessibility guidelines in core, and also has a responsibility to its users to meet those guidelines. Only a thorough accessibility audit can show whether this is the case or not. The accessibility team have flagged numerous issues, some of which have been handled, others which have not. At the very least, these must be addressed.

WordPress is an interface between the publisher, a database, and the visitor. How exactly the publisher chooses to input or manipulate the interface is up to that user. WordPress' job is to make the interface accessible for whatever input modality chosen by that user. This is neither new nor controversial: It is the foundation on which the web stands.

@bamadesigner
Copy link

@bamadesigner bamadesigner commented Oct 4, 2018

I agree with @mor10. Considering how invested Automattic and the core team has been in the project, and are still seeing issues, the only true and valid path forward is to have an independent, objective audit.

@postphotos
Copy link
Contributor

@postphotos postphotos commented Oct 4, 2018

WordPress has a stated policy of meeting accessibility guidelines in core

Yes, exactly. I'd want confidence from the leads in the larger w.org project - such as @afercia and @rianrietveld - in the work here.

In short, I believe Gutenberg CAN be accessible, but I'm concerned it might not be shippable in its current state based on the 117 items open in the project.

Hoping this can happen!

@rianrietveld
Copy link
Member

@rianrietveld rianrietveld commented Oct 4, 2018

I'd fully support an external, independent accessibility audit.

@nic-bertino
Copy link

@nic-bertino nic-bertino commented Oct 4, 2018

As you mentioned, there are some perception issues that I think are related to the lack of material and information regarding user testing, and specifically, re-testing as development progressed. Instead of chasing user outcomes, it feels like we are chasing down bugs and regressions.

I appreciate your honesty here, and I agree that an external/independent audit would be valuable. But, I do fear that this is retroactively addressing accessibility from a technical standpoint (i.e., checking the boxes for WCAG 2.x/AA) instead of looking to provide a better experience for those that might not be using a mouse and keyboard. For accessibility alone, qualitative and quantitative research of the Gutenberg experience without data relative to the current experience (or, even better, other similar products) might not tell a fair story.

I think transparency around this information would go a long with community support and ease a lot of friction, both from an accessibility standpoint and the larger Gutenberg rollout.

@tofumatt
Copy link
Member Author

@tofumatt tofumatt commented Oct 4, 2018

I'm not surprised to hear others are in favour of an external audit, which would be great to do. 👍 I can tell you I'd love that too, and will be looking into ways we can facilitate that over the next week.

Are there any accessibility-specialising agencies/companies involved with WordPress that would be able to prioritise their time to run through an accessibility audit of the project (probably as of 4.0, which is a big release)? I think it would be handy to establish a baseline sense of how Gutenberg stacks up against the Classic Editor.

I am sure there will be issues (as mentioned above, we have over 100 accessibility issues) but I'm trying to get a macro view of accessibility in Gutenberg. It should be pointed out we have well over 100 issues labelled "bugs" as well, but this is thanks to us considering no bug too small. We try to log everything, but remember this means 100 accessibility issues, not 100 accessibility blocks. The accessibility blockers (as of right now) are contained in the Merge Milestone this issue is a part of.

Thanks to those who have added their voices to support of an audit. I will provide a list of flows we want to test and ensure work for users as an MVP--when I do I'll post them here but I haven't fully formulated them yet.

@Shelob9
Copy link
Contributor

@Shelob9 Shelob9 commented Oct 5, 2018

An external audit is a great idea. I would be happy to contribute financially to hiring a qualified firm to perform this audit.

@bamadesigner
Copy link

@bamadesigner bamadesigner commented Oct 5, 2018

I would also contribute financially, especially for something of this magnitude and importance.

I would also be willing to lead the way to help raise money from the community.

@aardrian
Copy link

@aardrian aardrian commented Oct 5, 2018

I like this idea. For enterprises that require WCAG compliance in their software, an audit can go a long way to quelling concerns. Generating a VPAT from an audit may also help uptake in organizations that require it as part of their contract process. See Building Accessibility into Technology Vendor Contracts for related info.

@logoscreative
Copy link

@logoscreative logoscreative commented Oct 5, 2018

I also support an external, independent accessibility audit. I'll check into my day job's excellent a11y resources and see what they recommend. :)

@tofumatt
Copy link
Member Author

@tofumatt tofumatt commented Oct 5, 2018

That’s very kind! But we aren’t looking for financial contributions from community. I’m going to contact some accessibility auditors and see about getting them to run some audits on Gutenberg.

I’ll see what kind of costs and timelines are involved, but if we can get something reasonably priced with a timeline that will work for 5.0 I should be able to get Automattic to cover the cost. 😊

@tofumatt
Copy link
Member Author

@tofumatt tofumatt commented Oct 5, 2018

I was a bit unsure of financial support I could provide from Automattic when I posted an earlier comment and I was a bit unclear about expectations from community: sorry about that! I’ve updated the comment and clarified what I wasn’t looking for. Right now I’m going to see about arranging audits, it will take me probably a few days at least to wrangle the quotes! I will report back once I know more 😊

@grahamarmfield
Copy link

@grahamarmfield grahamarmfield commented Oct 5, 2018

I know this may not be the place to state this, but my company does accessibility audits. So let me know if you want to know more. Although not so active recently, I have been involved with WordPress for a while and have contributed accessibility ideas to Gutenberg as well as other parts of WP. If I'm not independent enough, I work with a trusted partner who has never used WordPress.

@nic-bertino
Copy link

@nic-bertino nic-bertino commented Oct 5, 2018

@tofumatt I think a request for proposals would be a good way to state exactly what an audit is looking for and give service providers an opportunity to provide a general outline of how they would proceed. Is there a precedent for this type of process at Automattic?

@CTenan
Copy link

@CTenan CTenan commented Oct 5, 2018

Hello, I am the IT Accessibility Coordinator for NC State University. We use WordPress for our campus websites and as a state institution we are required to follow Section 508 of the Rehabilitation Act of 1973. Because of this, our authoring tools like WordPress have to be accessible for individuals with disabilities.

We have a huge variety of users of WordPress including students, faculty and staff who use WordPress blogs and edit WordPress sites on behalf of their departments of their organizations. All that to say, WordPress must be accessible for these individuals.

As an institution of higher education, we want to stay current with new programs and technologies. It would be detrimental for us to not be able to progress and use Gutenberg or to do a "separate by equal" situation where people with disabilities are limited to using the classic editor while able-bodied people can Gutenberg.

@tofumatt
Copy link
Member Author

@tofumatt tofumatt commented Oct 5, 2018

That is so awesome to hear 👍

I totally agree! Of course, I want to set the clear expectation that there may be accessibility issues we discover (or possibly know about) that we are not able to fix before WordPress 5.0 is released.

There are plenty of sites, for plenty of reasons, that will want/need to use the Classic Editor plugin for a limited amount of time while Gutenberg–or outside plugins–are improved. There are probably going to be users who encounter accessibility issues–along with a host of other issues–that necessitate the Classic plugin for a few weeks/months while we improve our software.

I want the future of the WordPress Editor to be awesome and accessible for everyone. My apologies in advance for users who fall through the cracks at first, for whatever reason. But know we will work toward making Gutenberg and WordPress awesome for them too, even if it doesn't happen right away. ❤️

Regarding Audit

Hey all! I'm still a bit new to being a release lead and I'm coming from nearly a decade working on the @mozilla project where we too worked very much in the open, missteps and all. I'll be as transparent as possible about what's happening, including trying to give timely updates (hence my being unsure about being able to pay for accessibility audits until checking with some folks).

The kind of audit I'm looking to do may not be anything too legal/compliance-specific as I am first looking to get a macro sense of the usability of WordPress by users with accessibility needs. If that includes compliance stuff though: awesome.

Right now I've been in touch with some @WordPress Accessibility community members and knowledgable @Automattic employees who have provided me with some folks to contact about an Accessibility audit. Right now I want it to be impartial (somewhat outside of the WordPress community is great to avoid bias/past knowledge/etc.), high-level, actionable, and done somewhat quick. I know I'm asking a lot 😆

I have some folks in mind now and will be reaching out to them I have contacted several companies that specialise in accessibility audits. I'll post updates here, but I won't know more until next week. For now, there's:

  1. no need to offer financial support (but OMG thank you so much to those who have, that's super cool to see your commitment to WordPress)
  2. no need to offer your services (I think we want someone outside the WordPress ecosystem, but if you want to help with regular bugs/accessibility testing please do!)
  3. much ❤️ from me 🙂

Have a great weekend, I'll keep you all posted! 😄

@LakenH
Copy link

@LakenH LakenH commented Oct 5, 2018

If looking for further vendors for the audit, I recommend WebAIM at Utah State University, who is generally considered as one of the top web a11y resources in the United States. Not affiliated with them in any way other than reading their amazing web a11y documentation, but I have seen that they do offer audits.

@sbrinley
Copy link

@sbrinley sbrinley commented Oct 5, 2018

Although WordPress itself is open source and still has lots of volunteer labor, it's also hit the "big time," as of were, and can no longer afford to fall back on the, "But we're just open source," crutch. I think a full accessibility audit and the commitment to improve where WP is found to be lacking (not just Gutenberg) would be a great source of forward movement when it comes to WP adoption. Plus, of course, democratizing publishing, and all that.

@davisshaver
Copy link
Contributor

@davisshaver davisshaver commented Oct 10, 2018

There are plenty of sites, for plenty of reasons, that will want/need to use the Classic Editor plugin for a limited amount of time while Gutenberg–or outside plugins–are improved. There are probably going to be users who encounter accessibility issues–along with a host of other issues–that necessitate the Classic plugin for a few weeks/months while we improve our software.

This sentiment is just unacceptable to me and I hope many others who have internalized the WordPress values...

My apologies in advance for users who fall through the cracks at first, for whatever reason. But know we will work toward making Gutenberg and WordPress awesome for them too, even if it doesn't happen right away. ❤️

What is the upside of pushing through an experience that’s not up to the standards of everything else in Core? Deadlines aren’t arbitrary, but that doesn’t mean you cut corners. At least not according to the WordPress values as I understand them. These values are likely distinct from anything Mozilla had, as a newcomer to the community I encourage you to look into them.

In general the sentiment that mostly-working will be enough creates unnecessary risk for the community, and one must wonder who benefits from that. What is the upside to the Gutenberg project or WordPress? It’s not as if we need the help identifying issues.

@davisshaver
Copy link
Contributor

@davisshaver davisshaver commented Oct 10, 2018

Since Automattic will apparently be sponsoring and conceiving parameters for this audit, there’s the potential appearance of bias and that they are scoring their own work. I propose that the greater community proceed with crowdfunding to hire an accessibility specialist for a parallel report from the perspective of actual users. Maybe Automattic can do a technical audit or whatever, but the suggestion that Gutenberg accessibility is “not so bad” has been vehemently disagreed by real experts in this field (to wit, Sina Braham at WordCamp Publishers). Maybe we need something closer to a Mikey test where we get blind users to navigate through the app and say if they’d use it again.

@tofumatt
Copy link
Member Author

@tofumatt tofumatt commented Oct 10, 2018

This is what I've been sending out to companies I've been approaching for this work, by the way. It may help other folks involved in Accessibility, I'm not sure. At the very least I'm posting it here for transparency 🙂


Requirements

A series of accessibility tests to be done using the Gutenberg editor on a WordPress website using WP-Admin. These tests should include, at minimum, the following test flows:

Core Publishing Flow

  • Create a new post
  • Add common content (paragraph, image, list, HTML, and table content; see “Using Blocks” below)
  • Publish a post immediately and post is made as intended
  • Schedule a post for the future using date picker
  • Assign categories and tags to post
  • Assign a featured image to a post
  • Experiment with other Document-level settings, time-permitting

Using Blocks

These are the top-twenty most used blocks on WordPress.com and should be tested to find any accessibility issues not already filed in our list of accessibility issues:

  1. paragraph
  2. image
  3. heading
  4. list
  5. gallery
  6. quote
  7. embed/youtube
  8. html
  9. separator
  10. more
  11. cover-image
  12. shortcode
  13. button
  14. table
  15. spacer
  16. embed
  17. block
  18. embed/twitter
  19. columns
  20. code

These blocks should be tested with assistive technologies to ensure there are no blockers to their usage, and that their block-level settings are accessible.

Classic Editor

We should verify that, compared to the Classic Editor, there are not notable regressions with regard to accessibility. Creating a post with the same kind of content one can create in a barebones Classic Editor context should be doable with Gutenberg. Gutenberg having zero regressions in editing experience compared to the Classic Editor is my ultimate goal for saying we can recommend it as the default editing experience for users of assistive technologies.

Assistive Technologies to Consider

Because of the limited timeline available and the prioritisation of actionability over extreme thoroughness, I would prefer the top two-to-three browser+screenreader combinations to be used for testing. Using more if time allows is absolutely accessible, but I would prefer starting with the most-used combinations. From my understanding this is:

  • JAWS with Internet Explorer/Edge (Windows)
  • NVDA with Firefox (Windows)
  • VoiceOver with Safari (MacOS)
  • ZoomText

Mobile assistive technology is not a priority for this audit. VoiceOver testing on MacOS should hopefully include some crossover with iOS VoiceOver users. Mobile app usage is presumably more of a focus than web site usage.

Post Report and Issues in the Open

Our expectation is that any generated accessibility report should be posted in the open to the WordPress community and should not require me (or any other Automattician) as a gatekeeper. Issues should be posted with useful context to GitHub, with a focus on issue cause and reproducibility. Solutions or approaches for fixes can be posted if there is time in the scope or if the problem is especially complex.

@LukePettway
Copy link
Contributor

@LukePettway LukePettway commented Oct 10, 2018

@davisshaver Hello, and I totally agree with you. A thorough and unbiased audit is critical to ensuring that the maximum level of a11y is adhered to.

Not only is it important to make sure the editor is accessible but that there is also a foundation in place to ensure that modifications to the editor can also easily be made accessible. Obviously we can't enforce what people do on their own, but we can at least provide a foundation for people to implement that is accessible.

@tofumatt Also happy to assist in any way possible. I may have access to some pretty good resources for testing, specifically others with React and a11y experience. There are also some people on my team who specifically work with a11y in their day to day.

@bamadesigner
Copy link

@bamadesigner bamadesigner commented Oct 10, 2018

@tofumatt Thank you for the update. I have a few follow up questions:

  1. Is the plan for the audit to be completed before 5.0 is released?
  2. If the audit returns issues, what is the plan for addressing them? Will the 5.0 release be delayed until they are addressed?
  3. Are there plans being made for how the project will test for accessibility going forward?
@grahamarmfield
Copy link

@grahamarmfield grahamarmfield commented Oct 10, 2018

@tofumatt If you want some testing by users with a variety of disabilities/impairments, I've done some work with a company called Hassell Inclusion, and they can organise that sort of testing. They're completely removed from the WordPress community. The company is run by Jonathan Hassell who used to be in charge of accessibility at the BBC.

@davidberman
Copy link

@davidberman davidberman commented Oct 11, 2018

@tofumatt I am also very concerned about WordPress accessibility and eager to help you succeed. I can have our organization perform a full WCAG-EM compliant accessibility audit to whatever standard level you prefer, including both technical and functional testing, and donate whatever portion of the work you need to fit your budget.

I would think we should be looking at both ATAG and WCAG AA, and perhaps WCAG 2.1 rather than 2.0. We could also provide recommendations and coaching to help you choose the best and most sustainable way to close any gaps discovered.

We are conveniently arms length from your development community, yet know WordPress well for many years. I hold all IAAP certifications and have led dozens of such audits on the WordPress platform, over many years, for both private and public sector. Please check out our bona fides at davidberman.com/about to decide how we can best be of assistance.

@afercia
Copy link
Contributor

@afercia afercia commented Oct 11, 2018

Using Blocks

I'm not sure the team going to make the audit should be instructed or have knowledge of what a "block" is, at least not initially. They should be in the same initial condition of users who are going to use Gutenberg for the first time, with no special instructions. In a second phase, when going technical, they will probably already know what a block is 🙂

Core Publishing Flow

The accessibility team agrees the main accessibility issue in Gutenberg is the overall user experience. I'd suggest to expand this part: it shouldn't be limited to the publihsing flow, instead it should include more tasks, typically the ones that require navigating through the UI and jumping through the main sections (top bar, editing area, sidebar, publish panel). This is more related to the general Gutenberg design, rather than technical implementation details on the various components. In fact, accessibility is design.

I'd like to propose a few tasks:

  • build a post with a large number of blocks, say 20 as a minimum
  • edit one of the paragraphs in the middle of the post
  • change its font size and font color, then edit again its content
  • add new blocks using the inserters
  • insert a link
  • insert a mention
  • change block type
  • change something in the top bar settings and then go back to the block that was edited
  • jump to the sidebar, switch to Document / Block settings and then go back to the block that was edited
  • edit the post permalink
  • make a reusable block
  • add and edit an existing reusable block

Assistive Technologies to Consider

@tofumatt I'm a bit surprised you're mentioning only assistive technologies and, specifically, only screen readers. Accessibility evaluations can't be limited to only screen reader testing. Accessibility is a way broader topic and it's not limited to disabilities or impairments. It's about making the web accessible to everyone.

Even if we want to limit accessibility to disabilities and impairments, they're just too many to count. Just to name a few: speech recognition software users, motor disabilities, low vision, cognitive impairments, seizure disorders, alternative input devices, attention deficit hyperactivity disorder, dyslexia, short-term memory loss, etc., etc. What about things that can't be classified as real "disabilities", for example temporary impairments, environmental impairments, ageing?

The human condition is pretty unstable, and I've suggest you to consider We're all just temporarily abled.

too many to count – image courtesy of Denis Boudreau

@aaronjorbin
Copy link
Member

@aaronjorbin aaronjorbin commented Oct 16, 2018

@davisshaver Please remember the WordPress etiquette, especially:

Contributions to the WordPress open source project are for the benefit of the WordPress community as a whole, not specific businesses or individuals. All actions taken as a contributor should be made with the best interests of the community in mind.

The WordPress open source project is a volunteer-run community. Even in cases where contributors are sponsored by companies, that time is donated for the benefit of the entire open source community.

While @pento, @tofumatt, @m and others are donated by Automattic, they are working for the betterment of the project. Automattic is not the ones that are deciding when Gutenberg will be released.

Auditing can and will continue as noted by @pento:

now is an excellent time to be testing and reporting bugs.

Anything that can be done to help contribute to that testing and reporting process is beneficial. There are a number of older issues that could use testing to see if they are still an issue that don't require any specialized assistive technology to test. Doing so would be a great assistance and is in part what an independent audit would have provided.

@davisshaver

This comment was marked as off-topic.

@LukePettway
Copy link
Contributor

@LukePettway LukePettway commented Oct 16, 2018

I'm sure a lot of people are lurking this topic at the moment and I will say if you are interested and want to help but you haven't used assistive technology before (screen readers) join the discussion over at https://make.wordpress.org/chat/ in the accessibility channel. Feel free to reach out with a DM to me even. You might be surprised at how easy it is to get setup with NVDA/Voice Over/JAWS so that you can start testing things out.

Beyond that there are a lot of tools which are available for testing WCAG requirements:
Chrome aXe:
https://chrome.google.com/webstore/detail/axe/lhdoppojpmngadmnindnejefpokejbdd

Contrast Checker for WCAG: https://chrome.google.com/webstore/detail/wcag-contrast-checker/plnahcmalebffmaghcpcmpaciebdhgdf?hl=en

Chrome A11y Tools: https://chrome.google.com/webstore/detail/accessibility-developer-t/fpkknkljclfencbdbgkenhalefipecmb?hl=en

There are 113 open issues with the accessibility label that are currently open:
https://github.com/WordPress/gutenberg/labels/Accessibility

Whether we may agree with the current decisions being made or not, I think we can all agree that there is still a lot of work that we can all take part in lending a hand to help solve. Even in an absence of an official audit the community still has the power to help deliver a more accessible experience.

@amandarush
Copy link

@amandarush amandarush commented Oct 16, 2018

This issue was filed to publicise my work on the accessibility audit, to communicate that I'd like it to happen, and to track the sharing of that audit in a blog post. It's not meant as a discussion point around what should be considered a release blocker for WordPress 5.0. I've noted folks' opinions on the matter, and I'll relay them to @m, who is the release lead for WordPress 5.0. My understanding is:

1. I don't have have veto over 5.0 shipping, but even if I did:

2. I'm still not convinced there are sufficient Accessibility issues that prevent a release

If the second point changes, I'll relay that info. I plan to be an advocate, but I don't set the timelines and I also don't have solid data around accessibility. That's the point of the audit: so we can speak from a place of hard data. 🙂

Thanks for all the input, but as this discussion has become quite off-topic from its intended scope, I'm locking this issue's conversation for now. I'll leave it open and post updates surrounding the audit.

Literally every person with disabilities who has tested Gutenberg, both recently and at the outset, has flagged blocking issues as to why it's not accessible. And user testing is just as important to accessibility as is WCAG 2.0 level AA compliance. Probably should say WCAG 2.1 AA at this point but WCAG 2.0 still gets you a lot closer to where you need to be. Saying that you don't have data isinaccurate at best.

@amandarush
Copy link

@amandarush amandarush commented Oct 16, 2018

A lack of an audit in this case is irresponsible because, for starters, unlike WordPress in the past, a framework has been chosen that does not come with accessible components out of the box, and unlike with past WordPress, you are heavily modifying the document object model with Gutenberg, as well as removing things from the accessibility tree, and in the case of browsers, the DOM and the accessibility tree are literally how assistive technology retrieves information to interpret. Even if you leave assistive tech out of this, as Gutenberg currently stands, the cognitive load is enormous. The classic editor plugin is a completely unacceptable substitute. For starters, it's site-wide. So if a site's using Gutenberg, and I need to go in and do something with content, I can't do that on a per-user basis, and I can't install the classic editor plugin, do my work, and then uninstall it and expect that everything will behave as it should once the reversion back to Gutenberg takes place. And yes, I've tested this. The fact that accessibility is being left as an afterthought, again, is literally how WordPress got into the mess it was in back in 2011/2012 when the accessibility team first got off the ground. And the project is quite literally making the exact same mistake again, and asking us to trust that it'll eventually get this right, again. I'm sorry, but how many times do people with disabilities have to be sent to the back of the bus for the greater good? How many times to we have to be satisfied with a whole lot of pretty words about accessibility, about how we promise we're going to do better next time, only to find when it comes down to brass tacks that accessibility really isn't a priority?

@kevinwhoffman
Copy link
Contributor

@kevinwhoffman kevinwhoffman commented Oct 17, 2018

For at least the time being, Automattic has decided to forgo conducting an Accessibility audit on Gutenberg. The main reasons being:

  1. an audit will not be actionable given our release timeline, because…
  2. the audit will not affect release timing, so...
  3. it would be more prudent to explore an audit on a less rushed timeline in the future

@tofumatt @pento Can you clarify whether the timeline mentioned here refers to a release date of November 19, 2018 or January 22, 2019 as mentioned in the Proposed WordPress 5.0 Scope and Schedule?

The January 22, 2019 date would allow more than three months between today and the release of 5.0 to complete an audit and take action. The reasons above suggest that we cannot get an audit completed and significantly improve accessibility in three months time. If true, that is all the more reason to start the process now and respond to the audit by fixing as many issues as we can before 5.0 releases.

The idea that the timeline will become less rushed after 5.0 (when it's in the hands of real-world users who need it most) makes no sense at all.

@rcemory

This comment was marked as off-topic.

@pento
Copy link
Member

@pento pento commented Oct 17, 2018

Thank you everyone who's commented here so far and kept the discussion on topic. I really appreciate it: with so much going on around WordPress 5.0, it really helps me be able to put aside the time that this issue deserves. 🙂

@amandarush: You mentioned several concerns, I'll try to answer them all, but please let me know if I miss anything.

Literally every person with disabilities who has tested Gutenberg, both recently and at the outset, has flagged blocking issues as to why it's not accessible.

I'm truly grateful to everyone who's flagged these issues. I had thought that we'd done a reasonable job of addressing these issues as they came up, but clearly there's more work to do. When @tofumatt has been referring to "not having data", I believe he's looking at it from the perspective that we've fixed a lot of accessibility issues, we don't have a full idea current state of things, so we want to get a clearer picture of the severity of the remaining issues.

A lack of an audit in this case is irresponsible because, for starters, unlike WordPress in the past, a framework has been chosen that does not come with accessible components out of the box

Are you referring to React here? My understanding is that applications built with React are similarly accessible to any other library that dynamically updates the DOM. Provided we make appropriate use of aria-* attributes (as well as ensuring predictable keyboard focussing), that it should be usable with assistive technology. Is this not correct?

The classic editor plugin is a completely unacceptable substitute. For starters, it's site-wide. So if a site's using Gutenberg, and I need to go in and do something with content, I can't do that on a per-user basis, and I can't install the classic editor plugin, do my work, and then uninstall it and expect that everything will behave as it should once the reversion back to Gutenberg takes place.

Thank you for bringing up this use case, it's a good example of where the Classic Editor plugin currently fails. We can certainly add a per-user option.

And the project is quite literally making the exact same mistake again, and asking us to trust that it'll eventually get this right, again. I'm sorry, but how many times do people with disabilities have to be sent to the back of the bus for the greater good? How many times to we have to be satisfied with a whole lot of pretty words about accessibility, about how we promise we're going to do better next time, only to find when it comes down to brass tacks that accessibility really isn't a priority?

I'm sorry. Despite the best intentions of everyone on the Gutenberg team, we haven't done enough. I can honestly say that accessibility has always been a priority, but it hasn't been a high enough priority, and we've done a poor job of communicating where accessibility has been improved. I mentioned some of those improvements in my earlier comment, but those improvements are of no benefit if we haven't hit the baseline accessible experience.

Of course, an apology is no use without actions to make it right, so here's what I'm proposing. Right now, the intention is to prioritise and fix as many accessibility issues as possible. We've fixed hundreds already, and there are fixes already in progress for some of the open issues. We'll be testing, of course, but we do need the experience of people who use assistive technology to uncover bugs. Any testing and bug reporting you're able to make time for is greatly appreciated.

While I believe we can get the block editor to a state where it has an acceptable UX for assistive technology users by the time 5.0 is released, I do recognise there's a possibility we can't. Apart from the per-user setting you suggested, how else can we ensure the Classic Editor plugin is a reasonable option for folks to use until they're happy with the state of the block editor?

I would still like to see an independent accessibility audit happen, but without the rushed timeline of the 5.0 release. That's where this issue can be used to create the framework for a comprehensive audit. As problems are uncovered, they can be fixed in 5.0.x releases, there's certainly no need to wait for 5.1.

@pento
Copy link
Member

@pento pento commented Oct 17, 2018

@kevinwhoffman: The timeline refers to the Nov 19 (or Nov 27 at the latest) release date. Pushing the release date to January 22 wouldn't give us much extra time: between WCUS (and the preparation required leading up to it), needing to organise a WordPress 4.9.9 release, then folks being on vacation over Christmas and New Year, we might gain an extra two full work weeks by pushing the release date by two months. An extra three months sounds like a generous amount, but it would give us very little actual results, and we'd still need a speedy audit, so we could be working on addressing it in November.

@rcemory: While your comment is kind of off topic, it's worth noting that this discussion is related to the block editor interface itself, not the content that it produces to display on the front end of your site. As with WordPress now, content produced in the block editor is accessible. In many cases, it's more accessible than before, as the block editor does a better job of adding appropriate aria-* labels, it warns when you're using colour combinations that aren't WCAG compliant, or when you're putting heading elements in the wrong order.

@markkap

This comment was marked as off-topic.

@grahamarmfield
Copy link

@grahamarmfield grahamarmfield commented Oct 17, 2018

The comments from @amandarush are powerful and from the heart. And, I know, based on the past experience of trying to advance accessibility within WordPress.

Since I have been involved in the Accessibility Team there have been 4 (I think) major functionality upgrades in the admin area: Custom menu builder, Media manager, Customiser, and now Gutenberg.

As originally conceived and built all of these components were pretty much (or totally) inaccessible to screen reader users and sighted keyboard users. In my view this is due to accessibility not being thought about at all at the graphic design and UX stages. After the Accessibility Team have cried foul, accessibility improvements have been made to all 4, but the underlying design and UX remain largely the same.

So I share her view that WordPress keeps making the same mistake over and over again. And sadly I share her scepticism when we hear "Accessibility is really important to us". It's very frustrating, and one of the reasons I seldom contribute to WP anymore.

It seems unlikely now that accessibility concerns will halt the push to release WP5.0 in November, which of course is a huge let down after the list of accessibility 'blockers' was first drafted in the spring of this year. And the oft-quoted desire for WP to democratise the web, and the promise not to release any functionality that is not WCAG2.0 compliant.

But I think that an independent accessibility review - involving not just testing for WCAG2.1, but also proper user testing - would be a sensible thing to do now. At least that way we all would know where we stand, and appropriate focus can be given to prioritising and fixing the accessibility defects/issues in future releases.

In response to this from @pento

Are you referring to React here? My understanding is that applications built with React are similarly accessible to any other library that dynamically updates the DOM. Provided we make appropriate use of aria-* attributes (as well as ensuring predictable keyboard focussing), that it should be usable with assistive technology. Is this not correct?

Yes, that's true, but the devs need to understand how to make their components accessible in design, semantics, operation, focus management, and with the correct use of ARIA. Sadly, this seldom happens in my experience.

@aardrian
Copy link

@aardrian aardrian commented Oct 18, 2018

If/as Automattic goes down the path of an audit, it may be worth not just identifying critical flows / user journeys, but also common widgets to review. If these widgets can be identified and audited alone, they can also form the basis of an accessible pattern library. This can help mitigate accessibility risks as more features are built using those patterns.

@postphotos
Copy link
Contributor

@postphotos postphotos commented Oct 18, 2018

it may be worth not just identifying critical flows / user journeys, but also common widgets to review. If these widgets can be identified and audited alone, they can also form the basis of an accessible pattern library. This can help mitigate accessibility risks as more features are built using those patterns.

@aardrian That's an interesting starting point that may already exist that will be required for the audit that we may be able to use now to action on: I haven't seen user journeys for Gutenberg, and that is probably the most obvious starting point to starting to scope said audit.

@karmatosed @jwold - Do you know if said flows/journeys for using Gutenberg already exist? Perhaps a list of "widgets" or "patterns" as described here?

I think while this general thread has a lot of emotion (and I agree Gutenberg must be usable by all and ship when ready) - I'd love to find common ground and next steps. I'd love to shift the conversation places to identifying where we're ready to action.

Said user flow would also give us a task list of places to audit for developers, testers and bug hunters to start dividing up and looking for things to report.

@aardrian
Copy link

@aardrian aardrian commented Oct 18, 2018

@postphotos , I think @afercia outlined a good set of flows above in #10318 (comment).

@postphotos
Copy link
Contributor

@postphotos postphotos commented Oct 18, 2018

@aardrian Agreed, that's a great place to start! 👍 Thanks for recalling that.

My ask still stands - I'd love to know, beyond that initial critical flow if there's additional work that's already been put into user journeys as it relates to features as it'll help validate the work we're doing here. There may be other items we (this group) might want to add @afercia's initial list, or sequence past an initial audit. 😄

For example, we've not called out the Content Structure feature, undo/redo, etc. or viewing a post, and those tasks we might agree are important and helpful for users to be able to use - the things that we agree make Gutenberg incredibly useful and should be accessible. It'd be great to find consensus actively if we can.

@mor10
Copy link
Contributor

@mor10 mor10 commented Oct 18, 2018

An additional benefit to doing an audit is contribution back to the larger React project. React has its own accessibility issues, and an audit of Gutenberg may well identify both issues and solutions that relate to and can help React core.

@leepeterson
Copy link

@leepeterson leepeterson commented Oct 18, 2018

Well, this is exciting news: https://make.wordpress.org/core/2018/10/18/regarding-accessibility-in-gutenberg/

Keyboard shortcuts for region and block navigation, slash commands, focusable toolbar and sidebar improvements from improved ARIA landmarks and roles, automated end-to-end tests, and more.

@bamadesigner
Copy link

@bamadesigner bamadesigner commented Nov 28, 2018

ICYMI: The WPCampus organization is organizing an accessibility audit of Gutenberg.

We have finished our RFP and are now selecting a vendor.

However, as most of you are probably well aware, a professional accessibility audit is a large expense for a small nonprofit like WPCampus.

We’re asking for your help to fund the audit to ensure this vital research is completed.

You can read more about the initiative and make donations on our website: https://wpcampus.org/2018/11/fundraising-for-wpcampus-gutenberg-accessibility-audit/

@davidberman
Copy link

@davidberman davidberman commented Dec 1, 2018

@aardrian
Copy link

@aardrian aardrian commented Dec 1, 2018

@tofumatt , I saw this on Matt Mullenweg's personal blog on 29 November:

Finally, Automattic will be funding an accessibility study of WordPress, Gutenberg, and an evaluation of best practices across the web, to ensure WordPress is fully accessible and setting new standards for the web overall.

As you are an Automattic rep, now that the Automattic audit is back on, can you tell us if this will be separate from the WPCampus audit or will Automattic be funding the WPCampus audit instead?

On the one hand, I hate to see WPCampus have to crowd-fund it. On the other, I like the idea of two audits from two firms to compare and contrast.

@aardrian
Copy link

@aardrian aardrian commented Dec 3, 2018

Matt responded to my question on his blog:

I think it’s fine having two as well, I’ll talk to Rachel about how the funding is going.

@m
Copy link
Member

@m m commented Dec 10, 2018

For those following along, have committed to fully fund whatever Rachel's project needs on the WP Campus side, and in the vendor selection phase for the Automattic-sponsored one. The latter's goal is to also see what are the best practice accessibility approaches for other block editor style interfaces across the modern web.

@LukePettway
Copy link
Contributor

@LukePettway LukePettway commented Dec 10, 2018

@m Thanks for the update. It is very encouraging to see this moving along and I really can't wait to see where it leads. I would love for WordPress to one day be the example of how accessibility should be treated.

@aardrian
Copy link

@aardrian aardrian commented Dec 10, 2018

@m

For those following along, have committed to fully fund whatever Rachel's project needs on the WP Campus side

This is great and valuable. Is there any reason you don't just fund it in total, today? It seems weird to ask people to keep donating when they know it will be completely funded regardless.

@bamadesigner
Copy link

@bamadesigner bamadesigner commented Dec 10, 2018

@aardrian That's not in the spirit of what we're doing. We want this to be a community project and to allow people and organizations the opportunity to participate and show support, if they're able.

@gziolo
Copy link
Member

@gziolo gziolo commented May 4, 2019

In late 2018, WPCampus released a request for proposals to conduct an accessibility audit of the WordPress block editor, also known as Gutenberg. In early 2019, we announced our selection of Tenon, LLC to conduct the audit, at a cost of $31,200.

Today, we’re excited to release Tenon’s report on the accessibility of the new editor.

See more at https://wpcampus.org/2019/05/gutenberg-audit-results/

I think it concludes this issue 🎉🎉🎉🎉🎉

All reported issues are tracked under this project:
https://github.com/WordPress/gutenberg/projects/25

@gziolo gziolo closed this May 4, 2019
@afercia
Copy link
Contributor

@afercia afercia commented May 4, 2019

All reported issues are tracked under this project

Not entirely correct. The WPCampus/Tenon report includes additional important considerations that are not covered in the list of issues on GitHub. More specifically, the Executive Summary and the Usability Report show (with data) there are broader structural accessibility issues in Gutenberg that can't be addressed in small, actionable, GitHub issues and require solutions at a higher level instead.

This is to make clear that even if all the reported issues were solved, there will be important accessibility issues still to solve. These are typically related to the general design of the editor.

For reference, I'm attaching here two of the documents published on https://wpcampus.org/2019/05/gutenberg-audit-results/

Summary document describing Tenon’s usability testing
Gutenberg_UX_Report.pdf

Executive Summary
Gutenberg_Executive_Summary.pdf

@mor10
Copy link
Contributor

@mor10 mor10 commented May 6, 2019

I suggest opening a new issue, or even project, to handle the usability issues uncovered in the report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.