- 
                Notifications
    
You must be signed in to change notification settings  - Fork 1.2k
 
Reduce CI a11y test time by using Storybook provided axe runner #3284
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| 
          
 🟢 No significant changes to   | 
    
| 
           Interesting that a lot of the slowdown was due to storybook. I thought there was something weird when I tried running the tests locally and it took over 20 minutes, but was confused by the history since there didn't seem to be a smoking gun, and since the slowdown coincided with a move to Travis's free plan that was my first guess. I like the idea of using the all components pages to test here. As long as the error reporting is accurate, I think we should go for that.  | 
    
| 
           We should also add timeouts to these jobs. If we had a timeout then this would never have happened in the first place.  | 
    
| 
           I did some pairing with @dominic earlier around a11y stuff. I'm thinking incremental improvements are still good. I'm gonna open a PR that disables docs (which gets us half-way to our old speed) which we can merge right away. Then have a follow up with a PR to move to axe, @dfmcphee is going to look at fixing the issues that axe raises, and once those are fixed we can merge the axe PR which gets us the rest of the way there.  | 
    
| 
           Have rebased this atop master so now it's a lot smaller. Changes have been split out and merged elsewhere: Already merged: 
 Pending: 
  | 
    
84469e2    to
    5f8cc12      
    Compare
  
    | 
           Had a bit of slack time so I've came back to this. There is now a basic "here's a list of expected errors" that we can define that won't cause the build to fail. If there's an error in the expected errors list that is triggered then it won't cause the build to fail. Unexpected errors are logged out. Expected errors are now. Either way it also gives you the story id so you can go look at that page in storybook to diagnose the issue.  | 
    
| 
           I reverted all the component changes here. We can follow up with fixes separately.  | 
    
Use the instance of axe that storybook already embeds into stories
This should shrink over time to the point we can remove it
| 
           haha it works! On the other hand 68 existing errors, awwww maaaaan :( The code for this is probably a bit fever-dreamy. Thoughts on how we could make the reporting better would be appreciated.  | 
    
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a great first step, I think we can clean up logging and exceptions in follow ups.
Co-authored-by: Dominic McPhee <dominic.mcphee@shopify.com>
Why
A11y tests used to take ~12min. Before this PR they took ~25mins. With this PR they're back down to ~13mins.
Replacing pally with using the axe runner that is already loaded within Storybook's a11y addon serves two purposes:
What
Replace pa11y with running the axe runner that Storybook provides.
Bonus
This PR previously included several changes that fixed the newly raised issues. They were removed in 9a3d014. They can be addressed in separate follow-up PRs.