compress test server responses and add test matrix with partial retries #951
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since merging #924 test runs fail a lot more often.
I ran tests a bunch of times to see if there was any pattern and it seems that if the first request from
test-runner.handlebars
is a large bundle there is a higher chance of a failure.Failure rate on iOS Safari 10.x was about 60%.
This alone was enough to make CI frustratingly flaky.
Intl.*
test runs fail often.In the end I found that compressing responses from
server.js
helps enough.This change does add a new dependency : https://www.npmjs.com/package/compression
I also converted the github action to a matrix as it has the ability to continue with other jobs when one fails while still having an overal failure of the workflow. A test run "witness" is "recorded" to skip jobs when a retry is needed. This is done by caching a file (which only happens when the job succeeds) and checking for existence of this cache.
This also makes it easier to get a good overal view of which browsers have issues after a change instead of first fixing IE, retrying CI, then fixing Chrome,...
the name "witness" comes from make, but I can't find the reference anymore.
Android fails, Chrome still runs :
IE is skipped on retry :