Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

partial tests in ci #997

Conversation

romainmenke
Copy link
Collaborator

@romainmenke romainmenke commented Feb 20, 2021

Changes for tests in CI :

  1. testing polyfill combinations is no longer done per run, moved to a scheduled job on master.
  2. only needed polyfills and browsers are tested if a diff against master contains few changes only in polyfills/....

1 : Scheduled job for polyfill combinations.

These tests are still valuable as they revealed a lot of issues initially.
New errors however are rare and these tests should just guard against issues creeping in over time.

Testing once a week on master should be sufficient to keep the codebase healthy.

The caching mechanism in place to speed up retries will also help here.
If there were no changes in the past week this job will skip each step.

2 : Partial tests.

  1. A diff is taken when test-modified-only is used as an argument for node ./test/polyfills/remotetest.js.
  2. This compares the current commit and branch against master on https://github.com/Financial-Times/polyfill-library.git
    It doesn't take origin/master to make it easier for contributors who fork and open pr's. Their CI should not test against their own master but against upstream/master
  3. This file diff is then analysed to first determine if only polyfills were updated. If there are other file changes the full test suite is run. If there are too many polyfill changes the full test suite is also run.
  4. The full dependants graph for each modified polyfill is taken to construct the list of features to test.
  5. The list of browsers from Browserstack is filtered to only contain browsers which need at least one polyfill that will be tested.

Notes :

This is a rather big change and I would like to revisit it in the next couple of days with fresh eyes.
Any feedback in the mean time is more than welcome!

If there is a reason not to implement a feature like this at all, that is totally fine too! 🙂


This PR would be a good validation of this change as it's only needed in a few old browser : #991

@origamiserviceuser origamiserviceuser added this to incoming in Origami ✨ Feb 20, 2021
@github-actions github-actions bot added the library Relates to an Origami library label Feb 20, 2021
@@ -172,6 +172,6 @@ module.exports = class TestJob {
}

get configForLog() {
return `${this.mode.padEnd(" ", 8)} / ${(this.polyfillCombinations ? 'combined' : ' ')} / ${(this.shard ? 'shard ' + this.shard + ' ' : 'shard n/a')}`;
return `${this.mode.padEnd(" ", 8)} / ${(this.polyfillCombinations ? 'combined' : ' ')}${(this.shard ? ' / shard ' + this.shard + ' ' : '')}`;
Copy link
Collaborator Author

@romainmenke romainmenke Feb 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only IE8/9 need this, for all other tests it just bloats the logs.

@romainmenke
Copy link
Collaborator Author

@JakeChampion Also added some unit tests to validate the logic and at this point I am generally happy with the state of this change.

@romainmenke romainmenke marked this pull request as ready for review February 23, 2021 21:15
@romainmenke romainmenke requested a review from a team as a code owner February 23, 2021 21:15
@JakeChampion JakeChampion requested review from JakeChampion and removed request for a team February 23, 2021 23:24
Copy link
Owner

@JakeChampion JakeChampion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great!

Origami ✨ automation moved this from incoming to active Feb 27, 2021
@JakeChampion JakeChampion merged commit df5bab7 into JakeChampion:master Feb 27, 2021
Origami ✨ automation moved this from active to complete Feb 27, 2021
@romainmenke romainmenke deleted the partial-tests-in-ci--resourceful-discus-47b0e9c63b branch February 27, 2021 19:48
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 28, 2022
@robertboulton robertboulton removed this from Done in Origami ✨ Jul 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
library Relates to an Origami library
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants