Automate canary testing changes with downstream packages

[ChristianMurphy]

Subject of the feature

Automate canary testing changes with downstream packages

Prior art: https://github.com/nodejs/citgm

Problem

Unified is a large and highly connected community of packages.
Changes to any package could have impact downstream packages.
Currently anticipating impact is a purely manual process.

It could be helpful to use downstream packages, which themselves have comprehensive test suites, to help automatically detect what changes may be breaking or have unintended impacts.

Expected behaviour

The flow could work something like:

1. add a dependency on the new downstream testing tool
2. a package adds a json config file with:
   • a description of the build process for the package
   • a listing of trusted downstream packages with unit tests
3. add an npm test script that calls the downstream testing package

From then on, for each run of the continuous integration process the downstream testing tool would

1. build the package using the instructions from the configuration
2. download each downstream package listed in the configuration
3. for each package downloaded
   1. setup that package (npm install)
   2. link the build of the package being updated, into the downstream package's dependencies
   3. run the test suite of the downstream package
   4. report results

Alternatives

• manual testing (which is needed anyway)
• Find an existing tool like https://github.com/nodejs/citgm but for node modules

[wooorm]

Could something like https://github.com/juliangruber/test-npm-dependants help as well?

[ChristianMurphy]

Absolutely!

The underlying runner could be reused, a few tweaks we may want:

• use a specific list of packages, rather than running all dependents
• allow options to be stored in a configuration file, rather than requiring CLI args

both seem to be covered by juliangruber/test-npm-dependants#2

[wooorm]

for each package downloaded

I’m hoping that projects have test suites that pass, but maybe not, in which case it would be worth testing both before and after linking, and comparing the results.

[wooorm]

What would happen if package A depends on package B, both are set up to test 25 of their users in npm test. Are you now testing 50 packages?

[ChristianMurphy]

I’m hoping that projects have test suites that pass, but maybe not, in which case it would be worth testing both before and after linking, and comparing the results.

It does compare previous and current results.
It may require some changes to support a branch/local folder, logged as juliangruber/test-npm-dependants#7

What would happen if package A depends on package B, both are set up to test 25 of their users in npm test. Are you now testing 50 packages?

Another good question, currently I think it would, logged as juliangruber/test-npm-dependants#8

[ChristianMurphy]

Thanks for starting the discussion @ChristianMurphy!
We're in the process unifying ideas in with discussions unifiedjs/collective#44
If you'd like to continue this thread, or start a new one https://github.com/unifiedjs/unified/discussions will be the home for ideas going forward.