Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Separate apps tests into unit and integration tests #8504
Re-opening #8467 against staging-next.
Breaks apps tests into two tests suites: Unit tests (faster, fewer dependencies, less setup/teardown) and integration tests (slower, more dependencies, more setup/teardown). Lots of things change here; I'll try to break down the changes so they're easy to understand.
As you've probably guessed, the unit tests are in test/unit and the integration tests are in test/integration. Lots of supporting materials have been moved too.
Unit tests treat every file in the unit folder as a test (i.e.
Integration tests treat every file in the root of the integration folder as a test (i.e.
There are lots of ways to invoke apps tests, and they've changed at most layers. I'll go through the layers one-by-one to explain the changes.
There are two new commands
These tasks accept all the same arguments as before. One improvement - the
I took a lot of code out of testUtils.js. Some of it moved into utility files specifically for the integration tests, and some of it went to other, more specific utility files (see configuredChai.js) and some of it just evaporated because tests could include what they needed directly.
This is useful because we were introducing lots of dependencies to tests that didn't need them just by requiring testUtils.js. As a general practice moving forward, if you need a utility around a particular dependency (like Chai) it should become its own utility file so we avoid lumping dependencies together this way - that will help keep the bundle time down for unit tests.
Unfortunately, this makes running all of our apps tests more than 20 seconds slower. This is largely due to building two test bundles instead of one.
I think that slowdown will be more than compensated for by the iteration speedup when working on unit tests. Building and running the unit tests by themselves now takes about a minute, and when using the
There's also a small advantage to running all of the unit tests before all of the integration tests: Assuming we break tests with a fairly normal distribution, we'll learn about breakages faster on average if we run the faster tests first. It might seem like a small thing, but CircleCI failing faster may pay off in the long run.
The real breakthrough on the speed of running tests, though, is going to be incremental builds for the test bundles. That's future work. I'm hoping this big change will make it easier to get that working.
Running tests in CI
We already run apps tests in two chunks on CircleCI. Before, it was non-applab tests followed by applab tests. We did this as a workaround to a problem where we were running out of memory during our unit tests.
Here, we replace "non-applab" and "applab" with "unit" and "integration." Tests are passing on CircleCI and memory use doesn't seem to be a problem - and anyway, we have more than twice as many unit tests as integration tests, and now they load a lot less stuff. We still run them in order, and do up to one retry on the individual tasks.
I think this is valuable from a maintenance point of view, just because it's conceptually a simpler divide. It's also a lot closer to the way we usually run tests in development.