Only run tests for changed code

[sindresorhus]

Many times you only change code that affects one or a few tests, but still have to suffer through the whole test suite. Would be nice to only run the affected tests instead. This would be a huge win for people with lots of tests.

This is easy on a test file level as we can just recurse the dependency tree and see which test files depend on the changed file. It gets harder for individual tests. Here we could take advantage of code coverage, diff what's changed between runs, and check which lines were affected.

Prior art

• http://dm.gl/2015/01/30/wallaby/
• https://github.com/Genki-S/ttnt
• http://cherrie.io

This is not something that is likely to come in the short term, but opening for discussion and brainstorming. Help making this happen more than welcome.

[schnittstabil]

Nice challenge! But, to clarify: computability theory tell us, that we cannot only take the diff of two source states and calculate which tests are affected.

Possible approaches to solve this:

1. we run the test before the change and gain information from it (we may persist this in some way or other)
2. we use some restrictions, e.g.: if you use this or that naming pattern for your tests, we run the tests in cases of changes

It also means we cannot do it in a perfect way, we would always run tests unnecessarily, the benefit would be to decrease the number of tests to run.

[madbence]

imho just store dependencies of the tests, if some of them changes, we can rerun the tests. there is no better way to do this (that is simple).

[vadimdemedes]

This is the way I solved it in my side-project. Run tests under istanbul, check how coverage changes between tests and create a mapping of test -> depending file line. After the initial test run, which creates a mapping for all tests, I see what lines in my project have changed. And then I run only tests that "touch" those lines.

[Qix-]

@vdemedes that's clever, but still not theoretically sound. We're still dealing with the halting problem here; we have no idea if the program will run differently given different inputs. Keep in mind time is an input.

This means that even though a small code change might only affect one or two tests according to the last run's data, the inputs might change (or the way inputs are interpreted might change) to which it will actually affect many tests - which, given the isolate data (the profiling/coverage data), do not get run. This will almost definitely introduce regressions.

As well, this will require a command line flag for re-running all tests. Removing a cache file or something isn't really user friendly, and now you're storing something in the working directory in order to keep that data. People will have to migrate to this model and include something in their gitignore.

---

Side note: istanbul uses V8's profiler, which can dramatically slow things down. Keep that in mind, too.

[sindresorhus]

and now you're storing something in the working directory in order to keep that data. People will have to migrate to this model and include something in their gitignore.

We would store it in ~/.cache/ava/project-name-from-package-json, not in the project directory.

[Qix-]

What if you change branches in the project?

[sindresorhus]

We would use an LRU cache, so it wouldn't expunge it right away when switching branches.

[Qix-]

[Qix-]

[Qix-]

sigh and not even a history to prove that I'm not a potato. Thanks Github.

#BDFLAbuse #PotatoLivesMatter

[schnittstabil]

We're still dealing with the halting problem here; we have no idea if the program will run differently given different inputs.

We should not be worrying too much about obfuscated code, it should only be reasonable and practical…

require(new Date().getTime() === new Date(2017,3,16).getTime() ? './upload-genisys.js' : './');

[Qix-]

@schnittstabil not quite sure what you mean there.

[schnittstabil]

@Qix- I think it is ok to only consider code patterns usually used. As you already mentioned, this feature can not be theoretically sound.

[Qix-]

But by even introducing the possibility of skipping tests that could have been affected by change, you're introducing a chance of introducing regressions into your code base. Further, it becomes the test suite's fault - not something a test suite should do. Ever.

It's a sweet idea though.

[sindresorhus]

Wallaby.js somehow did it, though.

[schnittstabil]

Clearly this feature shouldn't be preset. I would imagine a --quick CLI flag, or similar, --help may note its inadequacies…

We may also use this in some watch mode: We run the possible affected test first and run the remaining tests afterwards.

[Qix-]

+1 for watch mode. That's a great idea, @schnittstabil. However, I think the functionality should be initial run of watch-mode runs them all, and then subsequent triggers as it's watching just does the affected tests. THAT would be cool.

[sindresorhus]

Clearly this feature shouldn't be preset. I would imagine a --quick CLI flag, or similar, --help may note its inadequacies…

👍

Yes, the intention is to combine it with some kind of watch mode. That's where it would shine. Being able to code and see the test results almost live.

[ArtemGovorov]

Wallaby.js core developer here, here's our 2c of experience with it:

As it has been mentioned a few times in the discussion, there's no theoretically waterproof solution. We use runtime analysis powered by instrumentation collected data plus a bazillion of heuristic rules of various nature, plus framework specific hacks, plus additional data received directly from a specific code editor that we integrate with. So while a simple dependency/call tree analysis covers like 70% of cases, the rest 29.9999% is really painful to cover.