Gradle Test runs should not be cacheable by default

[runningcode]

Tests, especially integration tests, may rely on eternal inputs such as time, date, external files, web resources. If the test runs themselves are cached, changes which cause the tests to fail will be hidden. Choosing to cache tests as the default choice is very dangerous because tests may pass when they should fail. Many projects do no keep up to date with release notes and this change may go unnoticed.

We have added the following workaround to our project to disable test caching:

tasks.withType(Test).configureEach {
  outputs.upToDateWhen { false }
}

Current Behavior:

Test task cacheability is on by default.

Proposed change:

Test task cacheability is off by default and opt in.

[realdadfish]

While I agree that there should be an easy, visible configuration to disable test caching per module, I would not make test caching overall default to false. Reason is that especially in multi-module projects a lot of the module tests are pure, isolated JUnit tests that do not have to be run again if no code changes where detected in either the source or the test file sets. We depend for example on incremental builds and tests for our PR branches, where both, compilation and test avoidance, keeps the times for individual rebuilds low.

[tasomaniac]

I agree. It's super valuable to cache them assuming that they are islolated (hermetic?) tests.

In case they are not hermetic (do network request), you can encapsulate all those tests in a separate module and use --rerun-tasks flag to fix your issue.

[runningcode]

It's a really common pattern to include files outside of the build source folder in integration tests.

For example: https://github.com/square/sqldelight/blob/master/sqldelight-gradle-plugin/src/test/kotlin/com/squareup/sqldelight/GradlePluginCombinationTests.kt

This test task does not declare the file dependencies.gradle as an input. Changes to dependencies.gradle like for example deleting versions.compileSdk would not cause the test to be re-run.

You raise a good point about build speeds. I guess it depends on what the priorities are for Gradle as a build system. While speed is extremely important for me as well, I would personally prefer a build system which favors correctness and reproducibility over speed. Dangerous potential speed improvements should be opt-in not opt-out.

[oehme]

This is true for any task that has some dynamic nature, even including the Java compiler (annotation processors could run arbitrary code). Also, this problem is not limited to caching. Not declaring your inputs will also break up-to-date checks. If we'd follow your line of reasoning, almost nothing would would be up-to-date or cacheable out of the box - a terrible first impression.

That's why when you do things outside of the box, you need to declare the additional inputs. This is really easy by using task.inputs. There's no need to do upToDateWhen {false}.

[lptr]

@runningcode it is possible to make that integration test cacheable and still get it re-executed when dependencies.gradle changes, you only have to declare it using Task.getInputs().file().

That said, it can be problematic not to forget to include extra inputs for integration tests. We are working on a plan to add something like a "verify mode" that would check/ensure that only files declared as inputs would be available to your tasks. In this case it would mean that your integration test would fail if you didn't include dependencies.gradle as its input.

[runningcode]

Closing in favor of #9210