cmd/go: cache test results in go test

[rsc]

It would be nice to have a mode for go test that cached test results, so that you could change a package and then do 'go test ...' and have it only actually rerun the tests that depend on that package.

This comes up especially in larger trees.

This is not trivial.

@thockin @joeshaw

[natefinch]

This would be amazing for juju. Our tests take on the order of 5-10 minutes depending on the speed of your hardware.

[thockin]

Worth adding another request at the same time: some tests have test data -
tests should re-run when that data changes.

For a fun but complex example, one of our tests parses example JSON and
YAML from our docs in the source tree and tries to validate it.
On Jun 12, 2015 4:37 PM, "Nate Finch" notifications@github.com wrote:

This would be amazing for juju. Our tests take on the order of 5-10
minutes depending on the speed of your hardware.

—
Reply to this email directly or view it on GitHub
#11193 (comment).

[davecheney]

What hardware are you running, it's more like 1/2 an hour for me, even the Ci machines with 8 cores can't do it in under 18

On 13 Jun 2015, at 09:37, Nate Finch notifications@github.com wrote:

This would be amazing for juju. Our tests take on the order of 5-10 minutes depending on the speed of your hardware.

—
Reply to this email directly or view it on GitHub.

[natefinch]

@davecheney
real 7m49.011s
user 33m29.257s
sys 4m54.547s

2013 XPS 15 (9530)
Quad Core i7
16 GB RAM
Upgraded drive to a Samsung 850 Pro 512GB SSD

It's probably the drive that makes it so much faster for me. I also run with gomaxprocs=8.

[rsc]

I wrote rsc.io/gt for this as a first draft. It seems okay, but if we're going to go down this route we need to capture all the relevant context - external files, environment variables, and so on - in the cache key. That will require significantly more design.

[mackstann]

It would be impossible to automatically track down all possible dependencies on external state (e.g. database contents, network interactions), so it seems like a line must be drawn in the sand, and users must understand where that line is and how it determines when they can rely on the caching mechanism.

Right now, it looks like it only hashes the source files, and to me that seems like a simple and easy to understand demarcation. It may be crude, but that also means it's simple and easy to remember.

Our tests take about 3.5 minutes and getting that down to a small fraction for incremental changes is really nice. I'm going to start using gt regularly and see how it goes. So far, so good.

[gkop]

Without this enhancement, Go is adverse to TDD. Just adding dependencies on gorm and sqlite adds a 20s overhead to every go test run, on modern hardware.

[davecheney]

Gabe, this is because you are recompiling sqlite every time. Use go install
or go test -i to build all test dependencies.

On Thu, 30 Jun 2016, 09:45 Gabe Kopley notifications@github.com wrote:

Without this enhancement, Go is adverse to TDD. Just adding dependencies
on gorm and sqlite adds a 20s overhead to every go test run, on modern
hardware.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#11193 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA5ETRBw1-LHzy2sIkvQsSztg5DcKks5qQwOggaJpZM4FBiez
.

[gkop]

Whew, thanks Dave! We're all good :)

[mewmew]

Right now, it looks like it only hashes the source files, and to me that seems like a simple and easy to understand demarcation. It may be crude, but that also means it's simple and easy to remember.

I would second this. Taint analysis of input variables and tracking external state changes is not simple.

Rather than making the implementation of test case output caching very complex, I'd much rather have a simple underlying principle which is easy to reason about for when test cases are re-run.

And, hashing of source file content is just that, simple and still rather effective.

If the file path of a testdata file change, the test would be re-run. If the URL of a test case changes, the test would be re-run. Easy to reason about and still effective.

A solution which attempts to track what files are actually used in a test case is bound to miss some, and thus making it difficult to reason about.