proposal: testing: separate integration and unit tests

[nim-nim]

$ go version
go version go1.11 linux/amd64

Right now go mixes integration and unit tests in a single category.

Unfortunately they are not the same: a unit test is something that only needs the go compiler to run, and an integration test needs some form of specific environment (running system daemon, database, working network, server, cloud, credentials, etc) to be executed.

That severely limits the ability of third parties to check existing Go codebases, since after the nth test that can not be run because of specific constrains, you just tend to give up, disable them all and pray for the best.

Separating unit and integration tests would incite third parties to actually run unit tests and would improve testing coverage, since third parties have often access to a wider range of systems and architectures than the original project authors.

[mvdan]

There are many ways to do this; for example, see https://stackoverflow.com/questions/25965584/separating-unit-tests-and-integration-tests-in-go.

I personally prefer a mix of testing.T.Skip and testing.Short. That way, -short doesn't run integration or slow tests, and the integration tests are skipped when the external component isn't available.

Like in the other issues, if you have a particular enhancement in mind, please file a proposal; this reads more like a complaint than a suggestion for a particular change to the testing package.

[bcmills]

There is a broad spectrum between “integration” and “unit” tests. Ideally, tests should start their own (hermetic) dependency services, and then you can run the tests even in a completely isolated environment. (Does that make them “unit tests”? I don't know, and I don't think the distinction is particularly meaningful.)

When that isn't possible, it is still usually possible for tests to probe for the existence of dependencies and call t.Skip if they are missing.

[andybons]

@nim-nim what do you propose we do (concretely) as a solution to your problem?

[nim-nim]

@bcmills

Yes there is a broad spectrum of tests, but practically there are only two cases:

• things which are safe to run by third-parties as-is (the test only needs the compiler, and if it needs something else like network, server of whatever it tests for the existence of this something else and PASSES if not available)
• things which are not safe and need special preparation/handling for the result to be meaningful (that will generate spurious FAILS if the preparation/handling/test env is not done right)

It would be very useful to have the ability to mark those two categories of tests in a different way (via naming conventions like _integration_test postfixes, source annotations, whatever) so go list could put both in separate buckets and one could run something like go test all -safe by default, knowing its result is meaningful and does not require sorting out integration tests first.

Right now when a test run is polluted by integration things, and upstream does not want to bother testing for the things the integration test needs, you have no way to ask them to just mark it as integration so everyone is happy and plays nice with the others

[ianlancetaylor]

In my opinion, tests that require special handling should define their own flag and should only be run if that flag is passed. If the flag is not passed, they should call t.Skip.

I don't see a reason to handle this explicitly in the testing package. There are too many different requirements to satisfy.

[nim-nim]

But test writers are not that careful in the real world. They don't want to code all that testing (in their own setup, integration tests just work).

They need a cheap standard way to mark "this test is not careful about its requisites and I can't be bothered to fix it"

[ianlancetaylor]

Adding and checking a flag is literally four lines of code, and one of those lines is }.

Adding functionality to the testing package is just more API that very few people will use.

[mvdan]

But test writers are not that careful in the real world.

Writing bad tests is certainly possible, but adding more API to the testing package is unlikely to improve that situation. If someone doesn't bother to write good tests today, they're even less likely to take the time if we make the testing package more complex.