testing: add "-integration" flag for running integration tests

[DmitriyVTitov]

Go testing framework has only one pre-built flag "-test.short" which is not very clear for me. Often I write integration tests which communicate with external dependencies such as databases and API across the network. These tests should not run along with unit tests.
I would like to have built-in flag "-test.integration" which works like "-test.short" to use within integration tests like this:

	if !testing.Integration() {
		t.Skip("use '-integration' flag to run this test")
	}

English is not my native so maybe name is not the best, but I'm sure we need such flag.

[mvdan]

Have you seen threads like https://stackoverflow.com/questions/25965584/separating-unit-tests-and-integration-tests-in-go?

You already have multiple ways to separate integration tests:

• If they're not too tricky/slow to run, you can just make them "long" tests via -short.
• If they are tricky or very slow, you can make them opt-in via a custom flag or an environment variable.
• A variation on the last method is to put them in a package not covered by go test ./..., such as a nested module.

I don't think a "blessed" and built-in way to do this is a good idea - everyone's integration tests have different requirements. I imagine most people's will require extra custom steps anyway, like installing dependencies or setting up a daemon, so I don't imagine that the consistency of go test -integration will help.

[DmitriyVTitov]

I use short flag in opposite meaning not to run integration tests. Really I don't get precise meaning of "-short" flag and I thought it would be super simple to add custom flag for integrations tests. I understand that I should not ask for change for every minor reason, but this is part of testing methodology.
I know that I can separate tests, but why should I use build tag (they affect whole file) or custom flag (I would need to parse flags explicitly)? Isn't it much easer to use built-in flag?
There are 1-2 levels of tests above unit testing. So we could have 1-2 flags for them.

[mvdan]

Isn't it much easer to use built-in flag?

Adding more built-in flags is not easier, you add more edge cases that we need to take care of and document.

For example, what should go test -short -integration do? What about go test -integration -bench=xxx?

I would need to parse flags explicitly

You can add var integration = flag.String(...) in just one line. You don't even need to call flag.Parse for tests.

There are 1-2 levels of tests above unit testing. So we could have 1-2 flags for them.

go test is only concerned about running tests - it does not attempt to classify them between "unit", "integration", etc.

[DmitriyVTitov]

I don't understand about edge cases. What should -short -integration do? Nothing, it's just flags.
But OK, if you think this is inappropriate - close the issue. I would use global vars for custom flags in every package.

[bcmills]

See previously: #30595, #31310, #34707.

(In particular, this looks like a near-exact duplicate of #30595.)

[ianlancetaylor]

Closing as dup. Sorry. Please comment if you disagree.