Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: add "-integration" flag for running integration tests #46409

Closed
DmitriyVTitov opened this issue May 27, 2021 · 6 comments
Closed

testing: add "-integration" flag for running integration tests #46409

DmitriyVTitov opened this issue May 27, 2021 · 6 comments

Comments

@DmitriyVTitov
Copy link

@DmitriyVTitov DmitriyVTitov commented May 27, 2021

Go testing framework has only one pre-built flag "-test.short" which is not very clear for me. Often I write integration tests which communicate with external dependencies such as databases and API across the network. These tests should not run along with unit tests.
I would like to have built-in flag "-test.integration" which works like "-test.short" to use within integration tests like this:

	if !testing.Integration() {
		t.Skip("use '-integration' flag to run this test")
	}

English is not my native so maybe name is not the best, but I'm sure we need such flag.

@mvdan
Copy link
Member

@mvdan mvdan commented May 27, 2021

Have you seen threads like https://stackoverflow.com/questions/25965584/separating-unit-tests-and-integration-tests-in-go?

You already have multiple ways to separate integration tests:

  • If they're not too tricky/slow to run, you can just make them "long" tests via -short.
  • If they are tricky or very slow, you can make them opt-in via a custom flag or an environment variable.
  • A variation on the last method is to put them in a package not covered by go test ./..., such as a nested module.

I don't think a "blessed" and built-in way to do this is a good idea - everyone's integration tests have different requirements. I imagine most people's will require extra custom steps anyway, like installing dependencies or setting up a daemon, so I don't imagine that the consistency of go test -integration will help.

Loading

@DmitriyVTitov
Copy link
Author

@DmitriyVTitov DmitriyVTitov commented May 27, 2021

I use short flag in opposite meaning not to run integration tests. Really I don't get precise meaning of "-short" flag and I thought it would be super simple to add custom flag for integrations tests. I understand that I should not ask for change for every minor reason, but this is part of testing methodology.
I know that I can separate tests, but why should I use build tag (they affect whole file) or custom flag (I would need to parse flags explicitly)? Isn't it much easer to use built-in flag?
There are 1-2 levels of tests above unit testing. So we could have 1-2 flags for them.

Loading

@mvdan
Copy link
Member

@mvdan mvdan commented May 27, 2021

Isn't it much easer to use built-in flag?

Adding more built-in flags is not easier, you add more edge cases that we need to take care of and document.

For example, what should go test -short -integration do? What about go test -integration -bench=xxx?

I would need to parse flags explicitly

You can add var integration = flag.String(...) in just one line. You don't even need to call flag.Parse for tests.

There are 1-2 levels of tests above unit testing. So we could have 1-2 flags for them.

go test is only concerned about running tests - it does not attempt to classify them between "unit", "integration", etc.

Loading

@DmitriyVTitov
Copy link
Author

@DmitriyVTitov DmitriyVTitov commented May 27, 2021

I don't understand about edge cases. What should -short -integration do? Nothing, it's just flags.
But OK, if you think this is inappropriate - close the issue. I would use global vars for custom flags in every package.

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented May 27, 2021

See previously: #30595, #31310, #34707.

(In particular, this looks like a near-exact duplicate of #30595.)

Loading

@mknyszek mknyszek changed the title go test: add "-integration" flag for running integration tests cmd/go: add "-integration" flag for running integration tests May 27, 2021
@mknyszek mknyszek added this to the Backlog milestone May 27, 2021
@mknyszek mknyszek changed the title cmd/go: add "-integration" flag for running integration tests testing: add "-integration" flag for running integration tests May 27, 2021
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented May 27, 2021

Closing as dup. Sorry. Please comment if you disagree.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants