Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: allow marking subtest function as Helper #24128

Closed
larhun opened this issue Feb 26, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@larhun
Copy link

commented Feb 26, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 windows/amd64

What did you do?

Run the following program with "go test":

helper.go:

package helper

helper_test.go:

package helper

import (
	"testing"
)

func TestRun(t *testing.T) {
	run(t, []string{"a", "b", "c"}) // this is line 8
}

func run(t *testing.T, values []string) {
	t.Helper()
	for _, value := range values {
		t.Run("", func(t *testing.T) {
			t.Helper()
			t.Fatal(value) // this is line 16
		})
	}
}

What did you expect to see?

--- FAIL: TestRun (0.00s)
    --- FAIL: TestRun/#00 (0.00s)
    	...\subtest_test.go:8: a
    ...

What did you see instead?

--- FAIL: TestRun (0.00s)
    --- FAIL: TestRun/#00 (0.00s)
    	...\subtest_test.go:16: a
    ...

The reported error line is 16 instead of 8, where the test data are defined. This makes it difficult to use the Run method to define a generic test on a list of cases. I would expect the Run method to be marked as a test helper by default. If the subtesting function is marked as helper, the reported line should be the line of the function that calls the Run method (unless skipped by the Helper mark).

@gopherbot gopherbot added this to the Proposal milestone Feb 26, 2018

@gopherbot gopherbot added the Proposal label Feb 26, 2018

@ALTree

This comment has been minimized.

Copy link
Member

commented Feb 26, 2018

Dup of #23249

@larhun

This comment has been minimized.

Copy link
Author

commented Feb 26, 2018

Not a duplicate. #23249 deals with using Helper within a closure over a testing.TB value. This issue deals with the constraint stated by the documentation of the Run method:

Helper has no effect if it is called directly from a TestXxx/BenchmarkXxx function or a subtest/sub-benchmark function.

@ALTree

This comment has been minimized.

Copy link
Member

commented Feb 26, 2018

It seems to me that the concrete issue you are reporting in this thread is the bad lines in the failure messages

The reported error line is 16 instead of 8, where the test data are defined. This makes it difficult to use the Run method [...]

If the issue reported in the thread I linked above (Helper's highest function call's information should be maintained for any subsequent/nested calls) is fixed in a general way, wouldn't this also fix the problem you are reporting here?

@larhun

This comment has been minimized.

Copy link
Author

commented Feb 26, 2018

Both issues have similar errors, bad lines in the failure messages. However the reasons are very different. Therefore a solution to #23249 may not be a solution to my issue, that basically requires removing the effect of the following lines in the Helper documentation (wrongly reported as Run documentation in my last comment):

Helper has no effect if it is called directly from a TestXxx/BenchmarkXxx function or a subtest/sub-benchmark function.

@ALTree

This comment has been minimized.

Copy link
Member

commented Feb 26, 2018

I see. Leaving this open, then.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 26, 2018

It seems like this should work. The intent of the code is clear, even though it will require a special case at the intersection of subtests and Helper.

/cc @mpvl

@rsc rsc changed the title proposal: testing: make Helper working in a subtest testing: allow marking subtest function as Helper Feb 26, 2018

@rsc rsc modified the milestones: Proposal, Go1.11 Feb 26, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Apr 23, 2018

Change https://golang.org/cl/108658 mentions this issue: testing: allow marking subtest and subbenchmark functions as Helpers

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 7, 2018

Marking release-blocker as there's a pending CL that needs review. /cc @rsc @mpvl

@gopherbot gopherbot closed this in 15f2cbf May 14, 2018

@golang golang locked and limited conversation to collaborators May 14, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.