Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Helper's highest function call's information should be maintained for any subsequent/nested calls #23249

Open
mattcary opened this issue Dec 26, 2017 · 9 comments
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@mattcary
Copy link

mattcary commented Dec 26, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.9.2 android/arm64

Does this issue reproduce with the latest release?

Yes (I think 1.9.2 is the latest release)

What operating system and processor architecture are you using (go env)?

GOARCH="arm64"
GOHOSTARCH="arm64"
GOHOSTOS="android"
GOOS="android"

What did you do?

Run the following program with "go test". I would have hoped that the reported error line is in the 2nd call to checkFor3s() in TestHelper(). Instead, the error line is in forEach() (line 9 for me but I think some blank lines may have been eaten below.

I think the problem is that checkFor3s() uses the method forEach which is not marked with t.Helper(). In a more realistic case, forEach would be a non-testing utility method and so would not have the possibility of calling t.Helper().

Perhaps the semantics of t.Helper should be that the displayed error line is in the stack frame above the highest t.Helper() call? That way this example would be fixed, and also it would not be necessary to sprinkle t.Helper() calls everywhere if there are complex testing utility functions.

package helper                                                                                                                 

import(
        "testing"
)                                                                                                                               
func forEach(data []int, f func(int)) {
        for _, i := range data {
                f(i)
        }
}

func checkFor3s(tb testing.TB, data []int) {
        tb.Helper()
        forEach(data, func(i int) {
                tb.Helper()
                if i != 3 {
                        tb.Error("Saw", i, "instead of 3")
                }
        })
}

func TestHelper(t *testing.T) {
        checkFor3s(t, []int{3, 3, 3, 3})
        checkFor3s(t, []int{3, 3, 2, 3})
        checkFor3s(t, []int{3, 3, 3, 3})
}
@odeke-em
Copy link
Member

Hello there @mattcary, thanks for reporting this. So if I understood your issue right, it is that the line number reporting is inaccurate right? That on Go1.9.2 you are getting the error reported on line 9

func forEach(data []int, f func(int)) {
        for _, i := range data {
                f(i)
        } // <--- right here

Please try with Go1.10 beta, the line numbers now seem to report on line 8

func forEach(data []int, f func(int)) {
        for _, i := range data {
                f(i) // <--- right here
        }
$ go test -v
=== RUN   TestHelper
--- FAIL: TestHelper (0.00s)
	tf_test.go:8: Saw 2 instead of 3
FAIL
exit status 1
FAIL	_/Users/emmanuelodeke/Desktop/openSrc/bugs/golang/23249	0.006s

@mattcary
Copy link
Author

Thanks for your reply Emmanuel.

Just downloaded the beta, it reports the error at the call to f(i) identically in 1.9.2 as in 1.10beta1.

However, the issue is that the line number is in the wrong function altogether: it should report an error in TestHelper(), not in forEach() at all. Putting a call to tb.Helper in forEach() makes the line number display correctly:

func forEach(tb testing.TB, data []int, f func(int)) {
tb.Helper()
for _, i := range data {
f(i)
}
}

But this requires plumbing tb through, which would not be possible if forEach were a non-testing utility method.

Happy New Year! :)

@odeke-em
Copy link
Member

Thank you @mattcary for the clarification, and happy new year to you too!

Oh I see, so if I may paraphrase, your issue is that you'd like *testing.B.Helper() and *testing.T.Helper() to propagate/nest the behavior(using the parent function's file and line information), to any callers that are any call-level after the original marking?

I'll also page @cespare and @ianlancetaylor.

@odeke-em odeke-em changed the title t.Helper and unmarked utility functions testing: Helper should maintain file/line printing behavior even with nested function calls Dec 27, 2017
@mattcary
Copy link
Author

Yes, something like that. I was thinking that the highest T.Helper() call in the stack should have as parent the testing line we want to display as an error.

Thx

@odeke-em
Copy link
Member

Awesome, thank you for the clarification @mattcary, I'll re-amend the title for even more precision, please feel free to edit it.

@odeke-em odeke-em changed the title testing: Helper should maintain file/line printing behavior even with nested function calls testing: Helper's highest function call's information should be maintained for any subsequent/nested calls Dec 27, 2017
@cespare
Copy link
Contributor

cespare commented Dec 27, 2017

With the suggested change, in order to decide whether or not to mark a method as a helper, you need to know whether there is a helper higher up in the call chain. In the given code, for instance, forEach will only be ignored as a helper if it is called from a helper; if you call forEach directly from TestHelper, then you'll have the same issue where the printed line number will be inside forEach.

With t.Helper's existing behavior, the rule is simple: if you want a function to be ignored when go test prints file:line, mark it as a helper. You can make this choice function-by-function without ever considering the full set of call paths that reach the function.

Additionally, I think it's too late to change this behavior because t.Helper is in a released version of Go.

@mattcary
Copy link
Author

mattcary commented Jan 3, 2018 via email

@mattcorbyeaglen
Copy link

I too am having this issue and it would be very useful for Helper() to do this.

@ianlancetaylor ianlancetaylor added the NeedsFix The path to resolution is known, but the work has not been done. label Mar 28, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Mar 28, 2018
@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jun 29, 2018
@karlkfi
Copy link

karlkfi commented Jul 29, 2024

An alternate way to work around this issue would be to do what Ginkgo does and allow Helper to specify how many stack frames to skip. This allows a helper function that knows it's going to be called by a non-helper function to specify that it wants to skip more frames. This would also allow *testing.B.Helper() and *testing.T.Helper() to be wrapped with abstractions without breaking Helper.

https://github.com/onsi/ginkgo/blob/master/types/code_location.go#L98

This also has the benefit that it can be added to Helper in a reverse compatible way to using a vararg.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

6 participants