proposal: testing: add Name to track file and line of test case declaration #52751
Comments
This is similar to the workaround we use in the stdlib, e.g., |
|
That said, it would be nice to have something ergonomic built in to |
|
To avoid declaring new for _, tt := range tests {
- t.Run(tt.name, func(t *testing.T) {
+ t.Run(tt.name.String(), func(t *testing.T) {
...
})
}where |
|
It seems like there are two things going on here:
It seems like we could keep t.Run the same, pass tt.name.String() to it, and then separately have a call at the start of the function body. Then other sources of file:line (like testscript failures) can hook into this too. |
|
This proposal has been added to the active column of the proposals project |
|
I have encountered this problem a few times. I'll share the workaround I came up with in case it helps this proposal in some way. Using To make this work, the traditional test case "table" has to be modified a bit.
Using the example test from the description, it might look like this: type testCase struct {
name string
input T
...
}
run := func(t *testing.T, tc testCase) {
t.Helper()
t.Log("case:", tc.name)
t.Run(name, func(t *testing.T) {
got, err := fizz(tt.input)
if err != nil {
t.Fatalf("fizz error: %v", err) // my_test.go:1234
}
})
}
run(t, testCase{
name: "Foo",
...,
}
run(t, testCase{
name: "Bar",
...,
}
... // maybe dozens or hundreds more casesDefinitely not as elegant as the |
|
I'd still like to understand better whether we can separate out the two different features being added, as I noted in #52751 (comment). Any thoughts, @dsnet? |
|
Since new type is introduced maybe there is no need to have package go52751
import (
"fmt"
"runtime"
"testing"
)
type TC struct {
name string
location string
}
func (tc *TC) Run(t *testing.T, tf func(t *testing.T)) bool {
t.Helper()
return t.Run(tc.name, func(t *testing.T) {
t.Helper()
// this should use internal undecorated logging to achieve desired output
t.Logf("Test case %q defined at %s", tc.name, tc.location)
tf(t)
})
}
func testingCase(name string) *TC {
_, file, line, _ := runtime.Caller(1)
return &TC{name, fmt.Sprintf("%s:%d", file, line)}
}
func TestFileLine(t *testing.T) {
tests := []struct {
tc *TC
input string
}{{
tc: testingCase("Foo"),
input: "x",
}, {
tc: testingCase("Bar"),
input: "",
},
}
for _, tt := range tests {
tt.tc.Run(t, func(t *testing.T) {
if tt.input == "" {
t.Fatalf("input error")
}
})
}
} |
|
@AlexanderYastrebov, you would need two different |
diff --git a/src/testing/testing.go b/src/testing/testing.go
index ec2d864822..dc1bfc301e 100644
--- a/src/testing/testing.go
+++ b/src/testing/testing.go
@@ -529,6 +529,8 @@ type common struct {
tempDir string
tempDirErr error
tempDirSeq int32
+
+ cases map[string]string
}
// Short reports whether the -test.short flag is set.
@@ -1451,6 +1453,27 @@ func tRunner(t *T, fn func(t *T)) {
t.mu.Unlock()
}
+func (t *T) Case(name string) string {
+ _, file, line, _ := runtime.Caller(1)
+ location := fmt.Sprintf("%s:%d\n", file, line)
+
+ t.mu.Lock()
+ if t.cases == nil {
+ t.cases = make(map[string]string)
+ }
+ uniqName := name
+ for i := 1; ; i++ {
+ if _, ok := t.cases[uniqName]; !ok {
+ break
+ }
+ uniqName = fmt.Sprintf("%s#%d", name, i)
+ }
+ t.cases[uniqName] = location
+ t.mu.Unlock()
+
+ return uniqName
+}
+
// Run runs f as a subtest of t called name. It runs f in a separate goroutine
// and blocks until f returns or calls t.Parallel to become a parallel test.
// Run reports whether f succeeded (or at least did not fail before calling t.Parallel).
@@ -1463,6 +1486,15 @@ func (t *T) Run(name string, f func(t *T)) bool {
if !ok || shouldFailFast() {
return true
}
+
+ if loc, ok := t.cases[name]; ok {
+ fo := f
+ f = func(t *T) {
+ t.Helper()
+ t.Logf("case at %s", loc)
+ fo(t)
+ }
+ }
// Record the stack trace at the point of this call so that if the subtest
// function - which runs in a separate stack - is marked as a helper, we can
// continue walking the stack into the parent test.package go52751
import (
"testing"
)
func TestFileLine(t *testing.T) {
tests := []struct {
name string
input string
}{{
name: t.Case("Foo"),
input: "x",
}, {
name: t.Case("Bar"),
input: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if tt.input == "" {
t.Fatalf("input error")
}
})
}
} |
|
@dsnet, any thoughts on #52751 (comment) ? |
|
In regards to #52751 (comment), it seems odd if the test name is annotated with file:line information if we can directly use the test name and file:line information together. There's a discrepancy between how creation and usage operates. If usage is segregated, then creation should also be segregated: tests := struct {
name string
+ location testing.SourceLocation
input T
...
} {{
name: "Foo",
+ location: testing.FileLine(),
...,
}, {
name: "Bar",
+ location: testing.FileLine(), // my_test.go:321
...,
}
... // maybe dozens or hundreds more cases
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
+ t.Annotate(tt.location)
got, err := fizz(tt.input)
if err != nil {
t.Fatalf("fizz error: %v", err) // my_test.go:1234
}
})
}This approach is more flexible as you can imagine using this for more purposes than just annotating test case names. In response to #52751 (comment), one significant detriment to |
|
@dsnet, you jumped to testing.SourceLocation, which I didn't really have in mind. I think testing.Name returning something that has a name and a file:line is fine. But there are other tests reading test data files that might also want to set the file:line, and I was suggesting that they could use that mechanism too if it were split out. I don't think we need a SourceLocation type though. So only the t.Annotate would be added in the diff, not all the testing.FileLine() calls. |
|
Ping @dsnet |
|
I tried out a version of this in one of my test cases, and it seems that at least GoLand IDE does not highlight multiple I'm less familiar with other environments, but I expect even in environments that don't provide hyperlinks to the source it wont be obvious to someone running the tests what each of the two
I agree this is what is missing, and I think this would be more valuable if it worked similar to #52751 (comment). Instead of a second As multiple people have mentioned on this thread, capturing the file and line number is pretty easy to do, and could be done in different ways depending on the test. If location := fileLine()
...
t.Println(fmt.Sprintf("%v: test case definition", location))Where func (c *common) Println(args ...any) {
... // handle c.done, and c.chatty branches
c.output = append(c.output, fmt.Sprint(args...))
}The output from the original proposal would look something like this: Edit: it is possible to use |
It's unfortunate that GoLand doesn't adequately hyperlink
I'm starting to lean towards this approach. I haven't quite come to terms with how much this should be in (I'm going to go on a tangent now regarding logging but it is related) The other day I was dealing with log functions and composing them together. In my use case, I had a func multiLogf(logfs ...func(string, ...any)) func(string, ...any) {
return func(f string, a ...any) {
for _, logf := range logfs {
logf(f, a...)
}
}
}The problem with this is that One way to address this is to have Imagine there was a: package runtime
// FileLine returns the file:line of the caller.
// The file is only the base file name.
func FileLine() stringthen I could have done: logf("%s: payload too large for %v", runtime.FileLine(), id)Similarly, we can accomplish what this proposal was originally about with: tests := struct {
+ where string
name string
input T
...
} {{
+ where: runtime.FileLine(),
name: "Foo",
...,
}, {
+ where: runtime.FileLine(),
name: "Bar",
...,
}
... // maybe dozens or hundreds more cases
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
+ t.Logf("%s: test case declaration", tc.where)
got, err := fizz(tt.input)
if err != nil {
t.Fatalf("fizz error: %v", err)
}
})
}Having a helper function in My concern with point 2 in #52751 (comment) is that I'm uncertain how this information should be presented in the terminal UI and also what happens when That said, I'd be okay even with something like |
|
I don't understand why we'd add a Println etc that does not log the source file line. If the caller is going to add their own explicitly, there's no harm in having two file:line in the output. I have tests that do that (for example cmd/go TestScript) and it works just fine. You get the source line of the log message as well as the source line of the corresponding data and can click on either one depending on what you are looking for. Suppose we had which adds that file:line to the list of file lines that get printed. For example if you do the message would look like That seems generally useful, and I would probably use it in TestScript instead of the manual prints that I have today. Then the remaining question is how to make it easy to get a file,line to pass to Source for a table-driven test. Suppose we defined Then we could do something like: It seems to me that this separation provides two generally useful concepts that work well apart as well as together, instead of tying them together into one specialized concept. Thoughts? |
|
Overall, SGTM. Some questions about semantics:
|
|
Let's try to answer those questions:
Thoughts? |
|
It seems like people are mostly happy with #52751 (comment)? |
|
On hold for an implementation to sanity check this idea. |
|
Placed on hold. |
|
Change https://go.dev/cl/444195 mentions this issue: |
|
I sent out CL 444195 as a prototype implementation. Example usage: func Test(t *testing.T) {
tests := []struct {
testing.Marker
in int
want string
}{
{testing.Mark("Foo"), 1, "1"},
{testing.Mark("Bar"), 2, "2"},
{testing.Mark("Baz"), 3, "3"},
{testing.Mark("Giz"), 4, "4"},
{testing.Mark("Daz"), 5, "5"},
}
for _, tt := range tests {
t.Run(tt.Name, func(t *testing.T) {
t.Source(tt.Source())
got := strconv.Itoa(2 * (tt.in / 2)) // faulty conversion
if got != tt.want {
t.Errorf("Itoa(%d) = %v, want %v", tt.in, got, tt.want)
}
})
}
}Example output: where lines 16, 18, 20 are where the test case was declared, Overall, I generally like the API except for the following:
Alternatively, the originally proposed API of have a new func (*T) RunMarker(Marker, func(*T))since each instantiation of |
|
I think my ideal API for this continues to be something like: type Label struct {
Name string
File string
Line int
}
func Mark(name string) Label
func (*T) RunLabel(Label, func(*T))
func (*B) RunLabel(Label, func(*B))As a bikeshed, renaming |
Why not |
It could be either two methods on one type: type TC struct {
Name string
File string
Line int
}
func Case(name string) TC
func (tc TC) Test(t *T, f func(t *T)) bool
func (tc TC) Benchmark(t *B, f func(t *B)) boolor two case types: func Case(name string) TC
func Benchmark(name string) TBC
func (tc TC) Run(t *T, f func(t *T)) bool
func (tc TBC) Run(t *B, f func(t *B)) bool |
|
Moving out of hold per #52751 (comment) since a prototype implementation exists. |
|
As some of the other posters mentioned func Test(t *testing.T) {
type ItoaTest struct {
name string
item int
want string
}
run := func(test ItoaTest) {
t.Helper()
got := strconv.Itoa(2 * (test.item / 2))
if got != test.want {
t.Errorf("%s - Itoa(%d) = %v, want %v", test.name, test.item, got, test.want)
}
}
run(ItoaTest{"First", 1, "1"})
run(ItoaTest{"Second", 2, "2"})
run(ItoaTest{"Third", 3, "3"})
run(ItoaTest{"Fourth", 4, "4"})
run(ItoaTest{"Fifth", 5, "5"})
} |
|
@flowchartsman that doesn't help the situation where |
|
Is this something where the clever use of type parameters can help? TableTest[squareTest](t).
Case("zero", squareTest{0, 0}).
Case("negative", squareTest{-2, 4}).
Case("positive", squareTest{3, 9}).
Case("huge", squareTest{1000, 1000_000}).
Case("fail", squareTest{1, 0}).
Run(func(t *testing.T, c squareTest) {
if res := c.in * c.in; res != c.out {
t.Errorf("%d*%d = %d, want %d", c.in, c.in, res, c.out)
}
})https://go.dev/play/p/bWVlsiS1jZ6 Though I suspect the fluent-style interface is a turn-off, there might be some extra tricks you can do. |
|
Or, perhaps as a generic slice, keeping that table-driven feel: TableTest[squareTest]{
Case("zero", squareTest{0, 0}),
Case("negative", squareTest{-2, 4}),
Case("positive", squareTest{3, 9}),
Case("huge", squareTest{1000, 1000_000}),
Case("fail", squareTest{1, 0}),
}.Run(t, func(t *testing.T, c squareTest) {
if res := c.in * c.in; res != c.out {
t.Errorf("%d*%d = %d, want %d", c.in, c.in, res, c.out)
}
}) |
|
@flowchartsman Embedding location into test name in this form makes it hard to run individual test via |
I think Maybe with some careful effort |
I agree. I should have been more clear, I am attempting to demonstrate a POC for general execution, however I might also argue that running individual tests out of a table-driven test is something that only certain IDEs do to my knowledge, and it might be out of scope. Still, there might be a solution if you're clever enough. If there's actual interest, I'd be willing to implement a prototype in |
|
Here is the API from CL 444195: And it is used as The t.Source line makes the failure logs print the additional file:line at the start of the messages: This does require having a name in the test struct, which not all table-driven tests have. Another option would be to separate them out so that Mark takes no arguments and Marker has no Name inside it. Then you'd use And at that point you could make Source take a Marker directly: and at that point maybe the type name be Source instead of Marker: The meaning of t.Source is still a bit hard to understand if you don't read the docs. And Source can't be both a method and a type. What about ? |
|
It sounds like the consensus so far is to include the extra If I understand correctly, that means a test with multiple calls to I would like to make one final attempt to argue for separate lines of output: My previous argument for this approach was that it's easier to understand which of these With the acceptance of #37708, I think multiple As soon as multiple Edit: the API suggestion below does not attribute the line to the right subtest. It would require at least another method or type. I believe the API to support separate lines would also be smaller. Instead of needing 3 new symbols (a type, a function, and a method on testing.T.Mark(args ...any)
One disadvantage to this API is that it would not provide anything new for other sources of |
This seems a bit unfortunate, since every line of output for a test will have the same prefix. Also, as @dnephin points out, I don't believe editors generally recognize multiple filename:line prefixes on the same line. What if the marker was associated with the test line? Also, bikeshedding on the API a bit, what if we passed the source information to The This requires no changes to test code other than to use Making the source information actually part of the test name does have the unfortunate property that the name is not stable over time, so perhaps that isn't a good idea. On the other hand, systems that care about name stability can strip off the bracketed suffix, so perhaps it wouldn't be a problem. |
I think there was an idea to support multiple source locations for a single test case.
Or |
I thought about what you're suggesting and rejected it myself for the technically backwards incompatibility argument. That said, I am personally in support of this approach for it's simplicity. If we can't change the behavior of
So long as the name used to match the argument given to With the recent adoption of #37708, we'll want to consider how that plays into this proposal. For @neild's proposal, the path passed to
Interesting idea, but how would you deal with conflicts? It is fairly conceivable that two cases in different contexts might both be named "Foo". |
Probably would need to count duplicates and return numbered names |
|
I don't think that works well for cases that would not normally need deduplication. For example: var fooCases = []struct { ... } {
{testing.Case("FizzBuzz"), ...},
...
}
func TestBar(t *testing.T) {
... // uses fooCases
}
var barCases = []struct { ... } {
{testing.Case("FizzBuzz"), ...},
...
}
func TestBar(t *testing.T) {
... // uses barCases
}Both occurrences of |
Instead of
That would allow the test author to replace the default type testCase struct {
name string
pos testing.Pos
...
}
...
// t.Log, t.Error, t.Fatal with a testing.Pos at args[0] replaces the default file:line marker
t.Log(tc.pos, "test case definition")
// source.go:123: test case definition
// testing.Pos at args[1:] would continue to work
t.Log("failed at", tc.pos)
// source.go:444: failed at source.go:123
// t.Logf, t.Errorf, t.Fatalf use the new verb to replace the default file:line marker
t.Logf("%l: test case definition", tc.pos)
// source.go:123: test case definition
// the position can also be used with the default file:line marker
t.Fatalf("%v: got: ..., want: ...", tc.pos, got, want)
// source.go:444: source.go:123: got: ..., want: ...
This approach would reduce the API to a single new |
|
Proposal: When a test name passed to t.Run contains a newline, the test name is everything up to the newline and the remainder is printed at the start of the test log. func Test(t *testing.T) {
t.Run("Foo\nmore information", func(t *testing.T) {
t.Fatal("oops")
}
}(To bikeshed: What if there are multiple newlines?) The new function func Test(t *testing.T) {
for _, test := range []struct{
name string
}{{
name: testing.Case("Foo"),
}}{
t.Run(test.name, func(t *testing.T) {
t.Fatal("oops")
}
}
}It's unlikely that anyone is currently defining a test name containing a newline, since I'm pretty sure test output breaks if you do. This provides a mechanism to add other per-case information than filename and line number, and doesn't require any changes to existing test output parsers. Updating existing tests is just changing |
I believe the newline is converted to a I like the approach of associating the marker with a name. Earlier suggestions like #52751 (comment) seemed like a nice way to make that work. |
|
It seemed like we were at a reasonable point with There was a concern raised about multiple file:line: on a given line, but that should be plenty legible, and at least the tools I use have no problem with linking to either one. If others know of specifically broken environments that don't support that, let's look at concrete cases. The discussion moved over to trying to smuggle information into the t.Run argument, but that's a pretty major change, and a confusing one. We should try to use data types and data structures instead of appending things to the t.Run argument. Other than the question of how to print two file:line associated with a message, are there other concerns remaining about this proposal? |
When combined with the new I also commented in #52751 (comment) that the API was easy to accidentally misuse with a one-line mistake like: for _, tt := range tests {
+ t.Source(tt.Source())
t.Run(tt.Name, func(t *testing.T) {
- t.Source(tt.Source())
...
})
}which would drastically change the way test output was formatted. The added line would (unintentionally) cause multiple source positions to be stacked up, while the removed line would (correctly) only print one source position per In #52751 (comment), I concluded that I still believed the best solution was still a new
While I advocated for a new |
|
Mixing the source into Run seems like it will end up being a non-orthogonal issue and lead to more, and then we'll need all the possible combinations of arguments to Run. A helper on testing.T seems clearer, and mostly separates it from Run. As for the mistake of calling t.Source in the outer loop, maybe Source should only be setting a single source instead of stacking up? Then it's not tied to Run. It would probably be t.SetPos then, and the test log messages would print the Go source line as well as the position set by SetPos. |
GoLand (the JetBrains IDE for Go) does not support this. Here's an example screenshot from the terminal window in GoLand:
|
Is this perhaps instead an argument that there's a variation of
Quite possibly. I would support this semantic more than the stacking up semantic, but it could interact poorly with usages of |
I think that would address half the problem. In the example above More importantly, I think that behaviour raises a new question. There's already a much more intuitive way of adding that information to a Given this implementation of type Pos = token.Position
func Mark() Pos {
_, file, line, _ := runtime.Caller(1)
return Pos{Filename: filepath.Base(file), Line: line}
}The pos := testing.Mark()
...
t.Fatalf("%v: fizz error: %v", pos, err)
// or
t.Log(pos, "test case data")I believe this would product the same output as |
In Go, it is very common to use table-driven tests:
When this test fails, it prints with something like:
my_test.go:1234tells us where in the test logic this failed.Test/Barname tells us which test case failed.Most code editors today identify
source.go:1234strings and automatically provide the ability to jump to that source code location. This is really helpful for jumping to the execution logic that failed, but is not helpful for jumping to the test data that caused the failure. It is impossible for editor tooling to automatically correlate the the test name (e.g.,Test/Bar) with the test case in the code since the association between the two can be determined by arbitrary Turing-complete logic.I propose the following API in the
testingpackage:Using this API, the example above would be changed as:
tests := struct { - name string + name testing.NameFileLine input T ... } {{ - name: "Foo", + name: testing.Name("Foo"), ..., }, { - name: "Bar", + name: testing.Name("Bar"), // my_test.go:321 ..., } ... // maybe dozens or hundreds more cases } for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { + t.RunName(tt.name, func(t *testing.T) { got, err := fizz(tt.input) if err != nil { t.Fatalf("fizz error: %v", err) // my_test.go:1234 } }) }testing.Namein every test case, which captures file and line information about where the test case was declared.testing.T.RunNameand pass it thetesting.TestNameso that the subtest knows what test case is associated with this subtest.Thus, the test output would be something like:
my_test.go:321tells us where the test data was declared.my_test.go:1234tells us where in the test logic this failed.Now, we can click on
my_test.go:321in our code editors and it will take us directly to the test case declaration.The text was updated successfully, but these errors were encountered: