testing: add Name to track file and line of test case declaration

[dsnet]

In Go, it is very common to use table-driven tests:

tests := struct {
    name string
    input T
    ...
} {{
    name: "Foo",
    ...,
}, {
    name: "Bar",
    ...,
}
... // maybe dozens or hundreds more cases
}
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
        got, err := fizz(tt.input)
        if err != nil {
            t.Fatalf("fizz error: %v", err) // my_test.go:1234
        }
    })
}

When this test fails, it prints with something like:

--- FAIL: Test/Bar (0.00s)
    my_test.go:1234: fizz error: the world blew up

• The my_test.go:1234 tells us where in the test logic this failed.
• The Test/Bar name tells us which test case failed.

Most code editors today identify source.go:1234 strings and automatically provide the ability to jump to that source code location. This is really helpful for jumping to the execution logic that failed, but is not helpful for jumping to the test data that caused the failure. It is impossible for editor tooling to automatically correlate the the test name (e.g., Test/Bar) with the test case in the code since the association between the two can be determined by arbitrary Turing-complete logic.

I propose the following API in the testing package:

// NameFileLine is a name combined with a file and line number.
type NameFileLine struct { ... }

// Name constructs a NameFileLine.
// It annotates the name with the file and line number of the caller.
func Name(name string) NameFileLine

// RunName runs f as a subtest of t called name.
func (t *T) RunName(name NameFileLine, f func(t *testing.T))

// RunName runs f as a subtest of b called name.
func (b *B) RunName(name NameFileLine, f func(b *testing.B))

Using this API, the example above would be changed as:

  tests := struct {
-     name string
+     name testing.NameFileLine
      input T
      ...
  } {{
-     name: "Foo",
+     name: testing.Name("Foo"),
      ...,
  }, {
-     name: "Bar",
+     name: testing.Name("Bar"), // my_test.go:321
      ...,
  }
  ... // maybe dozens or hundreds more cases
  }
  for _, tt := range tests {
-     t.Run(tt.name, func(t *testing.T) {
+     t.RunName(tt.name, func(t *testing.T) {
          got, err := fizz(tt.input)
          if err != nil {
              t.Fatalf("fizz error: %v", err) // my_test.go:1234
          }
      })
  }

• We call testing.Name in every test case, which captures file and line information about where the test case was declared.
• We call testing.T.RunName and pass it the testing.TestName so that the subtest knows what test case is associated with this subtest.

Thus, the test output would be something like:

--- FAIL: Test/Bar (0.00s)
    my_test.go:321: my_test.go:1234: fizz error: the world blew up

• The my_test.go:321 tells us where the test data was declared.
• The my_test.go:1234 tells us where in the test logic this failed.

Now, we can click on my_test.go:321 in our code editors and it will take us directly to the test case declaration.

[randall77]

tests := struct {
    name string
    fileLine string
    input T
} {
    name: "foo",
    fileLine: fileLine(),
    ...
}

func fileLine() string {
    _, file, line, _ := runtime.Caller(1)
    return file + ":" + line
}

This is similar to the workaround we use in the stdlib, e.g., reflect.verifyGCBits.

[randall77]

That said, it would be nice to have something ergonomic built in to testing. I wish testing.Run took an any instead of a string as the name, so we didn't have to introduce another method on Testing to enable this.

[dsnet]

To avoid declaring new RunName methods, we could do something like:

  for _, tt := range tests {
-     t.Run(tt.name, func(t *testing.T) {
+     t.Run(tt.name.String(), func(t *testing.T) {
          ...
      })
  }

where testing.NameFileLine.String prints in a special syntax recognized by testing.T.Run and testing.B.Run. However, this might be considered a breaking change.

[rsc]

It seems like there are two things going on here:

1. The trick about testing.Name recording where it was called from.
2. The ability to add file:line annotations to test failures.

It seems like we could keep t.Run the same, pass tt.name.String() to it, and then separately have a

t.AddFileLine(tt.name.FileLine())

call at the start of the function body. Then other sources of file:line (like testscript failures) can hook into this too.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[dnephin]

I have encountered this problem a few times. I'll share the workaround I came up with in case it helps this proposal in some way.

Using t.Helper and a run function I was able to have the tests emit at least 2 file and line number outputs: one for the first line of the test case definition, and one from the test logic function where the failure happened.

To make this work, the traditional test case "table" has to be modified a bit.

• the logic of the test is defined at the top in a run function, instead of at the end.
• instead of a slice or map of test cases, each one calls run. It's a few extra characters per test case.

Using the example test from the description, it might look like this:

type testCase struct {
    name string
    input T
    ...
}

run := func(t *testing.T, tc testCase) {
    t.Helper()
    t.Log("case:", tc.name)
    t.Run(name, func(t *testing.T) {
        got, err := fizz(tt.input)
        if err != nil {
            t.Fatalf("fizz error: %v", err) // my_test.go:1234
        }
    })
}

run(t, testCase{
    name: "Foo",
    ...,
}
run(t, testCase{
    name: "Bar",
    ...,
}
... // maybe dozens or hundreds more cases

Definitely not as elegant as the fileLine workaround. t.AddFileLine would be a great addition!

[rsc]

I'd still like to understand better whether we can separate out the two different features being added, as I noted in #52751 (comment). Any thoughts, @dsnet?

[AlexanderYastrebov]

Since new type is introduced maybe there is no need to have String and AddFileLine methods but just a Run wrapper:

package go52751

import (
	"fmt"
	"runtime"
	"testing"
)

type TC struct {
	name     string
	location string
}

func (tc *TC) Run(t *testing.T, tf func(t *testing.T)) bool {
	t.Helper()
	return t.Run(tc.name, func(t *testing.T) {
		t.Helper()
		// this should use internal undecorated logging to achieve desired output
		t.Logf("Test case %q defined at %s", tc.name, tc.location)
		tf(t)
	})
}

func testingCase(name string) *TC {
	_, file, line, _ := runtime.Caller(1)
	return &TC{name, fmt.Sprintf("%s:%d", file, line)}
}

func TestFileLine(t *testing.T) {
	tests := []struct {
		tc    *TC
		input string
	}{{
		tc:    testingCase("Foo"),
		input: "x",
	}, {
		tc:    testingCase("Bar"),
		input: "",
	},
	}
	for _, tt := range tests {
		tt.tc.Run(t, func(t *testing.T) {
			if tt.input == "" {
				t.Fatalf("input error")
			}
		})
	}
}

--- FAIL: TestFileLine (0.00s)
    --- FAIL: TestFileLine/Bar (0.00s)
        case_test.go:42: Test case "Bar" defined at /tmp/go52751/case_test.go:37
        case_test.go:44: input error
FAIL
FAIL	command-line-arguments	0.001s
FAIL

[dsnet]

@AlexanderYastrebov, you would need two different Run methods since the method must taking in either a func(*testing.T) or a func(*testing.B). It's not clear that it's a cleaner API than what was originally proposed.

[AlexanderYastrebov]

@dsnet

diff --git a/src/testing/testing.go b/src/testing/testing.go
index ec2d864822..dc1bfc301e 100644
--- a/src/testing/testing.go
+++ b/src/testing/testing.go
@@ -529,6 +529,8 @@ type common struct {
        tempDir    string
        tempDirErr error
        tempDirSeq int32
+
+       cases map[string]string
 }
 
 // Short reports whether the -test.short flag is set.
@@ -1451,6 +1453,27 @@ func tRunner(t *T, fn func(t *T)) {
        t.mu.Unlock()
 }
 
+func (t *T) Case(name string) string {
+       _, file, line, _ := runtime.Caller(1)
+       location := fmt.Sprintf("%s:%d\n", file, line)
+
+       t.mu.Lock()
+       if t.cases == nil {
+               t.cases = make(map[string]string)
+       }
+       uniqName := name
+       for i := 1; ; i++ {
+               if _, ok := t.cases[uniqName]; !ok {
+                       break
+               }
+               uniqName = fmt.Sprintf("%s#%d", name, i)
+       }
+       t.cases[uniqName] = location
+       t.mu.Unlock()
+
+       return uniqName
+}
+
 // Run runs f as a subtest of t called name. It runs f in a separate goroutine
 // and blocks until f returns or calls t.Parallel to become a parallel test.
 // Run reports whether f succeeded (or at least did not fail before calling t.Parallel).
@@ -1463,6 +1486,15 @@ func (t *T) Run(name string, f func(t *T)) bool {
        if !ok || shouldFailFast() {
                return true
        }
+
+       if loc, ok := t.cases[name]; ok {
+               fo := f
+               f = func(t *T) {
+                       t.Helper()
+                       t.Logf("case at %s", loc)
+                       fo(t)
+               }
+       }
        // Record the stack trace at the point of this call so that if the subtest
        // function - which runs in a separate stack - is marked as a helper, we can
        // continue walking the stack into the parent test.

package go52751

import (
	"testing"
)

func TestFileLine(t *testing.T) {
	tests := []struct {
		name  string
		input string
	}{{
		name:  t.Case("Foo"),
		input: "x",
	}, {
		name:  t.Case("Bar"),
		input: "",
	},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.input == "" {
				t.Fatalf("input error")
			}
		})
	}
}

--- FAIL: TestFileLine (0.00s)
    --- FAIL: TestFileLine/Bar (0.00s)
        case_test.go:20: case at /tmp/go52751/case_test.go:15
        case_test.go:22: input error

[rsc]

@dsnet, any thoughts on #52751 (comment) ?

[dsnet]

In regards to #52751 (comment), it seems odd if the test name is annotated with file:line information if we can directly use the test name and file:line information together. There's a discrepancy between how creation and usage operates.

If usage is segregated, then creation should also be segregated:

  tests := struct {
      name string
+     location testing.SourceLocation
      input T
      ...
  } {{
      name: "Foo",
+     location: testing.FileLine(),
      ...,
  }, {
      name: "Bar",
+     location: testing.FileLine(), // my_test.go:321
      ...,
  }
  ... // maybe dozens or hundreds more cases
  }
  for _, tt := range tests {
      t.Run(tt.name, func(t *testing.T) {
+         t.Annotate(tt.location)
          got, err := fizz(tt.input)
          if err != nil {
              t.Fatalf("fizz error: %v", err) // my_test.go:1234
          }
      })
  }

This approach is more flexible as you can imagine using this for more purposes than just annotating test case names.
However, it is a little more typing than my original proposal in #52751 (comment).
I would be okay with something like this API.

---

In response to #52751 (comment), one significant detriment to func (t *T) Case(name string) string is that it assumes that we have a testing.T on hand when crafting the test cases. I have some tests that share a global testcases table. There is no single testing.T around at the time that the table was constructed.

[rsc]

@dsnet, you jumped to testing.SourceLocation, which I didn't really have in mind. I think testing.Name returning something that has a name and a file:line is fine. But there are other tests reading test data files that might also want to set the file:line, and I was suggesting that they could use that mechanism too if it were split out. I don't think we need a SourceLocation type though.

So only the t.Annotate would be added in the diff, not all the testing.FileLine() calls.