Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: add Name to track file and line of test case declaration #52751

Open
dsnet opened this issue May 7, 2022 · 6 comments
Open

proposal: testing: add Name to track file and line of test case declaration #52751

dsnet opened this issue May 7, 2022 · 6 comments
Labels
Projects
Milestone

Comments

@dsnet
Copy link
Member

@dsnet dsnet commented May 7, 2022

In Go, it is very common to use table-driven tests:

tests := struct {
    name string
    input T
    ...
} {{
    name: "Foo",
    ...,
}, {
    name: "Bar",
    ...,
}
... // maybe dozens or hundreds more cases
}
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
        got, err := fizz(tt.input)
        if err != nil {
            t.Fatalf("fizz error: %v", err) // my_test.go:1234
        }
    })
}

When this test fails, it prints with something like:

--- FAIL: Test/Bar (0.00s)
    my_test.go:1234: fizz error: the world blew up
  • The my_test.go:1234 tells us where in the test logic this failed.
  • The Test/Bar name tells us which test case failed.

Most code editors today identify source.go:1234 strings and automatically provide the ability to jump to that source code location. This is really helpful for jumping to the execution logic that failed, but is not helpful for jumping to the test data that caused the failure. It is impossible for editor tooling to automatically correlate the the test name (e.g., Test/Bar) with the test case in the code since the association between the two can be determined by arbitrary Turing-complete logic.

I propose the following API in the testing package:

// NameFileLine is a name combined with a file and line number.
type NameFileLine struct { ... }

// Name constructs a NameFileLine.
// It annotates the name with the file and line number of the caller.
func Name(name string) NameFileLine

// RunName runs f as a subtest of t called name.
func (t *T) RunName(name NameFileLine, f func(t *testing.T))

// RunName runs f as a subtest of b called name.
func (b *B) RunName(name NameFileLine, f func(b *testing.B))

Using this API, the example above would be changed as:

  tests := struct {
-     name string
+     name testing.NameFileLine
      input T
      ...
  } {{
-     name: "Foo",
+     name: testing.Name("Foo"),
      ...,
  }, {
-     name: "Bar",
+     name: testing.Name("Bar"), // my_test.go:321
      ...,
  }
  ... // maybe dozens or hundreds more cases
  }
  for _, tt := range tests {
-     t.Run(tt.name, func(t *testing.T) {
+     t.RunName(tt.name, func(t *testing.T) {
          got, err := fizz(tt.input)
          if err != nil {
              t.Fatalf("fizz error: %v", err) // my_test.go:1234
          }
      })
  }
  • We call testing.Name in every test case, which captures file and line information about where the test case was declared.
  • We call testing.T.RunName and pass it the testing.TestName so that the subtest knows what test case is associated with this subtest.

Thus, the test output would be something like:

--- FAIL: Test/Bar (0.00s)
    my_test.go:321: my_test.go:1234: fizz error: the world blew up
  • The my_test.go:321 tells us where the test data was declared.
  • The my_test.go:1234 tells us where in the test logic this failed.

Now, we can click on my_test.go:321 in our code editors and it will take us directly to the test case declaration.

@dsnet dsnet added the Proposal label May 7, 2022
@gopherbot gopherbot added this to the Proposal milestone May 7, 2022
@randall77
Copy link
Contributor

@randall77 randall77 commented May 7, 2022

tests := struct {
    name string
    fileLine string
    input T
} {
    name: "foo",
    fileLine: fileLine(),
    ...
}

func fileLine() string {
    _, file, line, _ := runtime.Caller(1)
    return file + ":" + line
}

This is similar to the workaround we use in the stdlib, e.g., reflect.verifyGCBits.

@randall77
Copy link
Contributor

@randall77 randall77 commented May 7, 2022

That said, it would be nice to have something ergonomic built in to testing. I wish testing.Run took an any instead of a string as the name, so we didn't have to introduce another method on Testing to enable this.

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals May 7, 2022
@dsnet
Copy link
Member Author

@dsnet dsnet commented May 9, 2022

To avoid declaring new RunName methods, we could do something like:

  for _, tt := range tests {
-     t.Run(tt.name, func(t *testing.T) {
+     t.Run(tt.name.String(), func(t *testing.T) {
          ...
      })
  }

where testing.NameFileLine.String prints in a special syntax recognized by testing.T.Run and testing.B.Run. However, this might be considered a breaking change.

@rsc
Copy link
Contributor

@rsc rsc commented May 11, 2022

It seems like there are two things going on here:

  1. The trick about testing.Name recording where it was called from.
  2. The ability to add file:line annotations to test failures.

It seems like we could keep t.Run the same, pass tt.name.String() to it, and then separately have a

t.AddFileLine(tt.name.FileLine())

call at the start of the function body. Then other sources of file:line (like testscript failures) can hook into this too.

@rsc
Copy link
Contributor

@rsc rsc commented May 11, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc moved this from Incoming to Active in Proposals May 11, 2022
@dnephin
Copy link
Contributor

@dnephin dnephin commented May 12, 2022

I have encountered this problem a few times. I'll share the workaround I came up with in case it helps this proposal in some way.

Using t.Helper and a run function I was able to have the tests emit at least 2 file and line number outputs: one for the first line of the test case definition, and one from the test logic function where the failure happened.

To make this work, the traditional test case "table" has to be modified a bit.

  • the logic of the test is defined at the top in a run function, instead of at the end.
  • instead of a slice or map of test cases, each one calls run. It's a few extra characters per test case.

Using the example test from the description, it might look like this:

type testCase struct {
    name string
    input T
    ...
}

run := func(t *testing.T, tc testCase) {
    t.Helper()
    t.Log("case:", tc.name)
    t.Run(name, func(t *testing.T) {
        got, err := fizz(tt.input)
        if err != nil {
            t.Fatalf("fizz error: %v", err) // my_test.go:1234
        }
    })
}

run(t, testCase{
    name: "Foo",
    ...,
}
run(t, testCase{
    name: "Bar",
    ...,
}
... // maybe dozens or hundreds more cases

Definitely not as elegant as the fileLine workaround. t.AddFileLine would be a great addition!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

No branches or pull requests

5 participants