Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/test2json: Allow Go Tests to Pass Metadata #43936

Open
marwan-at-work opened this issue Jan 27, 2021 · 3 comments
Open

proposal: cmd/test2json: Allow Go Tests to Pass Metadata #43936

marwan-at-work opened this issue Jan 27, 2021 · 3 comments
Labels
Projects
Milestone

Comments

@marwan-at-work
Copy link
Contributor

@marwan-at-work marwan-at-work commented Jan 27, 2021

Test2json, more particularly go test -json, has been quite a pleasant discovery. It allows for programs to analyze go tests and create their own formatted output.

For example, using GitHub Actions' formatting capabilities, I was able to better format go tests to look more user friendly when running in the UI:

Before:

Screen Shot 2021-01-26 at 5 22 08 PM

After:

Screen Shot 2021-01-26 at 5 21 58 PM

With that said, there are still some missing features that would allow programs to better understand the JSON output of a test.

Proposal

It would be great if Go Tests can attach metadata to be included in the JSON output of a test2json run.

Something along these lines:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.WithMetadata(map[string]string{"requestID": "123"}).Errorf("Foo failed")
  // outputs: {"action": "output", "output": "Foo failed", "metadata": {"requestID": "123"}}
}

Benefits:

This allows for a few highly beneficial use cases:

  1. If a test fails, then the program that's analyzing the failed test's json can receive metadata about why it failed: such as requestID, userID etc and then provide the user helpful links to logs and queries.
  2. A test can provide source-code information about where things failed. Because right now test2json cannot distinguish between when a user called t.Fatal(...) or t.Log(...) which makes sense as t.Fatal just calls t.Log -- but the user can include metadata so we know exactly where the error occurred and use CI capabilities such as Actions' error command to set the file and line number to be displayed in the UI.

Alternative solutions:

Include directives in the output string that the json-parsing program can analyze to see if there's metadata. But this solution is very fragile and prone to error.

Thanks!

@bcmills bcmills added this to the Proposal milestone Jan 27, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Feb 17, 2021
@riannucci
Copy link

@riannucci riannucci commented Oct 22, 2021

I looked into this a bit; Unfortunately I don't think it can work quite the way you've proposed (at least, not with the current testing architecture). In particular, testing likes to emit everything in a text stream, and the JSON blobs are reconstituted with cmd/test2json from that text stream; it would be really tricky to attach metadata to a particular logging statement.

As an additional wrinkle, encoding/json has a dependency on testing, meaning that testing cannot actually use Go's json package for any encoding :(.

It SHOULD be possible, however, to have something which worked like:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.Meta("requestID", "123")
  // outputs: {"action": "meta", "meta": {"requestID": "123"}}
  t.Log("Something Else")
  // outputs: {"action": "output", "output": "foo_test.go:16 Foo\n"}
}

And it could work by emitting an output like:

=== RUN   TestFoo
    foo_test.go:12: Foo
--- META: TestFoo: requestID: 123
    foo_test.go:16: Something Else
--- PASS: TestFoo (N.NNs)

Where ":" is a forbidden character in the key, and the value is trimmed for whitespace.

I think that this functionality might be "good enough" when parsing the test output JSON; metadata would effectively accumulate for the duration of the test, since the test JSON is effectively scanned top-to-bottom anyway to extract information about a given test.

I can write up a CL that we can poke at, if folks don't hate this :)

@riannucci
Copy link

@riannucci riannucci commented Oct 22, 2021

Actually just went ahead and made a CL: https://go-review.googlesource.com/c/go/+/357914

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 22, 2021

Change https://golang.org/cl/357914 mentions this issue: testing: allow structured metadata in test2json

@rsc rsc changed the title cmd/test2json: Allow Go Tests to Pass Metadata proposal: cmd/test2json: Allow Go Tests to Pass Metadata Jun 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Proposals
Incoming
Development

No branches or pull requests

4 participants