testing: structured output for test attributes

Test2json, more particularly go test -json, has been quite a pleasant discovery. It allows for programs to analyze go tests and create their own formatted output.

For example, using GitHub Actions' formatting capabilities, I was able to better format go tests to look more user friendly when running in the UI:

Before:

After:

With that said, there are still some missing features that would allow programs to better understand the JSON output of a test.

Proposal

It would be great if Go Tests can attach metadata to be included in the JSON output of a test2json run.

Something along these lines:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.WithMetadata(map[string]string{"requestID": "123"}).Errorf("Foo failed")
  // outputs: {"action": "output", "output": "Foo failed", "metadata": {"requestID": "123"}}
}

Benefits:

This allows for a few highly beneficial use cases:

If a test fails, then the program that's analyzing the failed test's json can receive metadata about why it failed: such as requestID, userID etc and then provide the user helpful links to logs and queries.
A test can provide source-code information about where things failed. Because right now test2json cannot distinguish between when a user called t.Fatal(...) or t.Log(...) which makes sense as t.Fatal just calls t.Log -- but the user can include metadata so we know exactly where the error occurred and use CI capabilities such as Actions' error command to set the file and line number to be displayed in the UI.

Alternative solutions:

Include directives in the output string that the json-parsing program can analyze to see if there's metadata. But this solution is very fragile and prone to error.

Thanks!

I looked into this a bit; Unfortunately I don't think it can work quite the way you've proposed (at least, not with the current testing architecture). In particular, testing likes to emit everything in a text stream, and the JSON blobs are reconstituted with cmd/test2json from that text stream; it would be really tricky to attach metadata to a particular logging statement.

As an additional wrinkle, encoding/json has a dependency on testing, meaning that testing cannot actually use Go's json package for any encoding :(.

It SHOULD be possible, however, to have something which worked like:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.Meta("requestID", "123")
  // outputs: {"action": "meta", "meta": {"requestID": "123"}}
  t.Log("Something Else")
  // outputs: {"action": "output", "output": "foo_test.go:16 Foo\n"}
}

And it could work by emitting an output like:

=== RUN   TestFoo
    foo_test.go:12: Foo
--- META: TestFoo: requestID: 123
    foo_test.go:16: Something Else
--- PASS: TestFoo (N.NNs)

Where ":" is a forbidden character in the key, and the value is trimmed for whitespace.

I think that this functionality might be "good enough" when parsing the test output JSON; metadata would effectively accumulate for the duration of the test, since the test JSON is effectively scanned top-to-bottom anyway to extract information about a given test.

I can write up a CL that we can poke at, if folks don't hate this :)

Actually just went ahead and made a CL: https://go-review.googlesource.com/c/go/+/357914

Change https://golang.org/cl/357914 mentions this issue: testing: allow structured metadata in test2json

+1 for this feature. Being able to set arbitrary metadata would be a great way of helping to get our go test results into a test management system without having to rely on a third party testing library.

I see the CL is kinda stalled out, I can offer time to help push this forward if anything is needed.

Now that we have slog, I wonder if this proposal should be about adding some kind of slog API to testing.T, which is passed through when using test2json?

@aclements

cc @aclements @dmitshur as I believe y'all have been looking at structured output with cmd/dist.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Good discussion on #59928 to figure out how to hook up slog to testing. If we do that, I think that will take care of the need here.

Actually, #59928 (comment) convinced me of the opposite. Slog output should be action="output" but this information should be action="somethingelse".

If we do keep these separate, then I suggest TB.Data(key string, value any), where value is formatted with%q to keep it on one line. "Metadata" is the wrong word here (as it often is elsewhere). This isn't data about data, it's data about tests. I was also thinking of Info, but that might cause confusion with slog, whose Info method takes a message before its keys and values.

If we want to enforce uniqueness amongst attribute keys, should they be unique amongst the set of all attribute keys, or unique amongst the set of (test name, attribute key) combinations? Towards the latter, I think testing does enough to promote unique test names (e.g. the Name method of T, B, and F) that this might be reasonable, but I'm also not sure I'm anticipating all possible complications.

For reference, JUnit XML test properties allow duplicate keys, and the choice of how to interpret duplicates is generally up to the consumer.

I think duplicate keys could be useful to create multi-valued attributes without having to invent an encoding ("separated by commas," for example). The situation is analogous to HTTP headers. I don't know if the world considers multi-valued headers to be a good idea or a bad idea.

The proposed documentation states that keys are emitted in-order: "test attributes are emitted immediately in the test log". I think that suffices to document our behavior here, and I think that phrasing is already sufficiently binding to prevent us from making changes to the order of attributes in the future.

Attributes have no meaning other than what the consumer imparts to them. T.Attremits an attribute to the test log. What happens it after that point is up to the consumer. That goes for duplicates as well.

That seems reasonable, but what is "but they are intended to be treated as unordered" in the doc comment supposed to mean? Should we just drop that?

+1 to dropping "they are intended...", given that the testing package doesn't really have any intent around attributes.

I had been reading "they are intended to be treated as unordered" as a hedge against drawing the conclusion that "emitted immediately" implies it's entirely safe to draw conclusions from order of appearance in the test log. This is holding a lot of things wrong but seems like it can emit an odd "i" value immediately before logging "i is even":

	for i := range 100 {
		go func() {
			t.Attr("i", strconv.Itoa(i))
			if i / 2 == 0 {
				t.Error("i is even")
			}
		}()
	}

So I wonder if "emitted immediately" is helpful.

testing: add Attr

Navigation Menu

Repository navigation

testing: structured output for test attributes #43936

Proposal

Benefits:

Alternative solutions:

88 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

testing: structured output for test attributes #43936

Description

Proposal

Benefits:

Alternative solutions:

Activity

riannucci commented on Oct 22, 2021

riannucci commented on Oct 22, 2021

gopherbot commented on Oct 22, 2021

nine9ths commented on Apr 28, 2023

prattmic commented on Jul 18, 2023

prattmic commented on Jul 18, 2023

rsc commented on Jul 19, 2023

rsc commented on Jul 26, 2023

jba commented on Jul 27, 2023

88 remaining items

AndrewHarrisSPU commented on Apr 14, 2025

greg-dennis commented on Apr 14, 2025

jba commented on Apr 15, 2025

neild commented on Apr 15, 2025

prattmic commented on Apr 15, 2025

neild commented on Apr 15, 2025

AndrewHarrisSPU commented on Apr 15, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions