proposal: cmd/test2json: Allow Go Tests to Pass Metadata

[marwan-at-work]

Test2json, more particularly go test -json, has been quite a pleasant discovery. It allows for programs to analyze go tests and create their own formatted output.

For example, using GitHub Actions' formatting capabilities, I was able to better format go tests to look more user friendly when running in the UI:

Before:

After:

With that said, there are still some missing features that would allow programs to better understand the JSON output of a test.

Proposal

It would be great if Go Tests can attach metadata to be included in the JSON output of a test2json run.

Something along these lines:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.WithMetadata(map[string]string{"requestID": "123"}).Errorf("Foo failed")
  // outputs: {"action": "output", "output": "Foo failed", "metadata": {"requestID": "123"}}
}

Benefits:

This allows for a few highly beneficial use cases:

1. If a test fails, then the program that's analyzing the failed test's json can receive metadata about why it failed: such as requestID, userID etc and then provide the user helpful links to logs and queries.
2. A test can provide source-code information about where things failed. Because right now test2json cannot distinguish between when a user called t.Fatal(...) or t.Log(...) which makes sense as t.Fatal just calls t.Log -- but the user can include metadata so we know exactly where the error occurred and use CI capabilities such as Actions' error command to set the file and line number to be displayed in the UI.

Alternative solutions:

Include directives in the output string that the json-parsing program can analyze to see if there's metadata. But this solution is very fragile and prone to error.

Thanks!

[riannucci]

I looked into this a bit; Unfortunately I don't think it can work quite the way you've proposed (at least, not with the current testing architecture). In particular, testing likes to emit everything in a text stream, and the JSON blobs are reconstituted with cmd/test2json from that text stream; it would be really tricky to attach metadata to a particular logging statement.

As an additional wrinkle, encoding/json has a dependency on testing, meaning that testing cannot actually use Go's json package for any encoding :(.

It SHOULD be possible, however, to have something which worked like:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.Meta("requestID", "123")
  // outputs: {"action": "meta", "meta": {"requestID": "123"}}
  t.Log("Something Else")
  // outputs: {"action": "output", "output": "foo_test.go:16 Foo\n"}
}

And it could work by emitting an output like:

=== RUN   TestFoo
    foo_test.go:12: Foo
--- META: TestFoo: requestID: 123
    foo_test.go:16: Something Else
--- PASS: TestFoo (N.NNs)

Where ":" is a forbidden character in the key, and the value is trimmed for whitespace.

I think that this functionality might be "good enough" when parsing the test output JSON; metadata would effectively accumulate for the duration of the test, since the test JSON is effectively scanned top-to-bottom anyway to extract information about a given test.

I can write up a CL that we can poke at, if folks don't hate this :)

[riannucci]

Actually just went ahead and made a CL: https://go-review.googlesource.com/c/go/+/357914

[gopherbot]

Change https://golang.org/cl/357914 mentions this issue: testing: allow structured metadata in test2json

[nine9ths]

+1 for this feature. Being able to set arbitrary metadata would be a great way of helping to get our go test results into a test management system without having to rely on a third party testing library.

I see the CL is kinda stalled out, I can offer time to help push this forward if anything is needed.

[prattmic]

Now that we have slog, I wonder if this proposal should be about adding some kind of slog API to testing.T, which is passed through when using test2json?

[prattmic]

cc @aclements @dmitshur as I believe y'all have been looking at structured output with cmd/dist.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Good discussion on #59928 to figure out how to hook up slog to testing. If we do that, I think that will take care of the need here.

[jba]

Actually, #59928 (comment) convinced me of the opposite. Slog output should be action="output" but this information should be action="somethingelse".

If we do keep these separate, then I suggest TB.Data(key string, value any), where value is formatted with%q to keep it on one line. "Metadata" is the wrong word here (as it often is elsewhere). This isn't data about data, it's data about tests. I was also thinking of Info, but that might cause confusion with slog, whose Info method takes a message before its keys and values.

[riannucci]

One thing I would very much like to have (but maybe cannot 😄 ) is that if value is JSON, that it would be decoded and incorporated into the test2json output without requiring an additional parsing step.

For example:

func TestSomething(t *testing.T) {
  t.Data("mykey", `{"hello": "world"}`)
  t.Data("myotherkey", `hello world`)
}

could yield

{"action": "data", "key": "mykey", "json": {"hello": "world"}}
{"action": "data", "key": "myotherkey", "str": "hello world"}

IIUC one of the constraints here which makes this unpleasant is that testing cannot depend on encoding/json, so we can't make t.Data(key, value) pass value through encoding/json... However I think test2json can depend on more things, so, maybe, possibly, it could see and decode these?

[martin-sucha]

The proposal in #43936 (comment) looks more practical to me than #43936 (comment).

Specifically, different behavior in

func TestSomething(t *testing.T) {
  t.Data("mykey", `{"hello": "world"}`)
  t.Data("myotherkey", `hello world`)
}

could be a source of a lot of issues. How would it behave with input like t.Data("mykey", "{hello}")? We should only have one way to expose the information. It seems to me that the two keys (json and str) would complicate consumers of the data.

[riannucci]

Yeah I think you're right, given the constraints that "testing" cannot validate if a string is valid json or not. Small errors would lead to confusion in the output.

[riannucci]

Though I was thinking that, practically, test2json CAN validate, and the str/json division would show prominently when developing a producer/consumer pair. But really it's not a big deal for a consumer to decode the string as json, and pushing it out of test2json also means less overhead in test2json itself, too.

I think a bigger error would be if you had some consumer which expected a string (in some other encoding), but it just SO HAPPENS to be valid json, and test2json decodes it... that would definitely be annoying.

[rsc]

@riannucci How would LUCI make use of the proposed feature (something like t.WithMetadata)? It seems like it helps individual tests get structured output through to something like LUCI. Is that all you are going for? It would not let the overall execution of a test binary be annotated with JSON metadata.

[riannucci]

So the original context of my involvement in this proposal was maybe frivolous, but I think the proposal does generally have merit beyond that.

Originally I was interested in this because I was involved with the goconvey project (a now very-outdated testing library which did a lot of interesting things, as well as a lot of not-great things). One of the things this library did was that it reported metadata about every assertion which passed, and displayed these in a web UI... the WAY that it passes this data out from the test is pretty bad though (it dumps JSON objects to stdout at various points in the test execution with funny text markers and then tries to parse them out again. This basically doesn't work with subtests or with very much test parallelism). A lot of the weirdness here is due to goconvey's age - it was originally written in the Go 1.2ish era¹.

I was thinking about how to improve this metadata output situation though, and I think the general objective of "I want to be able to communicate data, correlated with individual tests, from within the test to a higher level tool sitting outside of go test" is a reasonable one. Reporting passing assertions is not very high value, but reporting metrics (or other statistics) or notating the location of testing artifacts (large log files, perhaps outputs from a multi-process testing scenario), etc. I think would be valid use cases.

The direct consumer of such data in LUCI would be ResultDB's streaming test result system; it has the ability to associate test artifacts and other metadata directly with test cases, archiving them to e.g. BigQuery.

It's possible to emulate this, of course, with specially crafted Log lines... but I would prefer if there was some out-of-band way to communicate (even if under the hood, currently, it's really 'testing' and 'test2json' trying their best to produce/parse stdout). I would rather have the 'communication channels' be something that go test owns rather than some other mechanism.

An alternative to this proposal which I thought of, but don't especially like, which would be to produce a second, independent channel/file/pipe from the test binary which only has metadata. There are a number of downsides to this, though:

• Unless go test implements a flag for this, any DIY solution would suffer from a discovery problem (i.e. "does this particular package support this custom flag?"). Goconvey 'solves' this by scanning the test imports which is both slow and weird. Additionally, a custom flag would render the test output uncacheable, which is unfortunate.
• If go test DOES support a way to do this... it seems to me that supporting it via go test -json is probably better than adding an additional flag/output file.
• It would be difficult to correlate the metadata with the other output (log lines/stdout) from the test, which could be useful, though not essential for the cases I have in mind.
• Another alternative would be to set an environment variable for this side channel, but environment variables have their own issues - this could also behave oddly with the test cache, since the test would have different cached output depending on the environment

(Now that I think of it... https://pkg.go.dev/cmd/go#hdr-Test_packages doesn't mention -json as a cacheable flag - is it?)

It would not let the overall execution of a test binary be annotated with JSON metadata.

I understand this to mean "adding metadata to go test as proposed would only allow a test to add metadata scoped to a single named test, not the overall test binary output", which is fine for the cases I had in mind.

edit: formatting

Footnotes

1. Coincidentally... I'm rewriting our repos' use of goconvey these last couple weeks... the new version is substantially more normal/modern Go. ↩

[riannucci]

(Oh, I forgot the other bit that goconvey did; for failing assertions it was able to write them out in a structured way, again so that the web UI had better ability to display them; this included things like outputting diffs between actual/expected values)

[aclements]

I think there are several distinct proposals here, all of which are about getting structured information out of tests in some way, but all of which seem to differ significantly in intent:

1. Attaching structured information to individual log lines, where the log lines themselves continue to be regular text. This is my read of the original post. I'm not clear on whether this information should or should not appear in plain text test output because it really is "metadata".
2. A way to emit structured information interleaved with the test log, which is specifically ordered with respect to the reset of the log lines. This is my read of riannucci's comment. I'm not clear how this differs from structured logging in testing. This type of output seems integral to the test log, and thus should be presented in some way even when not in JSON mode. This also doesn't actually seem like "metadata" to me, since it's not data about data, it's just structured data.
3. A way to attach metadata to a test, not in any way ordered with respect to the test log. I'm not sure anyone is asking for this, but this is the other way I could interpret "allowing Go tests to pass metadata".

I think we need concrete use cases to actually move this discussion forward.

[dnephin]

From my read of the original post the proposal could arguably be for category 3. That data may also be in regular log output, but the goal is for some other program to read the data. The data doesn't need to be associated with any particular log line, just the test case. The proposal happened to include it with a log line, but the benefits section seems to highlight the "read it from another program" more than the association with a log line.

The use case I'm familiar with is integration with systems like TestRail. My understanding is that they may have their own identifier for a test case separate from the name. And this "metadata" would be a way to associate test cases with their identifier.

As far as I can tell all the use cases described in the original post and in comments are all in category 3. Some of the comments related to log lines were an attempt to propose a solution, but none of the use cases required association or ordering with existing log lines.

[alexbakker]

Just chiming in to add another use case to the discussion: We use the Go test framework to run a fairly large set of integration tests across our telephony infrastructure. Hundreds of tests and subtests that take about 30 minutes to run in total. Most of these tests start by initiating one or more calls and then check if our services handle the calls and user actions on those calls correctly. Every once in a while, one of these tests will fail after we've made a change or added a new feature. The cause of the failure can not always be found in the logging of the integration tests. Sometimes, something will have gone wrong somewhere in the SIP path and we have to look at logging in other places of our infrastructure. Instead of having to first also dig through the logging of the integration tests to find the associated Call-ID's to query on and such, it would be nice if the Go test framework had a way of exposing some metadata for each test so that we can nicely present it in our test reports (generated from test2json output).

I'm not sure if the Go test framework is intended to be used in this fashion, but figured I'd explain our use case anyway just in case. I believe the proposed t.WithMetadata would work well for us.

[rsc]

I don't think we understand exactly what we need here yet. slog is attractive because it provides the kind of structured data we're talking about, but it's also probably wrong since we want data associated with the test, not a specific log line (see in particular this example from @dnephin). That suggests that we don't want

t.WithMetadata(map[string]string{"requestID": "123"}).Errorf("Foo failed")

but instead we just want something more like:

t.AddMetadata(map[string]string{"requestID": "123"})

That is, it seems like the metadata should not be attached to a specific error message, just to the test itself.

slog is attractive as a way to write structured data, just not the "logging a message" part.

I wonder if we should reuse slog's attribute syntax though. We could add

t.Attr(key, value string)

and define that the attribute list is exactly as defined by slog, including being allowed to pass slog.Attrs. These would be emitted in an

=== ATTR  TestName <key> <value>

line in the output, and in test2json mode would also appear in a

{"Action": "attr", "Test": "TestName", "Attr": {"Key": key, "Value": value}}

line.

Or we could go whole hog and say t.Attrs takes a ...any that it hands to slog to turn into a record and then marshals the record. In test2json mode slog's JSON would end up in the action as

{"Action": "attr", "Test": "TestName": "Attr": anything}

I'm brainstorming here, not arguing for a specific thing.

[rsc]

@jba, @bcmills, any thoughts on adding t.Attr as in my previous comment?

[dnephin]

@bcmills the original example you referenced is not a representative example of the discussion that followed. I believe we should ignore it, and I propose we remove it from the description because at this point it seems like it's a distraction.

Let's review what has been discussed so far. First, the original proposal

It would be great if Go Tests can attach metadata to be included in the JSON output of a test2json run.
If a test fails, then the program that's analyzing the failed test's json can receive metadata about why [the test] failed ... and then provide the user helpful links to [external] logs and queries.

The request is for data about a test, not about a particular operation the test performed. I believe we all agree that the second point from the proposal is already covered by #62728, as you've already mentioned.

The proposal also says:

[An alternative would be to] include directives in the output string that the json-parsing program can analyze to see if there's metadata. But this solution is very fragile and prone to error.

This is the first indication that mixing this with logging is not a good solution, but there were many more to follow. First let's look at all the use cases that support the idea of this being data about a test case (not about a log line).

#43936 (comment) says

Being able to set arbitrary metadata would be a great way of helping to get our go test results into a test management system without having to rely on a third party testing library.

#43936 (comment) says

I think the general objective of "I want to be able to communicate data, correlated with individual tests, from within the test to a higher level tool sitting outside of go test" is a reasonable one. Reporting passing assertions is not very high value, but reporting metrics (or other statistics) or notating the location of testing artifacts (large log files, perhaps outputs from a multi-process testing scenario), etc.

#43936 (comment) references

The use case I'm familiar with is integration with systems like TestRail. My understanding is that they may have their own identifier for a test case separate from the name. And this "metadata" would be a way to associate test cases with their identifier.

#43936 (comment) says

it would be nice if the Go test framework had a way of exposing some metadata for each test so that we can nicely present it in our test reports

All of these are great examples of use cases where we need to associate data with a test. This suggests the t.Attr approach is the correct solution here.

If we look for use cases for attaching data to logging, I think we'll notice there are none! There is not a single use case that would be better served by attaching the data to a specific log line, and there are many examples of why that would not work well.

#43936 (comment) looked into how easy it would be and found problems because of how test2json needs to decode the text format to encode it into JSON.

#43936 (comment) started a discuss about slog, but without presenting any use cases that required it. It was a thought that was explored, and was found to not be a good fit (1, 2).

#43936 (comment) says it well

see #59928 (comment). In short, providing a slog.Logger that works like t.Log, which is the subject of #59928, is intended to capture log output for the system under test. This proposal is about the test code itself providing data to some downstream system. In the examples discussed here, that data is associated with the test, not a specific log line.

All of this points at t.Attr being a more appropriate solution to the problem than anything associated with logging.

one might log separate messages for “request started” and “request failed”, and if the test happens to issue two concurrent requests (maybe it's a test of a concurrency property!), it has no way to associate a given request started message with the ID of that individual request.

That's fine, the test author can and should log separate messages for request started and request failed. If the test author wants to communicate that data using t.Attr it can do that easily:

t.Attr("request.start.0", ...)
t.Attr("request.start.1", ...)
t.Attr("request.end.0", ...)
...

[dnephin]

t.Attr(key string, value any)

Seems more appropriate given this encoding in the go text output format:

=== ATTR  TestName <key> <value>

Including multiple key/value pairs in that line would make parsing it into JSON impractical because value (at least) could contain spaces. Each key/value will need to be on a separate line:

=== ATTR  TestName <key1> <value1>
=== ATTR  TestName <key2> <value2>

That makes t.Attr(key string, value any) more consistent with the existing methods on testing.T. A t.Attr(...any) looks like t.Log(...any), but would output differently (the first prints multiple lines, the second a single line).

---

A couple question about the implementation:

1. How will test2json handle spaces in keys? Are spaces not allowed in keys?
2. Are there any limitations on the contents of value? Can it include newlines? How is it encoded so that test2json can turn it into JSON?

If we want to support json values in test2json then it seems like value needs to be encoded as JSON in text go test output, otherwise it might not be possible to properly encode it as json in test2json.

I'm quite happy with the proposed t.Attr(key string, value any), but I still wonder if t.Data(any) might be easier to handle in test2json:

t.Data(map[string]any{"key": "testID", "value": "abafaf"})
t.Data(testIDs) // where testIDs is a struct that can be JSON marshalled

[jba]

With the -json flag to go test, we could write t.Attr(key string, value any) as

=== ATTR  TestName <key> <value>

where <key> is a JSON string and <value> is a JSON value, encoded on a single line.

test2json could easily split that into parts and output

{"Action": "attr", "Test": "TestName", "Key": key, "Value": value}

It wouldn't even need to parse JSON to make that transformation. It just has to know how to skip over a JSON string, so it can find the whitespace between <key> and <value>.

So I agree @dnephin, encoding the value as JSON from the start makes the most sense.

To answer your questions: spaces are allowed in keys, and the restrictions on the value are same as for json.Marshal. In particular, newlines in strings are fine.

If go test is called without -json, we have a choice. We could do the exact same thing, or we could output a more human-readable form, closer to slog's TextHandler. I'd vote for the former, since JSON is readable enough and the main use for this feature is with -json.

[rsc]

I still don't think we understand exactly what we need here yet. I looked at gotestyourself/gotestsum#311 (comment) but I don't understand that either. If the test ID would be coming from an environment variable then it seems like it would be the same for every test, which suggests not something that you do with a testing.T. The link to TestRail goes to a marketing page, not something that explains how TestRail uses IDs.

Can someone explain how they would use this, with links to the systems they would use it with - not just advertising pages but the technical details of how those systems would consume the attributes?

[nine9ths]

My use case is getting our go test results into Allure (https://github.com/allure-framework). Allure is a test report tool that allows you to filter test results on a variety of metadata as well as display metadata in the report such as links to ticket tracking software and attachments (like an output file for a specific test). A live demo is here. A diagram of the Allure processing model is here: https://allurereport.org/docs/how-it-works/

There is prior art for Go Allure adaptors:

• https://github.com/ozontech/allure-go
• https://github.com/dailymotion/allure-go

However, both of these require writing Allure tests, not Go tests. I'd prefer to be able to just annotate my Go tests with the various metadata fields Allure tracks and then post process my go test json into the Allure result json.

There is prior art for this as well here https://github.com/ilyubin/gotest2allure. This provides a version of this functionality by emitting specially constructed log messages and then using a custom parser to extract the metadata back out

I'd like to make round-tripping this metadata more robust than relying on specific log message prefixes, and it would be nice if I didn't have to tie myself to a specific report framework in my implementation. Ideally data like a link to a ticket tracker should be able to be implemented generically within my test, and provided it is easily discoverable in the json output, any test reporting framework should then be able to pull the link back out with a suitable adaptor.

[seankhliao]

This proposal is becoming more like a partial solution to #41878

[dnephin]

@seankhliao #41878 seems to be about getting access to the data that is already provided by test2json. This proposal is about data that is not yet available in test2json output. I don't see how they overlap. Can you elaborate?

[dnephin]

still don't think we understand exactly what we need here yet.

@rsc I think there are two use cases that have been shared so far. I'll try to summarize.

Use case 1 - integration with test management systems

Some test management systems track tests using their own IDs. Anyone using these systems needs a way to map the Go test name to the ID used by the test management system. Existing solutions are unreliable and require parsing log output.

Examples: TestRail, Allure framework

This example is in python, but I think it makes it very clear: https://www.browserstack.com/docs/test-management/upload-reports-cli/frameworks/pytest. Each test uses record_property("id", "TC-#####") to identify itself using the TestID of the test management system.

t.Attr would make this possible in Go. Each test uses t.Attr("testID", "..."), a tool would parse the test2json output, find the testID attribute, build a report from that data, and send the report to the test management systems API.

Use case 2 - links to external logs and reports

Any test that runs multiple goroutines (or multiple processes) that produce significant log output need some place to store the logs. Trying to output all the logs from concurrent operations to a single stream often makes the test output unusable.

Instead of a mess of interleaved logs, each process or goroutine writes the logs to its own log file. When a test fails the user may need to dig through those log files to find more information about the failure.

There's no convenient way to expose those logs files (or links to logs) from the test2json output today.

Examples: integration tests across telephony infrastructure, I've experience this same problem when working with https://github.com/hashicorp/consul test suite.

Ci systems have some basic support for this today using test artifacts (see artifacts docs on github actions, circleci, gitlab CI), but all of those require local files, there's no way to create a link to external files.

t.Attr solves this problem. Each test would use t.Attr("processname.logfile", "https://..."), a tool would parse the test2json output, and create a report of failed tests that provides links to external artifacts. This would make it much easier to debug large integration tests.

[seankhliao]

#41878 is about giving the user the raw data to output in a format they need directly, rather than trying to parse go test's output. Additional metadata may be a part of that.

---

Rather than trying to hack links together with attr references, #59928 would be a better solution to output from parallel tests.

[dnephin]

I would expect that #41878 still needs some way to become aware of this data. t.Attr seems like that mechanism, so my impression is that #41878 could complement this proposal, but does not replace it.

Rather than trying to hack links together with attr references, #59928 would be a better solution to output from parallel tests.

I am looking forward to #59928, but I don't think it solves this problem either.

In one of the examples I linked the tests exercise a system that spans multiple machines. The logs for those requests are not in the current process, so t.Output() doesn't help. The test code can construct a url to find relevant logs from whatever system receives them, but trying to export all of the logs to the test process is often not feasible.

Even when all the logs are available on the machine, if any of the concurrent processes/goroutines have debug logging enabled, trying to output all of the logs to stdout makes the test difficult to debug. It's much easier when those verbose logs are sent to separate files and only the test writes to stdout.

[jba]

So @nine9ths, would the functions in pkg/allure change like this?

Before:

func Feature(t *testing.T, feature string) {
	t.Logf("%s%s", prefix.Feature, feature)
}

After:

func Feature(t *testing.T, feature string) {
	t.Attr("feature", feature)
}

[nine9ths]

@jba presumably, or better yet the entire pkg/allure goes away and I just call t.Attr("feature", feature) directly in my tests.

[aclements]

It seems to me that there are three use cases here:

1. Attaching a test management system's test ID with a test.
2. Attaching labels to a test that allow collections of test results to be filtered by those labels.
3. Attaching artifacts to a test.

I don't really understand 1. The name of a test is already a unique and stable identifier. Why does a test management system need another ID? What purpose does it serve?

I can see the use of 2, and perhaps it doesn't matter that I don't understand 1 because 1 can be subsumed by 2. However, case 2 also has clear overlap with Go sub-benchmark naming, and I'm not sure how to tease them apart. I think we use this much more extensively in Go sub-benchmarks, where sub-benchmarks are conventionally named Name/key1=val/key2=val/ and we have a whole domain-specific language and set of tools built around filtering on names formed this way.

Use cases 2 and 3 seem very different to me. In 2, the labels are independent variables, whereas in 3 they're dependent variables. For example, it wouldn't make sense to filter on metadata that is links to artifacts. And presumably you want tools to be able to identify and copy artifacts around without getting confused by metadata that's just meant to be filtered on.

[greg-dennis]

I think a good example of use cases are those used for JUnit XML test properties today, e.g.:
https://github.com/testmoapp/junitxml?tab=readme-ov-file#properties

@aclements, I would say the use cases definitely include use case 3 (links and references to artifacts). It also includes labels, which is roughly your use case 2, but usually for a purpose that is bit distinct from subbenchmarks. I think the purpose of the metadata labels is less about filtering the results from an individual test run and more about enabling tooling to aggregate results by label across many test runs of potentially many different tests. The subbenchmark key-value pairs seem primarily about making "intra-test" distinctions, whereas the metadata labels are more "inter-test" ones.

I don't foresee any request for go-supplied tooling to filter by metadata labels in the way that is provided for subbenchmarks. Just the opposite: the ask is to populate these values in the json output, which enables external tooling to do what it wants with that information. Many test runners populate XUnit XML test properties for the same reason.

[adonovan]

paging @neild

[neild]

Is there anything that can't be done, less elegantly, with t.Log and some custom well-known syntax? That is, what's the difference between t.Attr(name, value) and t.Logf("ATTR: %v %v", name, value)?

Most of the justifications for this feature request are to improve the integration between test output and various test management systems. It seems like it should be possible to demonstrate that usage that integration using t.Log and some custom syntax, which would give us a better understanding of what a test-metadata API needs.

[jba]

@neild It wouldn't only be less elegant, it would be more error-prone. You'd be parsing "ATTR x y" out of raw output.

[greg-dennis]

This external tooling wants to have the metadata attributes structured in json. The title of this issue is specific to support in test2json.

[jba]

It seems like it should be possible to demonstrate that usage that integration using t.Log and some custom syntax, which would give us a better understanding of what a test-metadata API needs.

Here is one concrete example that uses t.Log now but would use t.Attr: https://github.com/ilyubin/gotest2allure/blob/master/pkg/allure/allure.go
This is the allure package mentioned in #43936 (comment).

[aclements]

Thanks for the link to the example attributes from JUnit.

It looks like sometimes these are extraneous variables. That is, a variable that may affect the outcome of a test, but that the test can't directly control. The JUnit examples are frustratingly abstract, but I think "commit" would be an example of this. You wouldn't put these in the subtest part of the name because they can vary and you don't want that to break historical lineage. A parallel to benchmarks is that we report the CPU running the benchmark--it's important to interpreting the results and it's not a measured output ("dependent variable"), but it's not something a benchmark can directly control either.

Some of these are outputs beyond pass/fail or the test log. "Attachments" are a clear example of this, as is allure.Issue.

In the JUnit example, these still seem like they're being used for a lot of different things. That's not necessarily a bad thing, but it makes it a lot harder to understand what this is for, and harder to communicate in a doc comment to a potential user when they should use a t.Attr API.

Request: It would help if the people who believe they understand the intent of this proposal could write an example prescriptive doc comment for (*T).Attr(key string, value any).

Many test runners populate XUnit XML test properties for the same reason.

@greg-dennis (or anyone), could you point to concrete examples of test properties used by xUnit test runners? We have a handful from Allure (thanks @jba).

This external tooling wants to have the metadata attributes structured in json. The title of this issue is specific to support in test2json.

I don't see this as a strong argument. It would be easy to JSON encode more complex data and pass that to t.Log. It doesn't seem to add much value for the testing package itself to do this. In fact, there's some downside because it adds a new dependency on encoding/json to every test binary.

I see this as two basic questions: framing and standardization. You can emit these attributes in the test log and a tool will be able to pick them up. This proposal would raise the framing to a level that is harder (though not impossible) to break. That's certainly nice, but I haven't seen evidence that it's critical.

The more critical ask here is for standardizing the concept of test metadata/attributes/properties and how they are communicated in Go test output.

[neild]

Interestingly, the JUnit XML supports defining properties as part of the test output to support systems (like Go) that don't provide a way to set a property:

<testcase name="name">
  <properties>
    <property name="name" value="value">
  </properties>
  <system-out>
    Alternate way of setting the same property:
[[PROPERTY|name=value]]
  </system-out>
</testcase>

JUnit properties are a string key and a string value. Are there examples of commonly used systems that use anything other than a string value for test properties?

[neild]

In general, we add new APIs to std when the API is some combination of very useful, expected to be widely used, and difficult to implement outside of std. (For example: We added errors.Join because there were multiple third-party packages implementing equivalent functionality, those packages were widely used, and implementing the feature well without modifying the internals of errors.Is/As is difficult.)

Test properties can be emitted without any changes to std, by writing the properties to the test output. The JUnit XML format supports properties in test output (https://github.com/testmoapp/junitxml?tab=readme-ov-file#properties-output). For tests whose output is eventually converted to JUnit XML, using this output format should allow for conveying properties from the test to any eventual consumer without requiring any changes to the testing package or test2json.

For simple cases, a simple helper function should suffice to write properties in this format to the test output:

func TestProperty(t *testing.T, name string, value any) {
  // JUnit XML property syntax. The \n is unfortunate but necessary (I think).
  t.Logf("\n[[PROPERTY|%v=%v]]", name, value) 
}

A more sophisticated implementation could handle any necessary escaping in the name and value.

We do not yet have evidence that test properties will be widely used. For example, the "github.com/ilyubin/gotest2allure/pkg/allure" package is a good example of test properties being used in practice, but it has no importers as reported by pkgsite.

So I think that we should:

1. Add no new API for now.
2. Recommend that users who want to emit test properties use the JUnit XML output format.
3. Revisit this issue in the future if there is evidence for common use of test properties.

[greg-dennis]

Some added notes and comments.

The JUnit format itself does not authoritatively support properties in the test output. That referenced section discusses a non-standard encoding, supported by a subset of JUnit XML consumers, for adding properties in individual test cases (as opposed to test suites). It's a workaround to the fact that the JUnit XML schema specification only supports properties at the testsuite level. Some of the most common JUnit XML consumers, including Hudson/Jenkins, have no support for this workaround.

Even if one decides to use JUnit XML, they still need a way to convert the test output to JUnit XML. The most popular, if not only, project for converting Go test ouput to JUnit xml is (go-junit-report), and it has no support for this syntax today. It currently only supports properties being passed in at the command line to the conversion, which makes it very difficult to encode the properties in the test itself. Maybe support for the workaround encoding could be added? cc: @jstemmer in case he has thoughts.

Even if such support is added to go-junit-report, it's a solution only for properties at the individual test case level, whereas proper support for properties would probably allow you to add them at the suite/file level, too. Finally, this solution requires (unless you are very clever -- more on that in a moment), that you invoke the conversion separately, forgoing the convenience of just passing the -json flag.

This is all to say that, as of this writing, there isn't a straightforward alternative path for getting test properties into any sort of structured output. The only project I know of that successfully worked around all these limitations is ondatra. To allow the test to populate the JUnit XML test properties in a convenient way, it hijacks os.Stdout and pipes it to go-junit-report. It's probably not something we want emulated often.

I don't work at Google anymore, but I know from my time there that test properties were used extensively internally, including in Go tests. If you're looking to gather more data, it might be relatively easy to gather data from those internal uses, since the tests used a dedicated function for adding properties that would be easy to search for.

[neild]

Adding support for properties-in-output to go-junit-report seems like an excellent path to generating data on the general usefulness of properties.

Even if such support is added to go-junit-report, it's a solution only for properties at the individual test case level, whereas proper support for properties would probably allow you to add them at the suite/file level, too.

I think this is the first time in this discussion that the need for suite-level properties has been mentioned. What is a "suite" in Go terms? I'd think probably a single test package, but you say "suite/file" so perhaps it's a file instead? Or do we need both suite- and file-level properties? What does the API for this look like?

This sort of thing is why I think we need working examples of properties in use in the real world before we can attempt to design an API for the testing package. We can't tell if we've got the API right if we can't see the actual use cases we're satisfying. And given that it is possible (if clumsy) to emit and consume properties without testing package support, this doesn't need to be a chicken-and-egg situation.

I don't work at Google anymore, but I know from my time there that test properties were used extensively internally, including in Go tests

Thanks for that reference. I have not run into this package before. I poked around and for anyone with access, the place to start looking is AddProperty under google3/testing/gobase. It does appear to have extensive internal usage--without any direct support from the testing package.

[greg-dennis]

Yeah, I should have said suite/package, not suite/file, because of course packages can have many test files. Internally at Google, the go test output is transformed into Sponge XML, a superset of JUnit XML which allows properties at both the suite- and test case-level, where "suite" is at the level of a single blaze test target. I don't recall how the internal test runner is wired up to get the arguments to AddProperty into the XML, but I don't think it's trivial for external Go users to mimic.

If you're asking me for a proposed API, I would say something like func (t *T) Attr(key, value string) for adding testcase-level attributes and func (m *M) Attr(key, value string) for adding suite-level attributes would suffice. And maybe for Benchmarks and Fuzz tests, too. Whether to choose the name "Attr," "Prop," "Meta," or something else can be debated, but that's the API I would start with as a working solution.

[aclements]

I'm still looking for answers to a couple questions:

• Could someone write an example prescriptive doc comment for (*T).Attr(key string, value any)?
• What are some concrete, compelling examples of test properties used by xUnit test runners? We have a handful from Allure (thanks @jba).

[jba]

• Could someone write an example prescriptive doc comment for (*T).Attr(key string, value any)?

"Attr outputs the key and value in a form that allows them to be reliably extracted from the test output. It is useful when test output is consumed by tools outside the Go toolchain, and especially with the -json flag to go test.

Test authors can use Attr to associate information with a test, such as:

• paths to artifacts associated with a test, like output files
• test identifiers for some external test framework, such as a "story" (in the agile development framework) or "behavior" (behavior-driven development)

Programs that process test output can find these attributes reliably when using go test -json (or equivalently, the test2json command) because each call to Attr corresponds to a single TestEvent whose Action field is set to "attr". This is in contrast to using Log or Logf to output this type of information, requiring parsing of raw test output."

[greg-dennis]

What are some concrete, compelling examples of test properties used by xUnit test runners? We have a handful from Allure (thanks @jba).

The properties aren't generally used by xUnit test runners. They are output by the tests and consumed by tooling that is external to the test runner, usually written in-house by companies for test tracking and data analysis. The examples of allure and ondatra above may be the only two open-source examples to be found, and those aren't really examples of using the properties but really provide syntactic sugar for outputting them. As for examples of the types of properties, those have been given in the discussions above and in @jba's prior comment. They include: paths to test artifacts (e.g. logs and screenshots), identifiers that associates the test to an external test plan (e.g. a Jira issue ID); and labels that indicate the dimensions under which the test invocation was run (e.g. the platform or environment in which it was executed).