Skip to content

Loading…

Parser cannot handle tests run with t.Parallel() #5

Closed
Redundancy opened this Issue · 3 comments

2 participants

@Redundancy

When running tests in parallel, the test result may not immediately follow the RUN line:

=== RUN TestA
=== RUN TestB
=== RUN TestC
--- PASS: TestA (0.00 seconds)
--- PASS: TestB (0.00 seconds)
--- PASS: TestC (0.00 seconds)

I may try and fix this in my fork and send you a patch.

@tebeka
Owner

How can you tell which output belongs to which test?

To quote the zen of Python - "In the face of ambiguity, refuse the temptation to guess.".

This should be fixed in the go test utility, see this issue for example.

@Redundancy

I guess the question would be - if we can't reliably determine the correct test for any given output, would it be better to fail to interpret the results entirely, or to throw away the ambiguous information?

We could track the number of in-flight tests unambiguously based on the start/stop messages, and for the unambiguous cases, just put it in there. Iff there's any ambiguity, we can mark those tests as ambiguous, emit a line to the console explaining the issue, and potentially replace the output from the test with a marker string of some sort so that the test maintainer can solve the issue.

I'd argue that the output capture is less important than the correct interpretation of the test results in a lot of the use-cases that require you to convert to xunit output?

@tebeka
Owner

I'm a firm believer in the "fail fast" methodology and not doing GIGO.

Since what the user will see is only the Jenkins web page with the results - they'll have no idea if some of the tests we ignored.

and for the unambiguous cases, just put it in there

Can you elaborate? I don't think there's a bullet proof unambiguous cases.

I'd argue that the output capture is less important than the correct interpretation of the test results in a lot of the use-cases that require you to convert to xunit output?

I don't follow. If "correct interpretation" is what's important - than there's nothing we can do (IMO).

Maybe there users can provide a command line option to go2xunit telling it it's parallel tests and try to do you best. And we state that in this case - GIGO might follow.

@tebeka tebeka closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.