After updating from v1.16 to v2.9, JUnit report fails with "open <file>: no such file or directory"

[nathanperkins]

We are upgrading from ginkgo v1.16.5 to v2.9.5 and it seems that the junit report broke in the process. Now our tests are all failing during the report outputs by saying no such file or directory exists.

Script:

rm -rf "${TEST_OUTPUTS}"
mkdir -p "${TEST_OUTPUTS}"

${E2E_TEST_BINARY} \
  -test.v \
  -test.count=1 \
  -ginkgo.v \
  -ginkgo.focus="${TESTS_FOCUS:-}" \
  -ginkgo.junit-report="${PWD}/${TEST_OUTPUTS}/junit_test_results.xml"

Output:

+ rm -rf test_outputs
+ mkdir -p test_outputs
+ ./e2e -test.v -test.count=1 -ginkgo.v -ginkgo.focus= -ginkgo.junit-report=/root/test_outputs/junit_test_results.xml

...

------------------------------
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
  [FAILED] in [ReportAfterSuite] - /home/prow/go/src/..../vendor/github.com/onsi/ginkgo/v2/reporting_dsl.go:155 @ 05/29/23 08:17:25.265
[ReportAfterSuite] [FAILED] [0.019 seconds]
[ReportAfterSuite] Autogenerated ReportAfterSuite for --junit-report
autogenerated by Ginkgo
  [FAILED] Failed to generate JUnit report:
  open /root/test_outputs/junit_test_results.xml: no such file or directory
  In [ReportAfterSuite] at: /home/prow/go/src/.../vendor/github.com/onsi/ginkgo/v2/reporting_dsl.go:155 @ 05/29/23 08:17:25.265
------------------------------

In one of my test runs, I added ls -la to ensure the test_outputs folder is created before running:

+ mkdir -p test_outputs
+ report_file=test_outputs/junit_test_results.xml
+ ls -la
total 450904
drwx------ 11 root    root        4096 May 29 04:36 .
drwxr-xr-x 23 root    root        4096 May 29 03:51 ..

...

-rwxr-xr-x  1 root    root    65774440 May 29 04:28 e2e

...

drwxr-xr-x  2 root    root        4096 May 29 04:36 test_outputs

The test binary is compiled by our CI and sent to the system under test.

go test -c -o bin/e2e --tags=e2e_tests ./testing/e2e

[nathanperkins]

Based on what I'm seeing in issues, it seems the semantics of these flags changed and it's not clear to me how to get the same result we had before where the report file was generated inside a folder that is next to our test binary.

The ginkgo CLI allows specifying the output-dir, but this flag isn't present when running ginkgo tests with go test. We compile the test suite with go test -c so that we can send the test binary to our system under test without having to add a golang SDK or ginkgo CLI over.

Is it still possible to do this? If not, what is the suggested workflow here?

[onsi]

hey @nathanperkins ordinarily I'd say "use the ginkgo cli and set --output-dir alongside --junit-report" but in this case (and I just looked over the code again) where you are invoking a single precompiled test binary in series I would expect -ginkgo.junit-report with an absolute path (as you have done) to work. Can you confirm (in one of your tests) that you can os.Create a file at that path - this is what the JUnit report code in Ginkgo is doing. That might help you make some progress debugging it.

If you want even more control while you're debugging this you can drop the --ginkgo.junit-report flag and add something like:

var _ = ReportAfterSuite("junit", func(report Report) {
   reporters.GenerateJUnitReport(report, "/path/to/report.xml") 
})

that'll give you a bit more control over how/when the GenerateJUnitReport function is called.

In general, I do recommend using the ginkgo cli wherever possible. In your case you could continue to use go test -c to compile the test binary (or use ginkgo build instead - it's basically the same command at the end of the day). You could then ship both the test binary and the ginkgo binary to your system-under-test (no need for the Golang SDK) and then invoke ginkgo -junit-report=junit.xml -output-dir=./${test-outputs} -v ${E2E_TEST_BINARY} - Ginkgo will detect the precompiled suite and invoke it for you (there should be no need for any go cli/SDK at that point). The biggest benefit here is the ability to orchestrate and run the precompiled suite in parallel with ginkgo -p... something that just isn't possible without the Ginkgo CLI given Ginkgo's current architecture.

[nathanperkins]

@onsi thank you for the quick and detailed response!

Based on your comment, I looked into this more deeply. I found that my BeforeSuite code was deleting our test_outputs folder before running the tests. My guess is that this worked before because v1 must have automatically created the folder at some point, but v2 doesn't.

I see some relevant discussion here: #554 that implies v2 should recreate this folder. Maybe this only happens if you run with the ginkgo CLI.

[onsi]

yes it only happens if you run with the cli (and the only folder created is the one passed in to output-dir) but honestly this isn't by design. the intent is that users are runing Ginkgo suites via the cli and that is the most supported path. calling into the binary with your own flags generally works but it's not the workflow that Ginkgo's integration tests cover (or that most people tend to use). with that said, there's absolutely no harm in having GenerateJSONReport and GenerateJUnitReportWithConfig create the directories necessary to successfully write the file. If you're up for a PR for both of those I'd be happy to pull it in.

[nathanperkins]

@onsi, that sounds great. Can you assign this issue to me?

[onsi]

sweet, thanks!