Configure MustPassRepeatedly at suite level

[maxcleme]

We are looking at a way of configuring MustPassRepeatedly programatically at suite level before starting it, however it is not possible at the moment.

Ideally, something similar to FlakeAttempts, where it can be set at node level using the decorator but also at suite level .

Any insight on how we could perform this? Is there any reason why it is not present at suite level?

[onsi]

hey - you can use ginkgo -repeat=N and the CLI will orchestrate repeats and require them all to pass

[maxcleme]

@onsi Thanks for fast answer.

Sadly we aren't using the CLI, I forgot to mention that!

We'd like to set MustPassRepeatedly programatically before calling RunSpecs.

[onsi]

That isn’t supported yet (it’s just not implemented). i can imagine simply passing the decorator into RunSpecs would decorate all the nodes with MustPassRepeatedly and it would just work. If you want to work on a PR that does that that would be great.

Can you share why you aren't using the CLI? It’s required for running specs in parallel and I’d love to learn what issues you’ve bumped into.

[maxcleme]

I've made this exactly the same way as how FlakyAttempts are handled a suite level and it works using my fork, I'll open a PR so we can discuss about the required tests I guess.

Regarding our workflow, we compile the test binary and then spread it across N machines, each of them is calling the binary with different set of flags. These machines don't have Go or anything else installed.

[onsi]

thanks for the PR! i’ll take a look as soon as i can.

Regarding our workflow, we compile the test binary and then spread it across N machines, each of them is calling the binary with different set of flags. These machines don't have Go or anything else installed.

that’s great! you can also compile and distribute the ginkgo binary and then run ginkgo <flags> path/to/your/test/binary and it will Just Work™! that way you can run with -p to run the binary in parallel on each machine (eg if it’s got more than one core)

[maxcleme]

Works fine so I'll close!

On a side note, any idea how we could get the timing of all these N executions for each tests during reporting step (ReportAfterSuite("", func(report Report) {})) ? I guess I could rebuild the timeline with SpecEvents but it looks like reinventing the wheel 🤔

[onsi]

hey @maxcleme - the individual times aren't stored by Ginkgo but you can use Report Entries do something like this at the top level (e.g. right after the TextX function that invokes RunSpecs():

var _ = BeforeEach(func() {
    t := time.Now()
    DeferCleanup(func() {
		AddReportEntry(fmt.Sprintf("Run %d", CurrentSpecReport().NumAttempts, time.Since(t))
    })
})

If you share more about how you post-process the specs to generate reports I can offer some examples for how to then use these report entries. But, it's basically just data and data transformation at that point.

[maxcleme]

Thanks for the answer, to be fair that's what I kind had before using SpecEvents to recompute the test timings. I like SpecEvents because it doesn't add BeforeEach each nodes only for reporting purposes, everything can be self-contains within ReportAfterSuite.

Regarding context, we are reporting test suites in two formats, ideally we'd like:

1. Custom format that we send using an http API call (this one is parsing ginkgo.Report to our own type), there is no real limitation here since we already do a bunch of transformation.
2. JUnit report using built-in --ginkgo.junit-report (for UI purposes), however this one is reporting N iterations as one (which makes sense in Ginkgo model) where we'd like explicitly N tests for N iterations. At first we wanted to edit ginkgo.Report to add artificial SpecReport however since Report is passed by value to ReportAfterSuite we can't, we can only update already existing SpecReport. One way could be have our own custom reporter, tweaking that behaviour assuming it is too specific to upstreamed it. Post-processing the JUnit report might also be a viable alternative.

[onsi]

hey @maxcleme - ah, understood. yes parsing the SpecEvents is probably your best bet then as anything else will introduce an additional node into the report.

also makes sense about JUnit for UI - so many tools are built on top of JUnit's report format which is a bit of a shame since the format is quite anemic and not super well defined. One idea for you: you could take over generating the JUnit report yourself. You can mutate the ginkgo.Report as you please within ReportAfterSuite, adding additional SpecReports if that's what you'd like to do, and then call reporters.GenerateJUnitReport(modifiedReport, "path/to/report.xml"). It's not as convenient as --ginkgo.junit-report but it'll get the job done and give you the control you need (and you're correct that something this specific probably belongs in your suite and/or a separate support library rather than in Ginkgo core).

[maxcleme]

Right, reporters.GenerateJUnitReport is publicly exposed so that would help a lot! 🙇

Thanks for those pointers! ❤️

[onsi]

wonderful - glad to be helpful and thanks again for the PR!