proposal: Output format

[bkielbasa]

General idea

GoBDD should better integrate with external tools like TeamCity etc. What I suggest to add is a custom output file that will contain information like:

• feature/scenarios/steps etc
• time that every step/feature/scenario executed
• maybe memory usage?
• other statistics

The purpose of the file would be the starting point for other tools to generate reports/statistics or upload them to a SaaS for further analysis.

How would it look like?

The sample usage would be

go test gobdd.output=file.json

The file.json (but it doesn't have to be a JSON file - it's an example) can be interpreted in any way based on the end-user needs.

[mirogta]

Looking at https://pester.dev/docs/usage/test-results for inspiration…

Looks like just the file name is good for json, but when it comes to e.g. xml we would also need to specify the output format, because there is xUnitXML, jUnitXML, nUnitXML.

go test gobdd.outputfile=file.xml gobdd.outputformat=nunitxml

I assume if the filename extension is json, the format would be redundant, because it would have to be json… but considering potential different formats for the same extension, would it make sense to call the parameter gobdd.outputfile rather than just gobdd.output?

[mirogta]

Another point about the initial implementation of this: I like all the ideas, like time executed, memory usage etc. But for the first iteration, could we have just gobdd.outputfile=test.raw (or with gobdd.outputformat=cucumber) or something like that and write the output in a way close to how godog does it? I.e. output all feature files with scenarios and color coded steps - see the screenshot in #116

Maybe initially even without the timings (can be added later) or statistics (number of passing/failing/missing steps)… but at least color coded :-)

I'm worried that if we try to implement something generic, like output to json with all the bells and whistles, we only put the burden of parsing it and making it look nice (like the screenshot) onto the users of gobdd, rather than providing sensible default output (into a file, if we can't do it into the console the same way as godog, as you've mentioned earlier).

[bkielbasa]

thanks for the comments!

with scenarios and color-coded steps - see the screenshot in #116

I think it would be hard to put colors into a raw file :) but it can be an HTML file what would give as extra flexibility (different features in different tabs, tooltips, etc)

the raw version would contain only non-color output. But I like the idea to use gobdd.outputfile=test.html and gobdd.outputformat=htmlcucumber parameters

My initial idea was to create a "base" file that can be converted to anything you want but maybe you're right that it's a redundant step.

[mirogta]

I thought about using a raw file rather than HTML because it would be readable in a terminal where I run the command, or even in e.g. VSCode's output/terminal, and also in CI/CD tools that usually support parsing ANSI color codes, but not always support HTML output. Also it would be smaller and hopefully easier to produce than HTML.

So the raw file would just look a bit cryptic to humans, but after all it's a raw file.

You can have ANSI color coded text in a file and if you display it in a terminal which supports ANSI colors it should show the colors accordingly... Try this:

RED='\033[0;31m'
NC='\033[0m' # No Color
echo -e "${RED}[ERROR]${NC} Message" > test.raw && cat test.raw

So go test gobdd.outputfile=test.raw gobdd.outputformat=cucumber ./... && cat test.raw executed via terminal or via VSCode, would create the test.raw file with ANSI color codes in it and then display it in color. I hope it makes sense.

[bkielbasa]

@sagikazarmark WDYT? It makes sense to me

[sagikazarmark]

I agree that feature tests could potentially be formatted differently from the rest, but formats like junit wouldn't contain anything specific to GoBDD, so I'd let other tools (like gotestsum) handle them.

I wouldn't use file extensions to determine output format.

The rest makes sense to me.

[bkielbasa]

I think it's time to close the issue. Go has test2json tool as well as https://github.com/jstemmer/go-junit-report so everyone who wants can use it as he wants. Closing...