post_run for Test / TestBench

[felixn]

Are there any arguments against adding a post_run callback to Test / TestBench?
Similar to post_check, but called regardless of whether test passed or failed.

My use case: Processing log files written to output_path, even for failing tests. For example, archiving log files from automated simulation runs.
I know I could do this without a callback by parsing the mapping text file to get to the output directories, but the callback would be a much easier solution.
Or is there another easier way that I'm not seeing?

And I believe there could be other use cases where this callback would be beneficial.

If there is general consensus that this suggestion is useful, I could work on implementing it.

[kraigher]

The junit xml file contains all stdout from the passing and failing tests. Have you considered parsing this file?

[felixn]

You're right, and that's what we're using to get the stdout.
But there are additional files we write to output_path to be checked later by post_check, similar to how it's suggested here:
https://vunit.github.io/user_guide.html#special-generics
We want to archive those log files whether the test passes or fails.
Currently we're archiving the whole test_output directory, but that contains additional unnecessary files and the subfolders don't have userfriendly names (i.e. when later looking at the archived logs, the user may still have to consult test_name_to_path_mapping.txt).
With the callback, we can copy exactly what we need from the specific output folders (already works with post_check, but of course only for the passing tests).

[umarcor]

I have a similar concern because of #478. I'm currently using post_check just to expose output_path to the run.py, in order to do some tasks after vu.main():

https://github.com/dbhi/vunit/blob/vunitcosim/examples/vhdl/external_buffer/run.py#L38-L39

https://github.com/dbhi/vunit/blob/vunitcosim/examples/vhdl/external_buffer/run.py#L121-L127

I think that it would be interesting to enhance vu.main() to optionally return instead of terminating the execution with sys.exit. The returned value would be an object/struct/dictionary instead of a exit code only. It would allow users to do post-processing of the results without having to parse/browse either XML, JSON or vunit_out (test_name_to_path_mapping.txt).

In the context of co-simulation of Python and GHDL through 'external arrays', it is required to 'save' the CLI args that VUnit would pass to GHDL at runtime. However, I think that changing from sys.exit to return alone can already be useful for @felixn too, as he would be able to process in the run.py file (without callbacks) and then forward the return code to a sys.exit to preserve the exit code.

[felixn]

Yeah, @umarcor's suggestion would work for me as well. Any way to get to output_path after simulation would be welcome, whether it's through a callback of an individual Test / TestBench or a return value of vu.main(). I think @umarcor's suggestion is more flexible, because it would allow for access to all testresults at once instead of just scoped to individual tests.

[kraigher]

I am not against post_run it makes sense.

Alternatively I have had thoughts about making a class based hook interface where you would implement methods on the class and send an object encapsulating everything instead of many individual functions. It would not spam the public api with so many functions when adding hooks and the object can be used to share state between for example pre_config and post_check.

[kraigher]

main already has a post_run callback so should should not need to have try except arround main. It also ensures that - -list and - - compile wont run post_run since no tests have run.

[felixn]

The post_run callback is also a good candidate, but currently the Result object passed to it only exposes merge_coverage. We could get to more stuff via the private attribute _simulator_if, but that's hacky. If we extended the Result to also include information about all tests run, their result and their output path, then that would work for my use case. @umarcor, what about yours?

[kraigher]

Adding more public info to the result object passed to post run is reasonable.