Changing path of html directory using a fixture value for parametrized test

[jnunez10010]

Hello everyone,

I was wondering if there was a way to change the output path of the html report after the test has run by using a test fixture which contains the name and parametrized values of a test i.e. :

during the test there is a fixture that creates a directory whose name is based on the values of the current test, for example :
output_path = /path/to/test/test_name_parametrized_val1_val2.

However since the value of the config.option.htmlpath seems to be given during the pytest_configure hook and the tests have not been intialized yet (output_path is not yet created and, even if it was, it is not reachable from the pytest_configure hook) I haven't been able to figure this one out yet.

I would really appreciate any advice on how to accomplish this and I apologize if the solution is trivial as I'm quite new to pytest...

Thanks and have a good day!

[BeyondEvil]

What's the purpose/use case of those directories?

[jnunez10010]

Hi, thanks for your reply.
In fact we have added some options to pytest to output results of fluid dynamics computations to a separate folder for each parametrized test variant.
We would like to include a separate html report for each test variant stored within the same output folder along with the rest of the computation results in order to have an easy overview of each test and its output log without having all of the tests listed on the same html report (mainly for clarity since we are launching hundreds of tests).

[BeyondEvil]

I see.

Since (currently) the report is generated, like you said, on first init - I don't think there's a straightforward way of doing this.

One approach you could take is to use the pytest_runtest_makereport hook and then create a folder and put all the assets you want there, and then in the main report link to that folder using the url (or text) extra.

Another approach is simply to run pytest for each test. You could get all the tests using the --collect-only flag, parse the output and then loop over it - invoking pytest --html.... for each test.

[jnunez10010]

Thanks again for your reply @BeyondEvil !

In the end, I ended up using your second suggestion (sort of) and I'm launching tests one by one using reFrame which has a dedicated slurm scheduler allowing me to run the tests in parallel. Since reframe cannot see the output_dir function fixture from pytest, I write the ouput_dir into a file so that I can can copy the report.html from reframe stage dir (where the computations are launched) towards this output directory (where all the computaition artifacts are stored).

I think this works well enough for me for the time being but I would like to ask before closing this thread if there are any plans to add a feature to pytest-html (amazing job btw!) to allow someone to modify the html-path dynamically from the fixtures created during the tests?

Thank you & have a good day!

[BeyondEvil]

You're welcome @jnunez10010

In v4 (aka next-gen) of this plugin, we actually write the report for each test.

There are mainly two reasons for that:

• For long running test suites you can actually open the report and view the progress and don't have to wait until the suite finishes
• We're planning to add streaming support, so that you can stream the report to a different location

I'll see what how much work it would be to support writing a new report for each test.

I'll keep this issue open so that I can track it. 👍

[jnunez10010]

Thanks a lot 👍 !

[BeyondEvil]

4.0.0rc1 is out.

Question: Should this fixture write a new report "per test" or do you just want to be able to set the path to the report using a fixture?

[RonnyPfannschmidt]

I believe it's right to decide that when one wants per test set files, one has to run one pytest invocation per target file,

[jnunez10010]

Thanks a lot for this. Both would be useful IMO but if I had to choose one, I would rather be able to set the path of the report using a fixture since (in my personal case) I already invoke each test variant separateley with pytest (I'm just not able to send it to the right location). In any case, thanks again and congratulations for your all your achievements and work on this great addition to pytest!

[BeyondEvil]

No harm in adding a fixture, if it's possible without too much work.

It will be a Session-scoped fixture in that case.

[RonnyPfannschmidt]

@BeyondEvil my understanding here is that the feature request entails to report to multiple html reports from a single restrun by setting the path in a fixture

when one invokes pytest.main multiple itmes instead, things can easily be passed as a argument already

[BeyondEvil]

Maybe @jnunez10010 can clarify why it isn't possible to pass the argument today using command line, pytest.ini or setup.cfg, but would be possible using a fixture?

[jnunez10010]

Hello and sorry for the late reply,
When using pytest for parametrized tests, test names are generated dynamically as a function of the different values taken by the parameters. Since in my particular case, I use these test names to create output folders where I would like to have a separate html report by test, I'm am not sure that I can be able to use the command line or the pytest.ini to do this (since I do not know the name of each test before launching pytest and the folders are generated dynamically within each test)?
Please note that I am quite new to the pytest ecosystem, so it is totally possible that I'm missing out on something here and that this feature request might be superflous (if this is the case, I apologize in advance).
Thanks for any info!

[BeyondEvil]

Why do you need (want) a separate report per test?

[jnunez10010]

Hi, thanks for your reply.
I would like to include a separate html report for each test variant along with the rest of CFD computation results in order to have an easy overview of each test and its output log without having all of the tests listed on the same html report (mainly for clarity).
When all the testing is done, I use sphinx to generate an overview report that contains links to plots of each test variant and its respective report.html that contains the output log.
If i had all the tests results in the same report.html, this would be add odds with the rest of the plots from each variant that only contain results from that single test.
I have actually a workaround for this by using an external scheduler to run each of the test variants within a separate subprocess, so if this would be too specific of a development or not useful for others, there is no problem at all from my side!
Thanks again for your help and advice!

[BeyondEvil]

It's good you have a workaround as I think this is out-of-scope for this plugin unfortunately.