[BUG] Infinite running benchmarks

[Kashio]

Describe the bug
Benchmarks fixture don't seem to run unless in debug with breakpoints

System

• OS: Windows 10 x64
• Compiler and version: msvc 19.16.27034 x32/x64 (both)

To reproduce
You can clone my repository
https://github.com/Kashio/A5
and build with cmake

Expected behavior
Benchmarks should run normally and not get stuck somewhere infinitely

Additional context
If I place a break pointer in my first defined benchmark function
BENCHMARK_F(MultipleFixedAllocations, CAllocator)(benchmark::State& state)
and step through it to completion (by stepping into BenchmarkMultipleAllocations and then setting another breakpoint at the end of BenchmarkMultipleAllocations and jumping to it)
Then I manage somehow to get the first benchmark case finished, but unless I'm stepping through code with breakpoints and just running it then it get stuck somewhere and never finishes and memory goes up (probably by my allocators objects since they're never freed) but it's as if it constructing them infinitely or something but I'm not even sure about that since I even placing breakpoints on their constructors don't get hit.

You can go to this commit
https://github.com/Kashio/A5/tree/25aa53e372a4c6ae5cc02358383ac1d06f137792
which was before I've changed the benchmark function to fixtures.
Before doing that everything worked fine.
The reason I had to change that to fixtures instead of globally defined functions is that I wanted some setup code specific for these cases as can be seen on the master.

I'm not sure this is a bug, probably me not doing something right with setting the fixture tests.
I'm pretty sure this is not a bug in my library code causing it since I've linked to a commit where everything worked fine without fixtures.

Here's a gif showing the problem that no tests are run and the memory just keeping going up:
https://gfycat.com/physicalimmaterialgallinule

And Here's the same run with me just stepping with breakpoint and see how the first test somehow finished but then everything get stuck again:
https://gfycat.com/windyphonydamselfly

[Kashio]

I realized my problem the Setup code is being called for each iteration/test case.
I think this is counter intuitive that you define test case in fixture with BENCHMARK_F as it doesn't really make it a function of the class but make it a derived class and so Setup will be called for each test case separately instead of one Setup for one fixture. This and the fact that it also run for each iteration apparently (because I see Setup called multiple times for each created derived instance) couldn't be figured from the docs so I think a better documentation on what these Macros actually do and how this might affect you should be added.

[dmah42]

leaving this open as it's a FAQ and we probably do want to figure out the difference between a fixture and a 'test suite'. googletest has this notion with both static setup and teardown for the suite, and non-static setup and teardown per test unit.

i wonder if we should have something similar.

[saketh-bandi]

Hi! I’m interested in working on this. Is the current direction still to explore suite-level setup/teardown semantics similar to GoogleTest, or would maintainers prefer this be addressed as documentation only? Happy to put together a small design/implementation proposal if the API change is still wanted.