Before all iterations the fixture setup/Teardown code should be called.

[tahsinH]

Take the following code, which uses a Fixture
Since, the time to run the test is small, the framework decides to run the same test point multiple times to get a stable time. However, while trying to do so, it does not execute the setup/teardown code.
While the issue can be easily worked around, this kind of behavior violates the x-unit pattern fixture setup/teardown workflow.

The idea of the test is to time how long it takes to insert a set of strings to an EMPTY unordered_map.

 class MySUTClass
   {
    public: 
     void insert (std::string key, int val)
     {
       auto search = myMap.find(key);
    if(search != myMap.end()) {
      std::cout << "tried to insert duplicate key in the same map " << std::endl;
      std::abort();  
    }  else {
      myMap.insert({key, val});
    }
  }

  void clear()
  {
    myMap.clear();
  }

 private:

  std::unordered_map<std::string, int> myMap;
};



class MyFixtureTest : public ::benchmark::Fixture
{
 public:
  MyFixtureTest()
  {
    std::cout << "ran ctor" << std::endl;
  }

  void SetUp(const ::benchmark::State& state)
  {    
    std::cout << "ran SetUp" << std::endl;
  }

  ~MyFixtureTest(){}

  void TearDown(const ::benchmark::State& state)
  {
    m_Table.clear();
    std::cout << "ran teardown" << std::endl;
  }


 public:
  MySUTClass m_Table; 
};


BENCHMARK_DEFINE_F(MyFixtureTest, TableInsertion)(benchmark::State& st) {

  std::cout << "outside the while loop\n";
  while(st.KeepRunning()){
    st.PauseTiming();
    std::cout << "inside the while loop" << std::endl;

    std::set<std::string> stringSet;
    stringSet.insert("abc");
    stringSet.insert("def");
    stringSet.insert("hfash");

    st.ResumeTiming();    
    for (auto const & id: stringSet) {      
      m_Table.insert(id.c_str(), 1); // contrived example, all keys have same value;
    }
  }
}  

BENCHMARK_REGISTER_F(MyFixtureTest,TableInsertion);`

Benchmark Time CPU Iterations

ran SetUp
outside the while loop
inside the while loop
ran teardown
ran SetUp
outside the while loop
inside the while loop
inside the while loop
tried to insert duplicate key in the same map

The right sequence should be:

Benchmark Time CPU Iterations

ran SetUp
outside the while loop
inside the while loop
ran teardown
ran SetUp
outside the while loop
inside the while loop
ran teardown
...

[pleroy]

This looks like a terrible idea to me. In real life the setup can be fairly complicated and costly. Running it in the loop would defeat the purpose of the loop, which is to get stable numbers by adjusting the number of iterations.

[tahsinH]

perhaps on my part , this was jumping the gun. Stead of suggesting a solution, I should elaborate on the issue I am facing more eloquently. I understand , the need for the framework not to call the setup code due to expensive setup code overhead. In my real case(not shown here), I actually have an expensive setup.

But, on the other side, I also want to ensure that the thing I am measuring is measured correctly. Lets change the pseudocode slightly more.

  st.PauseTiming();
  ...
  std::set<std::string> stringSet;
  for (int int i=1: i< 100000 < i++) {
    stringSet.insert(getSomePrecalculatedString());
   }
  st.ResumeTiming();    
    for (auto const & id: stringSet) {      
      m_Table.insert(id.c_str(), 1); // contrived example, all keys have same value;
    }

The first time the gets called , I am measuring insertion of 100k elements in the unordered map.
The second time this is called inside the loop, chances of collision for the hash increases , because the unordered map already has 100k elements in it. So I am actually not getting to measure the correct piece of code, which was to time the insertion of 100k elements on a fresh unordered_map.

The only correct way to measure it would be to clear the resource before the next run happens. In my case, this maps out to be the piece of code in teardown.

In some x-unit frameworks (granted they were never geared towards performance testing/measurement), I have seen that they have both a Setup() and SetupTestCase();
Where SetUp usually involves the very expensive resource creation; but SetupTestCase is run before every testCase run to still hold on to the resource , but clear it in a way that the the next run would still provide a viable solution.
Perhaps, adding another hook, where the fixture SetupTestCase is called automatically would create more maintainable, clearer code. Otherwise I am duplicating code in my next testpoint and calling the same code again.

I am not sure what the correct solution for this would be. But , I hope the issue I am facing is a bit clear in this followup post.

[EricWF]

Why don't you just clear the map yourself in the loop?

[baryluk]

Just create a method in MyFixtureTest to do re-setup that is adequate for your use case?

[tahsinH]

sorry for the super late reply. I usually end up using another github account. In anycase,
@EricWF and @baryluk that's exactly what I am doing right now...

However, me and my colleagues were trying to use this google benchmark tooling as sort of a similar tooling to googletest/cppunit (other sort of x-unit) frameworks. And for the longest time , we were looking a builtin framework support in doing the same. At the very least we should document this non-feature ; so that other's don't trip over it. Coming from various x-unit testing frameworks, it's a bit of a stumble to realize that this feature isn't there. Although I do agree , making a call (could be expensive). But surely a templated code can be written such that if someone does not call a per-testcase-setup code , the call should be omitted by the compiler/optimizer.

Please, close the issue , if it does not seem highly desirable. But please consider adding a note regarding the run-sequence behaviour, since it will stumble people used to general x-unit framework testing.

[dmah42]

i always thought it was quite obvious that the setup/teardown would be around the entire benchmark method. The use of a while loop should be a clear signal that the code inside can be called more than once.

i don't think there's any violation of expectations here: your method is being called within the setup/teardown wrapper. the only issue is that you didn't expect the while loop to loop, as far as i can tell.

[EricWF]

@tahsinH Would you be willing to add such documentation, since you've experienced the issue personally.