[BUG] Memory manager counts allocations not part of benchmark

[Hexorcyst]

Describe the bug

When using a MemoryManager, the Start() function is called before some more "internal" objects are created. This makes it impossible to get e.g. a 100% accurate allocation count.

When the call to the MemoryManager's Start() is made, other objects are created/allocated before the test is started. Here is e.g. RunMemoryManager from src/benchmark_runner.cc:

MemoryManager::Result BenchmarkRunner::RunMemoryManager(
    IterationCount memory_iterations) {
  memory_manager->Start();
  std::unique_ptr<internal::ThreadManager> manager;
  manager.reset(new internal::ThreadManager(1));
  b.Setup();
  RunInThread(&b, memory_iterations, 0, manager.get(),
              perf_counters_measurement_ptr,
              /*profiler_manager=*/nullptr);
  manager.reset();
  b.Teardown();
  MemoryManager::Result memory_result;
  memory_manager->Stop(memory_result);
  memory_result.memory_iterations = memory_iterations;
  return memory_result;
}

As is clear from the code, after calling MemoryManager::Start(), a new internal::ThreadManager object is created on the heap, and b.Setup() is called. Then, before MemoryManager::Stop() is called, manager.reset() and b.TearDown() are called.

IMHO both the internal::ThreadManager creation and the b.Setup()/b.Teardown() should happen before, respectively after, calling Start()/Stop(). That way only the memory use inside the for (auto _ : state) { ... } loop will be tracked, which is what I assume most people would be interested in.

I.e. I would have thought RunMemoryManager would have been implemented like this (note that memory_result is also moved in front of Start(), although it currently doesn't allocate):

MemoryManager::Result BenchmarkRunner::RunMemoryManager(
    IterationCount memory_iterations) {
  std::unique_ptr<internal::ThreadManager> manager;
  manager.reset(new internal::ThreadManager(1));
  b.Setup();
  MemoryManager::Result memory_result;
  memory_manager->Start();
  RunInThread(&b, memory_iterations, 0, manager.get(),
              perf_counters_measurement_ptr,
              /*profiler_manager=*/nullptr);
  memory_manager->Stop(memory_result);
  manager.reset();
  b.Teardown();
  memory_result.memory_iterations = memory_iterations;
  return memory_result;
}

System
Which OS, compiler, and compiler version are you using:

• OS: Ubuntu 24.04 LTS
• Compiler and version: GCC 12

To reproduce
Steps to reproduce the behavior:

1. Implement MemoryManager that hooks malloc and friends
2. Reset your MemoryManager's num_allocs in Start()
3. Report your MemoryManager's num_allocs in Stop()
4. Create a non-allocating benchmark (e.g. do nothing or fill an std::array)
5. Run benchmark with --benchmark_format=json
6. Notice how "allocs_per_iter" in the JSON is not zero

[LebedevRI]

I think the design decision behind that feature is that you have to opt-in your custom allocations,
not just override global allocator. That being said, i've never used that feature.

[Hexorcyst]

To me that seems like an unnecessary restriction, which would severely limit the usability of the MemoryManager. It works totally fine with e.g. wrapped malloc and co, the code being benchmarked is completely unaware and needs no changes at all. It's just that as mentioned the count is the slightest bit off/incorrect due to the ordering of the Start()/Stop() calls.