Heap Profiler

[joe-maley]

This introduces a new C API:

TILEDB_EXPORT int32_t tiledb_heap_profiler_enable(
    const char* file_name_prefix,
    uint64_t dump_interval_ms,
    uint64_t dump_interval_bytes,
    uint64_t dump_threshold_bytes);

Executing the API does the following:

• The singleton HeapProfiler is enabled, which may only be done once in the
  lifetime of the process.
• On an uncaught std::bad_alloc (anywhere in the process) we will perform
  a stat dump and kill the process.
• If dump_interval_ms is non-zero, a thread is started which will dump
  the heap stats periodically.
• If dump_interval_bytes is non-zero, the next allocation that increases
  the lifetime total allocation byte size by this amount will perform a dump.
• If dump_threshold_bytes non-zero, labeled allocations with a total
  alloc size less than this amount will be ignored. This is used to reduce
  verbosity of the dump for trivial allocations.

This is dead code at the moment. The idea is that all dynamic memory allocations
in our source code will be accompanied by a call HeapProfiler::record_alloc().
When called, this save stats. Additionally, failed dynamic memory allocations
may invoke HeapProfiler::dump_and_terminate(). Similarly for dellocations.

The HeapProfiler is responsible for:

• Setting up the uncaught std::bad_alloc hook.
• Setting up a periodic dump thread.
• Reserving memory at initialization. This is an opaque block of dynamic memory
  that is freed when entering dump_and_terminate(). The intention is to release
  memory after a fail allocation to provide enough memory for the dump to be
  successful.
• Keeps a "labels cache" which is basically an object pool of strings to prevent
  duplication of string labels. For example, a caller may do:

tiledb::common::heap_profiler.record_alloc(p1, size1, "reader.cc::1923");
tiledb::common::heap_profiler.record_alloc(p2, size2, "reader.cc::1923");
tiledb::common::heap_profiler.record_alloc(p3, size3, "reader.cc::1923");

• Where "reader.cc::1923" could be generated by compilers macros, e.g. std::string(__FILE__ + "::" + __LINE__).
  In this scenario, we don't want to haev 3x "reader.cc::1923" strings allocated.
  We share them in the HeapProfiler.

The next step is to refactor our code base to find-and-replace all dynamic
memory APIs with a macro/typedef that will record alloc/dealloc stats as they
are used.

[eddelbuettel]

Looks good to me from a (arguably casual) glance.

[Shelnutt2]

The history file needs updating

[joe-maley]

The history file needs updating

Thanks -- I'll get it on the next related patch.

[ihnorton]

💯 Minor comment, otherwise LGTM.

[ihnorton]

I think this is good and useful as-is, but down the line we might want to do a simple bump allocator and placement new into this reserved space for the dump path data structures (we're still at the mercy of the memory manager if there's something else continuously eating memory on the system).

[joe-maley]

Yeah -- agreed. My initial approach was to allocate everything up-front but I decided that the label map would be valuable and that trying to manage+tune for a bunch of strings and hashes would be more trouble than it's immediately worth. Additionally, we'll still be at the mercy of the system if anything else in the dump path needs heap memory (for example, opening + writing to the output file). Finally, we don't have robust hooks for when we actually run out of memory. For example: if a malloc fails elsewhere in the process, we won't catch it.

For all those reasons, I see the preferred profiling as a periodic/interval dump. If we're lucky, we'll also get a dump at the time of going out of memory.