Add memory diagnostic

[sethrj]

This uses system tools to calculate the memory "high watermark" before and after blocks, recording the change in peak across the block (i.e. what's the most new memory allocated) and the high-water marks. The GPU memory may not be as accurate, since it's the reported "free" amount subtracted from the reported "total". Each block records its "parent" block (if applicable) so it will be possible to make a flame graph of memory usage.

simple-cms-cpu.out.json

  "memory": [
   {
    "_units": "kibi",
    "cpu_delta": 547920,
    "cpu_hwm": 732772,
    "label": "demo_loop.run"
   },
   {
    "_units": "kibi",
    "cpu_delta": 222236,
    "cpu_hwm": 407088,
    "gpu_delta": 2048,
    "gpu_usage": 319488,
    "label": "demo_loop.load_core_params",
    "parent_index": 0
   },
   {
    "_units": "kibi",
    "cpu_delta": 205968,
    "cpu_hwm": 390820,
    "gpu_usage": 317440,
    "label": "RootImporter.open",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_delta": 13880,
    "cpu_hwm": 404700,
    "gpu_usage": 317440,
    "label": "RootImporter.read",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_hwm": 404700,
    "gpu_usage": 319488,
    "label": "MaterialParams.construct",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_hwm": 404700,
    "gpu_usage": 319488,
    "label": "GeoMaterialParams.construct",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_hwm": 404700,
    "gpu_usage": 319488,
    "label": "ParticleParams.construct",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_hwm": 404700,
    "gpu_usage": 319488,
    "label": "CutoffParams.construct",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_delta": 444,
    "cpu_hwm": 407088,
    "gpu_usage": 319488,
    "label": "PhysicsParams.construct",
    "parent_index": 1
   },
   {
    "_units": "kibi",
    "cpu_hwm": 406644,
    "gpu_usage": 319488,
    "label": "SeltzerBergerModel.construct",
    "parent_index": 8
   },
   {
    "_units": "kibi",
    "cpu_hwm": 407088,
    "gpu_usage": 319488,
    "label": "CoreParams.construct",
    "parent_index": 0
   }
  ]

In the "simple GPU demo loop" above, initializing ROOT takes over half of the total CPU usage. The CPU reporting statistics are more granular on macOS:

macOS CPU demo lookp

"memory": [
 {
  "_units": "kibi",
  "cpu_delta": 258208,
  "cpu_hwm": 289904,
  "label": "demo_loop.run"
 },
 {
  "_units": "kibi",
  "cpu_delta": 253872,
  "cpu_hwm": 286176,
  "label": "demo_loop.load_core_params",
  "parent_index": 0
 },
 {
  "_units": "kibi",
  "cpu_delta": 234624,
  "cpu_hwm": 266944,
  "label": "RootImporter.open",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_delta": 15744,
  "cpu_hwm": 282704,
  "label": "RootImporter.read",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_delta": 240,
  "cpu_hwm": 284400,
  "label": "MaterialParams.construct",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_hwm": 284400,
  "label": "GeoMaterialParams.construct",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_delta": 48,
  "cpu_hwm": 284512,
  "label": "ParticleParams.construct",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_delta": 32,
  "cpu_hwm": 284624,
  "label": "CutoffParams.construct",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_delta": 864,
  "cpu_hwm": 285984,
  "label": "PhysicsParams.construct",
  "parent_index": 1
 },
 {
  "_units": "kibi",
  "cpu_delta": 112,
  "cpu_hwm": 285360,
  "label": "SeltzerBergerModel.construct",
  "parent_index": 8
 },
 {
  "_units": "kibi",
  "cpu_delta": 144,
  "cpu_hwm": 286336,
  "label": "CoreParams.construct",
  "parent_index": 0
 }
]

[amandalund]

Nice @sethrj! Looks like the HIP build needs to be fixed, but otherwise looks good to me 🐰🥚