Add a CUDA memory tracker and use it in voxtral runner

[larryliu0820]

This PR adds a CUDA memory tracker and integrates it into Stats of LLM Runner. We added gpu_total_bytes, gpu_free_before_load_bytes, gpu_free_after_load_bytes, gpu_free_after_generate_bytes and gpu_peak_usage_mb information.

See logging:

PyTorchObserver {"prompt_tokens":387,"generated_tokens":68,"model_load_start_ms":1762976881583,"model_load_end_ms":1762976883487,"inference_start_ms":1762976887396,"inference_end_ms":1762976888589,"prompt_eval_end_ms":1762976887815,"first_token_ms":1762976887815,"aggregate_sampling_time_ms":17,"gpu_total_bytes":17094475776,"gpu_free_before_load_bytes":15589179392,"gpu_free_after_load_bytes":11455692800,"gpu_free_after_generate_bytes":10530848768,"gpu_peak_usage_mb":4824,"SCALING_FACTOR_UNITS_PER_SECOND":1000}

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15780

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job, 5 Unrelated Failures

As of commit 59e6a65 with merge base 1034a0f ():

CANCELLED JOB - The following job was cancelled. Please retry:

• Test Metal Backend / test-model-metal-e2e (openai, whisper-large-v3-turbo, non-quantized) / macos-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-moshi-linux / linux-job (gh) (trunk failure)
  RuntimeError: Could not load libtorchcodec. Likely causes:
• pull / unittest / linux / linux-job (gh) (trunk failure)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / macos / macos-job (gh) (trunk failure)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Gasoonjia]

Thanks for adding that! The logic LGTM, i just have some concern if we should use the word peak gpu memory

[Gasoonjia]

why do we want to delete the loading time recording?

[larryliu0820]

Moved it inside load()

[Gasoonjia]

I think we can rename it for better description. I don't think it is actually the peak gpu usage, it is the memory usage gap between start and finish running. It could be happen if the peak moment happen in the middle of execution.

Same as other places for "peak gpu memory"

[larryliu0820]

The meaning of this field is "among all the samples that I collected using log_sample() what is the peak gpu memory". I think we can add a few more sample points, but from what I observed, the memory usage has been quite stable after load() finishes.