[Documentation] How to profile the GPU memory info for a onnxruntime session?

[pineking]

Describe the documentation issue

How to profile the GPU memory info for a onnxruntime session?
e.g.

onnx_session = onnxruntime.InferenceSession(detect_model,sess_options,providers=exec_providers)
print(onnx_session.memory_consumption())

Is there a python API to show the GPU memory consumption?

Page / URL

No response

[yuslepukhin]

You can try this setting with CUDA EP and have a pretty good idea of the consumption. The memory arena may actually consume more as it caches allocations.

https://github.com/microsoft/onnxruntime/blob/c9f67717d5736d48fbe3107e528cde9d6de623b7/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h?plain=1#L321C1-L321C119

[pineking]

@yuslepukhin Thank you for the suggestion.

sess_options = onnxruntime.SessionOptions()

sess_options.enable_cpu_mem_arena = False
sess_options.enable_mem_pattern = False
sess_options.enable_mem_reuse = True

# onnxruntime >= 1.21.0, python >= 3.10
sess_options.add_session_config_entry("session.collect_node_memory_stats_to_file", "/tmp/memory.log")

and write a python-script to accumulate the value in memory-stat-file

filename = "/tmp/memory.log"
with open(filename, 'r') as file:
    lines = file.readlines()
    input_sizes = 0
    init_memory_size = 0
    allocated_memory_size = 0
    for line in lines[1:]:
        arr = line.strip().split(',')
        input_sizes += int(arr[1])
        init_memory_size += int(arr[2])
        allocated_memory_size += int(arr[3])
    print(f"Input Sizes: {input_sizes/ 1024 / 1024:.2f} MB")
    print(f"Initial Memory Size: {init_memory_size / 1024 / 1024:.2f} MB")
    print(f"Allocated Memory Size: {allocated_memory_size / 1024 / 1024:.2f} MB")

The output is:

Input Sizes: 45.08 MB
Initial Memory Size: 0.00 MB
Allocated Memory Size: 3218.26 MB

But I have two more questions about the output:

1. set the sess_options.enable_mem_pattern = False or True?
   I found this param makes different result of input_size in memory-stat-file , according this link, enable_mem_pattern should set to False, is this correct? : https://github.com/microsoft/onnxruntime/blob/v1.21.1/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h?plain=1#L277
2. The value of Allocated Memory Size is larger than the output of GPU memory by watch -n 0.5 nvidia-smi, what's the reason and how to make the value of Allocated Memory Size close to the GPU memory consumption shown in nvidia-smi

[pineking]

Also, I test with/without memory shrink option, the result is the same :

run_option = onnxruntime.RunOptions()
run_option.add_run_config_entry("memory.enable_memory_arena_shrinkage", "gpu:{}".format(gpu_idx))