Add --output-warmup-metrics flag to cpu userbenchmark scripts

[murste01]

Adds a new --output-warmup-metrics flag which adds warmup metrics to benchmark result JSON files. This allows us to analyse warmup iterations and decide how many are enough.

[murste01]

An example with and without the new flag:

Without --output-warmup-metrics:

Example command:

$ python3 run_benchmark.py cpu -t eval -m llama --precision fp32 --nwarmup 10 --niter 10 -o results
    
Running benchmark: /home/murste01/miniforge3/envs/pytorch/bin/python3 /home/murste01/git/benchmark/userbenchmark/cpu/run_config.py -m llama -d cpu -t eval --precision fp32 --metrics latencies --nwarmup 10 --niter 1
0 -o results
Running TorchBenchModelConfig(name='llama', test='eval', device='cpu', batch_size=None, extra_args=['--precision', 'fp32'], extra_env=None, output_dir=None, skip=False) ... [Done]

Output tree:

...
metrics-20250331150609.json
results/                                                                                                                                                                                                              `-- llama-eval 
    `-- metrics-515.json
... 

metrics-20250331150609.json contents:

$ cat metrics-20250331150609.json
{
    "name": "cpu",
    "environ": {
        "pytorch_git_version": "2236df1770800ffea5697b11b0bb0d910b2e59e1"
    },
    "metrics": {
        "llama-eval_latency": 114.53452100000001
    }
}

results/llama-eval/metrics-515.json contents:

$ cat results/llama-eval/metrics-515.json
{
    "name": "cpu",
    "environ": {
        "pytorch_git_version": "2236df1770800ffea5697b11b0bb0d910b2e59e1"
    },
    "metrics": {
        "latency": 114.53452100000001
    }   
}   

With --output-warmup-metrics:

Example command:

$ python3 run_benchmark.py cpu -t eval -m llama --precision fp32 --nwarmup 10 --niter 10 -o results --output-warmup-metrics
Running benchmark: /home/murste01/miniforge3/envs/pytorch/bin/python3 /home/murste01/git/benchmark/userbenchmark/cpu/run_config.py -m llama -d cpu -t eval --precision fp32 --output-warmup-metrics --metrics latencies --nwarmup 10 --niter 10 -o results
Running TorchBenchModelConfig(name='llama', test='eval', device='cpu', batch_size=None, extra_args=['--precision', 'fp32'], extra_env=None, output_dir=None, skip=False) ... [Done]

Output tree:

metrics-20250331150656.json
results/
`-- llama-eval
    `-- metrics-710.json

metrics-20250331150656.json contents:

$ cat metrics-20250331150656.json
{
    "name": "cpu",
    "environ": {
        "pytorch_git_version": "2236df1770800ffea5697b11b0bb0d910b2e59e1"
    },
    "metrics": {
        "llama-eval_warmup_latency": 109.78462999999999,
        "llama-eval_latency": 108.5176915
    }
}

results/llama-eval/metrics-710.json contents:

$ cat results/llama-eval/metrics-710.json
{
    "name": "cpu",
    "environ": {
        "pytorch_git_version": "2236df1770800ffea5697b11b0bb0d910b2e59e1"
    },
    "metrics": {
        "warmup_latency": 109.78462999999999,
        "latency": 108.5176915
    }
}

[murste01]

cc: @FindHao

Thanks in advance!

[FindHao]

I'm concerned that this PR introduces too significant a change to the core APIs, not only for this line. As an alternative, can you consider adding an option to skip the warmup phase and use the actual run results as the 'warmup' results?

[murste01]

I've decided to drop this change in favour of using --output-iter-metrics + --nwarmup 0 + --niter $N and removing the first $M runs externally.

Thanks @FindHao.