How to run test_bench.py with ROCM?

[jinsong-mao]

Hi @xuzhao9,

I don't know how to create a dockerfile for AMD ROCM, is there any example?

Best Regards

[xuzhao9]

You could use the userbenchmark rocm-test: https://github.com/pytorch/benchmark/blob/main/userbenchmark/rocm-test/run.py

Note that test_bench.py only covers the default, PT1 eager mode runs. It does not support any of the FP16, ROCM, or PT2 Inductor.
We use theuserbenchmark framework to address this issue: https://github.com/pytorch/benchmark/blob/main/userbenchmark/ADDING_USERBENCHMARKS.md

For example, to test TorchInductor, you could use the dynamo userbenchmark: #1960

To test ROCM, you could use the rocm-test userbenchmark.

There is no current userbenchmark for FP16 right now, but we welcome you to contribute your own.

[jinsong-mao]

Hi @xuzhao9 thanks for you remind, I can reproduce the userbenchmark command for llama on cuda like this:

1, the output metric is very simple, it doesn't give me any information like inference token per second, input/output sequence length, batch size(default 32), how should I get that information?

2, I also wants to test different composition of input/output sequence length and even batch size of llama test, what should I do to achieve this?

3, the training test for llama looks failed with some issues, is it my fault or it's the current status of llama?

many thanks for your kindly feedback.
Best

[xuzhao9]

@jinsong-mao You could read the README.md file at https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo for more details. The dynamo userbenchmark is owned by the PyTorch 2.0 team, so my answers might be inaccurate, but here are my takes:

1. The input/output sequence length is hard-coded and not tunable: https://github.com/pytorch/benchmark/blob/main/torchbenchmark/models/llama/__init__.py#L21. It is possible to make it a model argument, but that would need additional work. Based on my understanding, the input are tensors, not tokens, so it cannot provide information like tokens per second.
2. Since the seqlen is hardcoded right now, you can't test different seqlens. However, different batch sizes are tunable with --batch-size (https://github.com/pytorch/pytorch/blob/main/benchmarks/dynamo/common.py#L2905)
3. Train test is not implemented right now. We welcome the community contribution to enable it.