Benchmarking - Add tensor_parallel_size arg for multi-gpu benchmarking

[varun-sundar-rabindranath]

SUMMARY:

• Add tensor_parallel_size arg for multi-gpu benchmarking.
• Update the benchmark config files to use all available gpus by default.
• Other minor changes :
  • Fix script args outer-product enumeration.
  • Limit dataset sample-space when using sharegpt dataset.
  • Remove redundant benchmarks.
  • Standardize benchmarking result JSON.

TEST PLAN:
Run manual benchmark jobs
multi-gpu benchmark : https://github.com/neuralmagic/nm-vllm/actions/runs/8086009234/job/22094787943
single-gpu benchmark : https://github.com/neuralmagic/nm-vllm/actions/runs/8086016169/job/22094812742
(Then benchmarks didn't run to completion as huggingface went down mid-way. but the artifacts seem reasonable for what it did run)

[mgoin]

Why the underscore? I don't see it for other arguments

[varun-sundar-rabindranath]

The benchmark_throughput.py script is not meant to use this argument directly.
it is part of a mutually exclusive arg group,

    tp_group = parser.add_mutually_exclusive_group(required=True)
    tp_group.add_argument("--tensor-parallel-size_", type=int, default=None)
    tp_group.add_argument("--use-all-available-gpus_", action="store_true")

and the script derives the correct tensor_parallel_size using the utility,

def get_tensor_parallel_size(args: argparse.Namespace) -> int:
    tensor_parallel_size = num_available_gpus() \
        if args.use_all_available_gpus_ else args.tensor_parallel_size_
    assert tensor_parallel_size > 0 and \
           tensor_parallel_size <= num_available_gpus()
    return tensor_parallel_size

[varun-sundar-rabindranath]

This actually translates to use_all_available_gpus = True in python.. If we want to set it to false, we remove this key.

This is really a pain - I am considering moving to yml or some other format. on top this, JSON doesn't even let you add comments :(

[varun-sundar-rabindranath]

removed redundant benchmark.

[varun-sundar-rabindranath]

same as above - an empty use-all-available-gpus_ translates to --use-all-available-gpus_ arg

[mgoin]

looks reasonable and nice job with the multi-gpu usage

[andy-neuma]

cool