Run AWS DeepLearning Benchmarks agains TS containers

[mycpuorg]

AWS runs Deep Learning benchmarks that measure serving latency and throughputs against different models. Run these benchmarks against TorchServe Containers.

Here's the AWS Deep Learning Benchmark:
https://github.com/awslabs/deeplearning-benchmark/tree/master/mms

Some important files to get quick started are:
https://github.com/awslabs/deeplearning-benchmark/blob/master/benchmark_driver.py
https://github.com/awslabs/deeplearning-benchmark/blob/master/task_config_template.cfg#L360
https://github.com/awslabs/deeplearning-benchmark/blob/master/task_config_template.cfg#L384
https://github.com/awslabs/deeplearning-benchmark/blob/master/benchmark_runner.py

[mycpuorg]

A report from the quick run of benchmarks on select models.

TorchServeBenchmarkingResults.pdf

What's Useful:

• Gauge the Inference Times, Client Round Trip Times.
• Stability of Server over sustained Client requests, internal tests show this sustains for much longer.
• Worst Case Latency expected.

What's missing are:

• Needs running on GPU instances
• More models to be covered
• Separation of Inference Time and Overhead from TorchServe

[chauhang]

@mycpuorg For Throughput Benchmarking you can use: https://github.com/pytorch/pytorch/blob/master/torch/utils/throughput_benchmark.py#L62

Please ensure to have some warmup threads prior to collecting the actual stats.

[harshbafna]

Closing this as we now have an apache-bench based benchmarking script which is used for benchmarking TorchServe against different PyTorch models.