Tensorflow serving performance and optimization tips

[viksit]

Would someone post some data on their performance numbers for their Tensorflow serving systems in production? I'm curious about some latency numbers like tp99/90/50, QPS numbers, response/request sizes, and some comparisons of numbers within data centers vs the open web?

Also, what are some best practices in squeezing out performance? For instance, streaming/batching?

[kirilg]

We did some performance benchmarking when we first released the project, mentioned in this blog post. Specifically "TensorFlow Serving introduces minimal overhead. In our benchmarks we recoded ~100,000 queries per second (QPS) per core on a 16 vCPU Intel Xeon E5 2.6 GHz machine, excluding gRPC and the TensorFlow inference processing time." We didn't collect latency numbers, but in this test it should be very low. Note though that it's only covering the actual Tensorflow Serving overhead, and not model evaluation time which may be very high depending on the model.

It would be interesting to hear from other users that have tried deploying and/or benchmarking it in a production environment.

Regarding optimization tips, the default configuration should be good for most cases, adding little overhead. Model evaluation will almost always be the slowest part, and for that the main thing to try is batching which you've mentioned. You can see this readme for specific things to try.

[viksit]

@kirilg thanks for the information.

I'm trying to adopt SharedBatchScheduler for my needs - are there any examples of usage anywhere? The test (Basic) seems to create queues, but I'm not sure if thats to test functionality, or thats whats expected by a user using that API?

[sukritiramesh]

Hi @viksit, it looks like #96 addresses the questions here. Any further open issues on this thread to resolve? Thanks.