how to get the bert inference performance on A100?

[wangweiw]

I could run the benchmark on A100, and get the results like this :

But how to convert the latency time into sentence/s?

[ttyio]

Hello @wangweiw , thanks for reporting.

we can use formula 1000ms / time_per_batch_in_ms * batch_size to calculate sentences/s, and in your case the batch_size is 1, so you only need 1000/1.70694....

[ttyio]

Close since get no response for more than 2 weeks, please reopen if you have more questions, thanks!