Questions on inference latency/cost

[hshen14]

Hello,

I am understanding the latency rule in DAWNBench:
• Latency: Use a model that has a top-5 validation accuracy of 93% or greater. Measure the total time needed to classify all 50,000 images in the ImageNet validation set one-at-a-time, and then divide by 50,000

I am not sure how to better understand "one-at-a-time" here, so I raised some questions here and need your confirmation:

1. Does it allow the pipeline of image processing and CNN inference?
2. Does it allow preprocessed images (resize and crop done offline)?
3. Does it allow dummy data?

Thanks.

[codyaustun]

Great questions!

1. Yes, you can pipeline image processing and CNN inference
2. Yes/No. We don't explicitly require resizing and cropping to be done online, but more information never hurts. If possible, please include latency times with and without preprocessing
3. No, you must load real data. If we need to verify your results, we would need to feed in real data to validate the accuracy threshold.

[jzwang123]

Hello,

"one-at-a-time" means we cannot use batch size > 1, say 50, get the time spent on that batch, and then divide it by 50, right? @codyaustun

Thanks!

[codyaustun]

Yes, that is correct. For latency, you must use a batch size of 1.

[hshen14]

Thanks @codyaustun for your kind explanation.

[codyaustun]

No problem