Benchmarking script

[robmarkcole]

I will happily create a benchmarking script, to evaluate speed of deepstack on different platforms. It would do a batch job of say 30 detections and present the runtime. I don’t want to reinvent the wheel, perhaps you have a script already?

[johnolafenwa]

We don't have a standard script yet for that. Please create it and make a PR to include it in benchmark directory.

[robmarkcole]

OK this will be a script which just posts every photo in a directory for processing. Wont include the photos, although we could have a standard set of photos in a seperate repo perhaps

[robmarkcole]

Created a notebook here (will breakout into script) and the results are in, with tflite being lightning fast, processing 15 images (obj detection):

Platform	Speed (sec)	Predictions
Mac Pro with deepstack (add version tag)	51.9	91
RPi4 with tflite-server	9.33	159

ps which is the 'benchmark directory'?

[johnolafenwa]

Interesting. Thanks for these @robmarkcole , i will push a nightly tag pending the setup of the cicd pipeline so you can benchmark against the new detection pipeline.

BTW, as i have not setup the benchmark directory, can you make a PR for these to include the benchmark scripts and we can review and make neccessary updates.

I am also curios, what are the specs for the MacBook Pro?

[robmarkcole]

Hardware Overview:

Model Name: MacBook Pro
Model Identifier: MacBookPro14,1
Processor Name: Dual-Core Intel Core i5
Processor Speed: 2.3 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache (per Core): 256 KB
L3 Cache: 4 MB
Hyper-Threading Technology: Enabled
Memory: 8 GB