Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmark cases for MobileBERT #6095

Merged
merged 1 commit into from
Jun 9, 2021

Conversation

antiagainst
Copy link
Contributor

This commit is meant to be an example for showing how to register
benchmark cases for a new model that cannot be folded into the
common list of models. In addition to the changes in CMake, one
would also need to generate the input model and place it in a
cloud storage.

@antiagainst antiagainst added the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label Jun 3, 2021
@google-cla google-cla bot added the cla: yes label Jun 3, 2021
@iree-github-actions-bot
Copy link
Contributor

Abbreviated Benchmark Summary

@ commit 0f153b0ddc6420c96971c8ad7a496e95f8b2dee2 (vs. base ff38a688914f2fdde7c271227a6361bd2bb643d0)

Regressed Benchmarks 🚩

Benchmark Name Average Latency (ms) Median Latency (ms) Latency Standard Deviation (ms)
MobileNetV3Small [fp32,imagenet] (TensorFlow) 3-thread,big-core,full-inference with IREE-Dylib @ Pixel-4 (CPU-ARMv8.2-A) 134 (vs. 122, 9.84%↑) 132 15
MobileNetV2 [fp32,imagenet] (TensorFlow) kernel-execution with IREE-Vulkan @ SM-G980F (GPU-Mali-G77) 18 (vs. 17, 5.88%↑) 18 0

Improved Benchmarks 🎉

Benchmark Name Average Latency (ms) Median Latency (ms) Latency Standard Deviation (ms)
MobileNetV2 [fp32,imagenet] (TensorFlow) 1-thread,little-core,full-inference with IREE-Dylib @ SM-G980F (CPU-ARMv8.2-A) 1130 (vs. 1255, 9.96%↓) 1092 70
MobileNetV2 [fp32,imagenet] (TensorFlow) 3-thread,little-core,full-inference with IREE-Dylib @ SM-G980F (CPU-ARMv8.2-A) 787 (vs. 870, 9.54%↓) 757 54

Similar Benchmarks

Benchmark Name Average Latency (ms) Median Latency (ms) Latency Standard Deviation (ms)
MobileBertSquad [fp16] (TensorFlow) kernel-execution with IREE-Vulkan @ SM-G980F (GPU-Mali-G77) 157 156 1
MobileBertSquad [fp32] (TensorFlow) 3-thread,little-core,full-inference with IREE-Dylib @ SM-G980F (CPU-ARMv8.2-A) 2091 2132 91
MobileBertSquad [fp32] (TensorFlow) 1-thread,little-core,full-inference with IREE-Dylib @ SM-G980F (CPU-ARMv8.2-A) 5419 5453 497

[Top 3 out of 15 benchmark results showed]

For more information:

This commit is meant to be an example for showing how to register
benchmark cases for a new model. In addition to the changes in
CMake, one would also need to generate the input model and place
it in a cloud storage.
@antiagainst antiagainst removed the (deprecated) buildkite:benchmark-android Deprecated. Please use benchmarks:android-* label Jun 9, 2021
@antiagainst antiagainst enabled auto-merge (squash) June 9, 2021 11:50
@antiagainst antiagainst merged commit c6b2e49 into iree-org:main Jun 9, 2021
@NatashaKnk NatashaKnk mentioned this pull request Jun 9, 2021
@antiagainst antiagainst deleted the add-mobilebert-benchmark branch October 2, 2021 18:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants