Skip to content

MLPerf Mobile App v6.0

Choose a tag to compare

View all tags
@anhappdev anhappdev released this 09 Jun 06:00
· 13 commits to master since this release
v6.0
f6a5e4b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Description

MLPerf Mobile Inference Benchmark is an open-source benchmark suite for measuring how fast mobile devices (e.g. phones, laptops) can run AI tasks. The benchmark is supported by the MLPerf Mobile App which currently supports Android and iOS. Please see the MLPerf Mobile Inference benchmark paper for a detailed description of the benchmarks along with the motivation and guiding principles behind the benchmark suite.

Release Notes

🚀 New Features

  • New LLM (Large Language Model) benchmark. Added a generative AI workload that runs Llama on-device, reports token throughput, and supports:
    • Three model sizes: 1B, 3B, and 8B parameters.
    • Two evaluation datasets: MMLU and IFEval (with dedicated JSON and language validators).
    • Configurable input-token limit and config-based pipeline thread count.
  • Benchmark Sets — benchmarks can now be grouped and run as configurable sets, with hidden-option support.
  • Performance benchmarks now run against the full dataset.

🐛 Bug Fixes

  • Fixed Loadgen null-value extraction / result parsing for token-throughput benchmarks.
  • Fixed empty (0) performance-sample datasets.
  • Improved backend matching to handle multiple matches.
  • Resources downloaded only for the selected run mode.
  • Added EGL call validation checks.

⚙️ CI / Build & Infrastructure

  • Updated Loadgen to 5.1.1.
  • Added iOS integration tests on BrowserStack (incl. Galaxy S26 / S26 Ultra).
  • Command-line binaries now built in CI.
  • Xcode 26+ (Clang 21) build compatibility.
  • Fixed Android builds on macOS; resolved Dockerfile linter warnings.

Full Changelog: v5.0.4...v6.0

Supported OSs

The MLPerf Mobile app supports Android 11 (API level 30) and above, as well as iOS 13.1 or later.

Supported SOCs

Mediatek

  • Dimensity 9000 series (9000/9000+/9200/9200+/9300/9300+/9400/9500)
  • Dimensity 8000 series (8000/8020/8050/8100/8200/8300)

Qualcomm

  • Snapdragon 8 Elite Gen 5
  • Snapdragon 8 Gen 5
  • Snapdragon 8 Elite
  • Snapdragon 8s Gen 4
  • Snapdragon 7 Gen 4
  • Default fallback for all other Snapdragon mobile platforms

Samsung

  • Exynos 2600
  • Exynos 2500
  • Exynos 2400
  • Exynos 2300
  • Exynos 2200
  • Exynos 2100

Google Pixel

  • Pixel 10/9/8/7/6 and Pixel 10/9/8/7/6 Pro (Tensor G5/G4/G3/G2/G1 SoC)

The MLPerf Mobile App will also run on a host of other devices via our default path, which uses TensorFlow Lite on Android devices.

Installation instructions

  1. Allow installation of unknown apps in Settings > Apps > Special Access.
  2. Download the MLPerf Mobile APK.
  3. Find the APK in 'Downloads' or file browser.
  4. Tap the APK file. Approve installation when prompted.
  5. Confirm 'Install'.
  6. Once installed, tap 'Open' to launch MLPerf Mobile.

Note:

In testing, we found issues with the following devices:

  • Samsung Galaxy Tab S9
  • Samsung Galaxy S23/S23 Ultra
Assets 3
Loading