Skip to content

feat: add experimental MTP setting, off by default - #7

Merged
kdroidFilter merged 1 commit into
mainfrom
feat/experimental-mtp-setting
Aug 14, 2026
Merged

feat: add experimental MTP setting, off by default#7
kdroidFilter merged 1 commit into
mainfrom
feat/experimental-mtp-setting

Conversation

@kdroidFilter

Copy link
Copy Markdown
Collaborator

Summary

  • Add a Settings toggle for LiteRT-LM Gemma 4 multi-token prediction (MTP), labelled experimental and off by default.
  • When enabled, set ExperimentalFlags.enableSpeculativeDecoding before engine creation (Android, desktop, and the Linux GPU worker).
  • Persist the flag in the settings snapshot; older installs stay off. Reloading the model picks up a change (immediately if keep-alive is on).

Test plan

  • :shared:compileKotlinJvm / :shared:compileAndroidMain
  • :shared:jvmTest (AppViewModelTest, StringResourcesTest, LinuxGpuPolicyTest)
  • Open Settings → Model and confirm Multi-token prediction (experimental) is Off
  • Turn it On, translate on GPU, confirm the engine reloads and decode is faster

Expose LiteRT-LM Gemma 4 multi-token prediction as a Settings toggle
so decode can be ~2x faster on GPU without turning it on for everyone.
@kdroidFilter
kdroidFilter merged commit ca73b9a into main Aug 14, 2026
@kdroidFilter
kdroidFilter deleted the feat/experimental-mtp-setting branch August 14, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant