feat: AI-powered Reply Drafter#2258
Merged
Merged
Conversation
Implement issue #2256: from the Main Menu, capture a screenshot and use an on-device LLM (Gemma 3n via MediaPipe GenAI) to suggest replies that insert into the focused text field. - Add LlmManager for MediaPipe GenAI vision inference - Add ReplyDrafterManager and ReplyDrafterHUD for the screenshot-to-reply flow - Add ModelManager and a WorkManager-based downloader for the on-device model - Add ReplyDrafterSettingsScreen to download and manage the model - Add a Pro-gated Reply Drafter action to the Main Menu 🤖 Auto-generated
isModelReady now treats a model file of the expected size as ready, so a model provisioned by any means (in-app download or manual sideload) is detected. Set EXPECTED_SIZE_BYTES to the gemma-3n-E2B-it-int4 size, which also activates the size and integrity check. 🤖 Auto-generated
AccessibilityService.takeScreenshot() yields a HARDWARE-config bitmap backed by a HardwareBuffer that ScreenshotManager releases once the callback returns; MediaPipe's addImage requires ARGB_8888. Copy the screenshot into a software ARGB_8888 bitmap synchronously in the callback, before the buffer is released. 🤖 Auto-generated
ModelDownloadWorker runs as a foreground worker that requests the dataSync service type, but the manifest never declared that type on WorkManager's SystemForegroundService. Since Android 14, the type passed to startForeground must be a subset of the type declared in the manifest, so the process crashed with an IllegalArgumentException the moment a model download started. Merge android:foregroundServiceType="dataSync" onto the WorkManager service via the app manifest. The FOREGROUND_SERVICE_DATA_SYNC permission was already present. 🤖 Auto-generated
Reply Drafter suggestions rendered in ReplyDrafterHUD, a plain overlay that switch users could not reach by scanning. Route the suggestions through the menu and scanning pipeline so they are selectable. - Add a linear menu layout (MenuLayoutType.LINEAR) alongside the radial one; suggestions render as scannable MenuItem rows via inflateAsRow - Replace ReplyDrafterHUD with ReplyDrafterMenu, opened through MenuManager; move text insertion into ReplyDrafterTextInserter - Add a Gemma Terms of Use acceptance gate before the model download - Point MODEL_URL at the hosted model to enable the in-app download 🤖 Auto-generated
LlmManager is an object with mutable LlmInference state. close() runs on the service-destroy thread while generateReplySuggestions() runs on a background thread; if the service was destroyed mid-inference, close() freed the native model while generateResponse() was still using it. Coordinate initialize/generate/close through a lock. The inference still runs outside the lock so close() never blocks the main thread; a close() arriving mid-inference defers the native release to the generate call's finally block. 🤖 Auto-generated
The radial menu page reserves vertical space for its overhead and clamps the ring to fit; the linear page did neither, so long suggestion rows could push the nav row, and its close button, off the bottom of the screen, leaving a switch user unable to dismiss the menu. buildLinearLayout now reserves the same overhead and caps the list's measured height to what is left. 🤖 Auto-generated
markModelReady() and clearModelState() wrote the model-downloaded and model-path preferences, but nothing has read them since model detection became file-based. Remove the two methods, the two preference keys, and ModelManager's now-unused PreferenceManager dependency. 🤖 Auto-generated
The model URL was hardcoded in ReplyDrafterModelConfig. Inject it at build time instead, as a required value (env var or local.properties), matching how the other secrets are handled. - build.gradle.kts: add the REPLY_DRAFTER_MODEL_URL buildConfigField via the existing configValue helper; the build fails if it is unset - ReplyDrafterModelConfig: MODEL_URL now reads BuildConfig - Drop isDownloadConfigured() and the reply_drafter_not_configured state, now unreachable with the URL always present - play-release.yml: pass REPLY_DRAFTER_MODEL_URL to the release build 🤖 Auto-generated
enaboapps
force-pushed
the
feature/reply-drafter-2256
branch
from
b21cc09 to
23ae871
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On-device AI Reply Drafter (issue #2256): from the Main Menu, capture a
screenshot of the conversation and use Gemma 3n (via MediaPipe GenAI) to
suggest short replies, entirely on-device.
(
MenuLayoutType.LINEAR) so switch users can scan and select them;selecting one inserts it into the focused text field.
the ~3 GB Gemma model; size-verified, free-space pre-checked.
REPLY_DRAFTER_MODEL_URLenv var /
replyDrafter.modelUrlinlocal.properties/ CI secret) —not committed to source.
Test plan
REPLY_DRAFTER_MODEL_URLprovidedthe model -> completes and shows "ready"
suggestions appear in the scannable linear menu; selecting one inserts it
./gradlew testDebugUnitTestpassesCloses #2256