Repository navigation
System Assistant Overlay
PAIOS integrates directly with Android's Digital Assistant subsystem so you can summon Gemini Nano over any app without launching the full Flutter UI.
The assistant pipeline is implemented natively in Kotlin for immediate startup:
-
PaiosVoiceInteractionService(android.service.voice.VoiceInteractionService): Registered as a system Digital Assistant service (power button long-press or corner swipe). OnonReady(), it registers the headless M3x Mesh handler and preloads localized overlay strings asynchronously. -
PaiosVoiceInteractionSessionService/PaiosVoiceInteractionSession: ReceivesAssistStructureview hierarchies and screen captures (onHandleScreenshot) from Android'sVoiceInteractionManagerService, filtering out generic system chrome (com.android.systemui) and display cutouts/system bars before handing off toMeshOverlayActivity. -
PaiosRecognitionService: Companionandroid.speech.RecognitionServicerequired by AOSPVoiceInteractionServiceInfovalidation (res/xml/interaction_service.xml) so Android recognizes PAIOS as a valid Digital Assistant target. -
MeshOverlayActivity: Separate-task (page.puzzak.paios.overlay,launchMode="singleTask", excluded from recents) native Kotlin Activity that bypasses Flutter engine initialization.
MeshOverlayActivity operates in two distinct modes depending on how it is invoked:
- Triggered when an external M3x Mesh client calls
query_local_aiviaM3xMeshContentProvider.headlessHandler. - Appears momentarily as a translucent, non-focusable bottom status pill to satisfy Android AICore's foreground window requirement (
BACKGROUND_USE_BLOCKEDprevention). - Executes on-device Gemini Nano text or vision inference, returns the JSON-RPC payload to the calling app via
MeshOverlayActivity.pendingCallback, and finishes automatically.
- Activated via
Intent.ACTION_ASSIST,Intent.ACTION_VOICE_COMMAND, or M3x Meshopen_assistant_bubble(interactive_bubble = true). - Renders a dynamic Material You (Monet) bottom sheet with:
- Drag-to-expand (full-screen fill) and drag-to-close physics, outside-tap dismissal, and zero-flicker IME inset handling.
- Native Markdown chat bubbles, collapsible
Thought processreasoning blocks, and live tool-status cards. - Active persona system prompt injection (
MeshOverlayActivity.resolveActivePersonaContent) and configured temperature/max-token budgets. - Direct Open in Full App handoff button that pops to the root navigator and opens
ChatPagedirectly.
Earlier versions passively attached screenshots to every turn's system prompt, causing unrelated general questions to misread whatever app was behind the sheet and wasting vision token budgets.
Since v1.8.5 (hardened through v1.8.7), PAIOS uses lazy on-demand screen inspection:
-
Silent Buffering (
SCREEN_CONTEXT_TTL_TURNS = 3): When summoned over an external app (and skipped automatically when opened directly inside PAIOS),MeshOverlayActivityholds the captured screenshot bitmap (trimmed of status bar, navigation bar, and display cutout insets) and filteredAssistStructure/ live foreground Mesh context inpendingScreen*buffers for a 3-turn TTL. -
Tool Exposure Only: The screen capture is not injected into the prompt automatically. Instead, the
inspect_screen(app: "paios") tool is exposed to Gemini Nano. -
Active Inspection: Only when the user asks about what is on their screen (e.g., "Summarize this page", "What does this error mean?") does Gemini Nano invoke
inspect_screen, triggering VLM +AssistStructureanalysis inInternalToolsHelper.kt/PaiosModelHelper.kt. -
TTL Preservation & Auto-Drop:
- Aborted or unparseable tool-call attempts preserve
pendingScreenTurnsRemaining(v1.8.7). - Once 3 turns pass without invoking
inspect_screen(or on session reset/image attachment), the buffered screenshot bitmap is recycled and dropped so stale screen context never leaks into follow-up questions.
- Aborted or unparseable tool-call attempts preserve
- OS-Level Opt-Out: Users can disable screen capture and UI text entirely in Android Settings -> Apps -> Default apps -> Digital assistant app -> gear icon by toggling off Use screenshot and Use text from screen.
-
Gallery & Camera (
Take Photo): Attach photos from Gallery or launch the system camera (ACTION_IMAGE_CAPTURE) via AndroidFileProvider(${applicationId}.fileprovider) without requesting the runtimeCAMERApermission. - Vision Isolation: Attaching a camera photo or gallery image clears any passive background screenshot and prior visual turn memory so 100% of Gemini Nano's vision resolution budget is dedicated to your photo.
-
EXIF & Downsampling: Images undergo automatic EXIF orientation correction, optional
768pxfast NPU downsampling, native multi-imageImagePartencoding (under ML Kit1.0.0-beta5), and explicit.recycle()cleanup after encoding/inference. - Post-Turn Visual Memory: After the first vision reply, PAIOS summarizes the image in the background so follow-up text turns retain visual context without re-sending raw bitmaps on every turn.
Assistant sessions are never lost when you dismiss the overlay:
- When
MeshOverlayActivitycloses (closeInteractiveSession()/onStop()), the session transcript is queued intoSharedPreferencesunderflutter.pending_assistant_chats(skipped whenopenInFullApp()setsskipPendingArchiveOnDismiss = trueto avoid duplicate archiving). -
AIEnginedrainspending_assistant_chatsboth on cold boot and automatically on warm foreground return viaAppLifecycleState.resumed(v1.8.9), importing overlay conversations into Hive (ChatRepository) and routing Open in Full App launches straight toChatPageviaappNavigatorKey.
Related Pages: Architecture & Model Variants · M3x Mesh Ecosystem & Extensions · Localization & Prompts
PAIOS (page.puzzak.paios) — Synced with v1.8.9+26112001 | GitHub Repository · Nightly Google Group · Play Testing Opt-In · Report an Issue