Skip to content

System Assistant Overlay

Puzzak edited this page Nov 20, 2026 · 1 revision

System Assistant Overlay & Screen Inspection

PAIOS integrates directly with Android's Digital Assistant subsystem so you can summon Gemini Nano over any app without launching the full Flutter UI.


Android Voice Interaction Architecture

The assistant pipeline is implemented natively in Kotlin for immediate startup:

  • PaiosVoiceInteractionService (android.service.voice.VoiceInteractionService): Registered as a system Digital Assistant service (power button long-press or corner swipe). On onReady(), it registers the headless M3x Mesh handler and preloads localized overlay strings asynchronously.
  • PaiosVoiceInteractionSessionService / PaiosVoiceInteractionSession: Receives AssistStructure view hierarchies and screen captures (onHandleScreenshot) from Android's VoiceInteractionManagerService, filtering out generic system chrome (com.android.systemui) and display cutouts/system bars before handing off to MeshOverlayActivity.
  • PaiosRecognitionService: Companion android.speech.RecognitionService required by AOSP VoiceInteractionServiceInfo validation (res/xml/interaction_service.xml) so Android recognizes PAIOS as a valid Digital Assistant target.
  • MeshOverlayActivity: Separate-task (page.puzzak.paios.overlay, launchMode="singleTask", excluded from recents) native Kotlin Activity that bypasses Flutter engine initialization.

Dual Operating Modes (MeshOverlayActivity)

MeshOverlayActivity operates in two distinct modes depending on how it is invoked:

Mode A — Headless / One-Shot RPC Mode

  • Triggered when an external M3x Mesh client calls query_local_ai via M3xMeshContentProvider.headlessHandler.
  • Appears momentarily as a translucent, non-focusable bottom status pill to satisfy Android AICore's foreground window requirement (BACKGROUND_USE_BLOCKED prevention).
  • Executes on-device Gemini Nano text or vision inference, returns the JSON-RPC payload to the calling app via MeshOverlayActivity.pendingCallback, and finishes automatically.

Mode B — Interactive Material 3 Expressive Assistant Sheet

  • Activated via Intent.ACTION_ASSIST, Intent.ACTION_VOICE_COMMAND, or M3x Mesh open_assistant_bubble (interactive_bubble = true).
  • Renders a dynamic Material You (Monet) bottom sheet with:
    • Drag-to-expand (full-screen fill) and drag-to-close physics, outside-tap dismissal, and zero-flicker IME inset handling.
    • Native Markdown chat bubbles, collapsible Thought process reasoning blocks, and live tool-status cards.
    • Active persona system prompt injection (MeshOverlayActivity.resolveActivePersonaContent) and configured temperature/max-token budgets.
    • Direct Open in Full App handoff button that pops to the root navigator and opens ChatPage directly.

Privacy-First On-Demand Screen Inspection (inspect_screen)

Earlier versions passively attached screenshots to every turn's system prompt, causing unrelated general questions to misread whatever app was behind the sheet and wasting vision token budgets.

Since v1.8.5 (hardened through v1.8.7), PAIOS uses lazy on-demand screen inspection:

  1. Silent Buffering (SCREEN_CONTEXT_TTL_TURNS = 3): When summoned over an external app (and skipped automatically when opened directly inside PAIOS), MeshOverlayActivity holds the captured screenshot bitmap (trimmed of status bar, navigation bar, and display cutout insets) and filtered AssistStructure / live foreground Mesh context in pendingScreen* buffers for a 3-turn TTL.
  2. Tool Exposure Only: The screen capture is not injected into the prompt automatically. Instead, the inspect_screen (app: "paios") tool is exposed to Gemini Nano.
  3. Active Inspection: Only when the user asks about what is on their screen (e.g., "Summarize this page", "What does this error mean?") does Gemini Nano invoke inspect_screen, triggering VLM + AssistStructure analysis in InternalToolsHelper.kt / PaiosModelHelper.kt.
  4. TTL Preservation & Auto-Drop:
    • Aborted or unparseable tool-call attempts preserve pendingScreenTurnsRemaining (v1.8.7).
    • Once 3 turns pass without invoking inspect_screen (or on session reset/image attachment), the buffered screenshot bitmap is recycled and dropped so stale screen context never leaks into follow-up questions.
  5. OS-Level Opt-Out: Users can disable screen capture and UI text entirely in Android Settings -> Apps -> Default apps -> Digital assistant app -> gear icon by toggling off Use screenshot and Use text from screen.

Camera, Gallery & Visual Memory

  • Gallery & Camera (Take Photo): Attach photos from Gallery or launch the system camera (ACTION_IMAGE_CAPTURE) via Android FileProvider (${applicationId}.fileprovider) without requesting the runtime CAMERA permission.
  • Vision Isolation: Attaching a camera photo or gallery image clears any passive background screenshot and prior visual turn memory so 100% of Gemini Nano's vision resolution budget is dedicated to your photo.
  • EXIF & Downsampling: Images undergo automatic EXIF orientation correction, optional 768px fast NPU downsampling, native multi-image ImagePart encoding (under ML Kit 1.0.0-beta5), and explicit .recycle() cleanup after encoding/inference.
  • Post-Turn Visual Memory: After the first vision reply, PAIOS summarizes the image in the background so follow-up text turns retain visual context without re-sending raw bitmaps on every turn.

Auto-Archiving & Foreground Synchronization

Assistant sessions are never lost when you dismiss the overlay:

  1. When MeshOverlayActivity closes (closeInteractiveSession() / onStop()), the session transcript is queued into SharedPreferences under flutter.pending_assistant_chats (skipped when openInFullApp() sets skipPendingArchiveOnDismiss = true to avoid duplicate archiving).
  2. AIEngine drains pending_assistant_chats both on cold boot and automatically on warm foreground return via AppLifecycleState.resumed (v1.8.9), importing overlay conversations into Hive (ChatRepository) and routing Open in Full App launches straight to ChatPage via appNavigatorKey.

Related Pages: Architecture & Model Variants · M3x Mesh Ecosystem & Extensions · Localization & Prompts

Clone this wiki locally