Skip to content

Architecture and Model Variants

Puzzak edited this page Nov 20, 2026 · 1 revision

Architecture & Model Variants

PAIOS (page.puzzak.paios) pairs a Flutter / Dart main application (AIEngine) with a pure-Kotlin native Android Digital Assistant (MeshOverlayActivity) so both full-screen multi-chat sessions and instant system-level assistant overlays share the same on-device Gemini Nano engine and user state.


Dual-Layer Runtime Architecture

1. Flutter / Dart Layer (AIEngine & Repositories)

  • Central State Machine (lib/engine.dart): AIEngine (ChangeNotifier) orchestrates initialization, model variant selection, tool-calling turns, streaming updates, auto-continuation, and AppLifecycleState.resumed synchronization with dismissed overlay chats.
  • Hive Storage ('paios_storage'): MigrationService (lib/storage/migration.dart) migrates legacy SharedPreferences payloads into a unified 'paios_storage' Hive box backing dedicated repositories:
    • ChatRepository (conversations, pinned/archived state, per-turn "author" and epoch-ms "time", thoughtProcess)
    • PromptRepository (bundled & custom Markdown personas, refreshed from assets for the default persona)
    • ResourceRepository (in-app resources & sanitized links)
    • IntroSlideRepository (onboarding SWR slides)
    • AppConfig (model channel/preference, token/temperature settings, extension toggles)
  • Dart <-> Kotlin Bridge (SharedPreferences / FlutterSharedPreferences): Bridges lightweight cross-runtime state between Flutter and native Kotlin services (such as pending_assistant_chats queued by MeshOverlayActivity and flutter.mesh_extension_<moduleId>_enabled flags read by InternalToolsHelper).

2. Pure-Kotlin Native Assistant Overlay (MeshOverlayActivity)

  • MeshOverlayActivity + PaiosVoiceInteractionService: Runs natively in Kotlin inside a dedicated page.puzzak.paios.overlay task (launchMode="singleTask", excluded from recents) without spinning up a Flutter engine, enabling instant overlay launches over any foreground app.
  • FlutterLocalAiPlugin.kt & PaiosModelHelper.kt: Shared Kotlin inference helpers powering both the Flutter MethodChannel/EventChannel bridge and the native overlay.

Gemini Nano Variants (v3 vs v4) & Model Manager

PAIOS enumerates all 4 (releaseStage, preference) combinations natively via PaiosModelHelper.probeAllComboStatuses (ComboStatuses -> "modelVariants"):

Channel Internal Model IDs Availability Capabilities
Stable gemini-nano-v3 (stable/full, stable/fast) Production Everyday chat, System Instructions, Function Calling, Multimodality
Preview / Beta gemini-nano-v4-full, gemini-nano-v4-fast Preview / Beta Expanded context (up to 8,192 tokens), native Thinking Mode (thoughtProcess), native <ctrl42>call:... & FunctionCall tool calling
  • Unified Model Selector & Decoupled Downloader: showModelSelectorDialog (from the "nano-v3"/"nano-v4" header chip or Settings -> Model) lets users switch channels and download variants in parallel (FlutterLocalAiPlugin.variantDownloadJobs ConcurrentHashMap) independently of the active client, displaying real DownloadStarted.bytesToDownload totals, B/s–GB/s speed, and remaining bytes.
  • Active-Inference Lock: FlutterLocalAiPlugin.activeGenerationCount (AtomicInteger) and AIEngine.isLoading block concurrent model switches while inference is running and defer closing the previous GenerativeModel until the active stream completes.

ML Kit GenAI 1.0.0-beta5 Integration

Upgraded in v1.8.9 (com.google.mlkit:genai-prompt:1.0.0-beta5), PAIOS leverages the latest native Prompt API capabilities:

  • model.warmup(): Pre-loads NPU/TPU weights when idle (triggerWarmupIfIdle and interactive overlay startup) to cut first-token latency.
  • PromptPrefix Caching: Attaches PromptPrefix(cacheSystemInstruction = true, cacheTools = true) when caching is supported, reusing KV-cache state across turns.
  • Native Multi-Image Content (ImagePart): Sends multiple attached images as native ImagePart entries in Content rather than stitching bitmaps vertically onto a single composite canvas.
  • Token Telemetry (countTokens / getTokenLimit): Queries exact token counts and model token limits natively.
  • Sampling Controls: Supports topK and seed alongside temperature and maxOutputTokens.
  • Non-Deprecated Feature Discovery: Uses model.getAvailableFeatures() instead of deprecated capability probes.
  • Streaming FunctionCall Capture: Captures native FunctionCall objects (onNewFunctionCall / candidate.functionCalls) alongside <ctrl42> fallback parsing.
  • ML Kit beta5 zzaxl.zzr getBaseModelName() Quirk: Inside genai-prompt:1.0.0-beta5, zzaxl.zzr maps PREF_FULL feature IDs {645, 646, 651, 653} inside getBaseModelName() to "nano-v4-fast" / "gemini-nano-v4-fast". PAIOS separates internalModelName from display labels so variant badges and selectors remain accurate even when getBaseModelName() reports nano-v4-fast for PREF_FULL.

On-Device Reliability & Continuation Pipeline

Because Gemini Nano enforces tight hardware limits (such as a 4,096-token window on v1–v3, 256-token/step output bounds, and a 25-second NPU execution watchdog), PAIOS implements a shared Dart + Kotlin hardening pipeline:

  1. Token-Budgeted Sliding Window: Prompt (defaultMaxHistoryChars = 2800, dynamic historyBudget & schemaBudget) prunes older history before hitting REQUEST_TOO_LARGE (12), with an automatic compact retry (600 history / 400 system chars) if overflow still occurs.
  2. Tail-Bounded [CONTINUATION] Prompts: Multi-round continuations never re-stuff full persona/history into SystemInstruction; instead Prompt.generateContinuation and PaiosModelHelper.buildOverlayContinuationPrompt emit a compact [CONTINUATION] directive (<= 240-char persona, <= 220-char question, ~450-char trailing slice via boundTailAtBoundary).
    • Caps: maxProseContinuations = 3 (soft/hard stops at 1600/2400 chars), maxContinuations = 10 for open code blocks, and MAX_OVERLAY_CONTINUATIONS = 6 in the overlay.
  3. Tool-Call Repair & Fallback:
    • Truncated JSON Repair: PaiosModelHelper.extractToolCallJson + sanitizeJsonLikeString perform string/brace-aware closure repair (recovering truncated drafts like 109-char inspect_screen drafts missing "}}), normalize "name"/"method" and "parameters"/"args"/"arguments", and cap tool drafts at MAX_TOOL_JSON_CONTINUATIONS = 2.
    • Nano v4 <ctrl42> & Thought-Trapped Tool Promotion: Parses <ctrl\d+>call:tool{...}<ctrl\d+> tokens (and app:tool prefixes), scrubs raw control tags from thought bubbles (stripControlToolTags), and promotes tool calls trapped in thought when text is empty.
    • Tool-Free Fallback: Invalid or unparseable Turn-1 tool JSON triggers an automatic tool-free fallback (includeTools: false / runToolFreeFallbackAttempt) instead of leaking raw JSON into chat history.
  4. Deduplication & Cutoff Rescue:
    • Overlap & Stutter Guard: 240-char suffix-prefix overlap stitching (stitchContinuation / stitchOverlayContinuation), ellipsis seam stripping (..., …), 28-char mid-word backoff (splitTrailingPartialWord), and >= 24-char earlier-sentence loop stripping (detectAndTrimStutterLoop / hasContinuationStutter).
    • Mid-Clause STOP Rescue & Final Trimming: Detects dangling prepositions, determiners, contractions ("I can'"), transitive -ing participles ("and ensuring"), and negations ("and do not") for a bounded 1-shot STOP rescue (maxStopCutoffContinuations = 1, maxNullReasonContinuations = 3), and runs trimTrailingIncompleteSentence at all final commit paths (including unpunctuated multi-sentence cutoff tails).
  5. Lifecycle & Performance Hardening (v1.8.9):
    • AIEngine.cancelGeneration() bumps _streamGenerationId++ and clears turn flags/buffers so stale callbacks become no-ops.
    • FlutterLocalAiPlugin.dispose() cancels coroutineScope and variantDownloadJobs; FileAccessPlugin offloads SAF DocumentFile I/O to Dispatchers.IO; bitmaps are explicitly .recycle()d after inference; and v1.8.9 removed the per-token 500ms streaming delay, per-turn syncRemoteDocs() HTTP calls, and background ChatsPage full-history JSON re-parsing.

Related Pages: Supported Devices & AICore Error Taxonomy · System Assistant Overlay & Screen Inspection · M3x Mesh Ecosystem & Extensions

Clone this wiki locally