The wilderness companion that stays useful when the signal dies.
Gemini Live online. Gemma 4 offline. One mission ledger. One survival surface.
WildPal is the Apex Survival build: an offline-first wilderness companion for Android that blends live multimodal AI, local fallback, tactical maps, live sensors, and emergency tooling into one field-ready app.
WildPal is built to prove two things at once:
- Gemini Live online for real-time hands-free camera + voice guidance.
- Gemma 4 offline for context-aware, local-first survival chat when the network disappears.
The app has one surface and two brains. Signal state decides which brain leads, and the mission state stays in a shared ledger so the handoff feels seamless instead of brittle.
- Problem Statement 1 — Real-Time Multimodal Interaction using Gemini Live API (
gemini-3.1-flash-live-preview). - Special Prize — Best Use of Gemma 4 using an on-device, local-first agent.
| Live mode Gemini Live handles hands-free camera + voice guidance. |
Offline mode Gemma 4 keeps reasoning from local context and the mission ledger. |
| Maps Tactical maps, waypoint tools, route context, and navigation support. |
Sensors GPS, heading, altitude, pressure trend, battery, network, daylight. |
| Emergency SOS helpers, contacts, signaling tools, and quick checklists. |
Continuity Shared mission ledger for resume-after-disconnect behavior. |
|
LIVE Gemini Live handles the camera, voice, and fast survival questions. |
OFFLINE Gemma 4 takes over with local reasoning, context, and continuity. |
RESUME The shared mission ledger keeps the story intact across disconnects. |
One surface. Two brains. One ledger.
- Shows live sensor state at all times: GPS, heading, altitude, pressure trend, battery, network, and daylight remaining.
- Switches automatically between the online Gemini Live session and the offline Gemma 4 mission agent.
- Keeps a shared mission ledger so offline mode can resume from the exact state the live session left behind.
- Supports maps, waypoints, emergency actions, first-aid lookup, and survival checklists.
- Keeps everything local on-device except the Gemini Live connection and optional API keys.
- Home: tactical dashboard and quick status view.
- AI: the live chat surface, including camera-assisted Gemini Live and offline Gemma fallback.
- Maps: tactical map, waypoint tools, and route context.
- Sensors: compass, weather, altitude, and device telemetry.
- Emergency: SOS helpers, contacts, and signaling tools.
- Extras: survival references and utility views.
- Gemini Live for interactive voice and camera-assisted survival help.
- Context-aware prompts that include the current environment state.
- Best for fast questions, situation awareness, and visual identification.
- Gemma 4 local chat when the connection drops.
- Context-aware mission state from sensors and the ledger.
- Intended for planning, checklists, guidance, and continuity.
- Leaflet-powered tactical maps.
- Waypoint management and route context.
- Useful when the user needs to mark locations or navigate under pressure.
- GPS, altitude, heading, pressure trend, battery, and network state.
- Daylight remaining to help with planning.
- Sensor status is visible throughout the app.
- SOS-style actions and emergency contacts.
- Checklists and quick survival references.
- Designed for fast access under stress.
|
Top mission state, connectivity, battery, daylight |
Middle live AI, maps, sensors, and survival context |
Bottom emergency tools, approvals, and resume actions |
┌──────────── connectivity watcher ────────────┐
▼ ▼
ONLINE: Gemini Live OFFLINE: Gemma 4 E2B (llama.rn)
WS: audio 16k PCM up / 24k PCM down MISSION → SENSE → DECIDE → VALIDATE
camera JPEG frames ~1fps → ACT → CHECK → (REPLAN | finish)
sensor context via session update bad JSON → repair ≤2 → defer
│ guarded tool → APPROVAL GATE
└────────► shared mission ledger ◄─────────────┘
(AsyncStorage, append-only, survives kill → RESUME)
The shared ledger is the handoff. Live session events and Gemma agent transitions append to the same persisted timeline, so signal loss or app restarts do not break the mission.
| Hackathon bar | Where WildPal shows it |
|---|---|
| PS1: real-time, not a chatbox | Live voice + camera with barge-in and sensor context streamed into the session |
| PS1: decisions a chat UI never makes | Connectivity handoff mid-conversation and camera-noticed observations |
| Gemma: not a straight arrow | Visible SENSE → DECIDE → ACT → CHECK timeline with replans on tool failure |
| Gemma: local error recovery | JSON repair re-prompts and tool-failure recovery in amber timeline entries |
| Gemma: local state | Append-only ledger persisted every transition, including kill-and-resume |
| Gemma: human deferral | Approval gates on SOS and emergency SMS requests |
| Gemma: best use of E2B/E4B | RAM-tier model selection and on-device JSON function calling |
| 1 Go live with camera + voice. |
2 Cut signal and show the handoff. |
3 Let Gemma continue from the ledger. |
4 Deny SOS approval and show recovery. |
- Install dependencies.
npm install- Add a Gemini API key for the Live layer.
Create a .env file with EXPO_PUBLIC_GEMINI_API_KEY=..., or paste the key in the app on the Home tab.
- Add a Google Maps Android API key.
Put it in app.json under expo.android.config.googleMaps.apiKey, then re-run prebuild.
- Download the Gemma model on-device.
Open the app on a phone, go to Home, and download the recommended model once on Wi-Fi. The model tier is chosen from device RAM.
- Build the Android release APK.
$env:JAVA_HOME = "C:\Program Files\Android\Android Studio\jbr"
$env:ANDROID_HOME = "$env:LOCALAPPDATA\Android\Sdk"
npx expo prebuild --platform android --no-install
cd android; .\gradlew assembleReleaseThe APK ends up at android/app/build/outputs/apk/release/.
- React 19, TypeScript, and Vite.
- Capacitor for Android delivery.
@google/genaifor Gemini integration.llama.rnfor local Gemma 4 inference.- Leaflet and React Leaflet for map surfaces.
- Dexie and local storage for persisted app state.
- Start online in Assistant, point the camera at a plant or mushroom, and ask what it is. Interrupt the answer mid-sentence to show barge-in.
- Start a mission like “storm’s coming, get me ready,” then switch to airplane mode to show the handoff banner.
- Watch Gemma continue offline: sense the environment, build a checklist, set a waypoint, log the journal, and request approval for an emergency action.
- Deny the approval request and show the agent adapt and finish.
- Kill the app mid-mission and relaunch it to demonstrate resume-from-ledger.
If you need a dry run without the model, the app includes a demo mode for rehearsing the UI flow.
|
Not built turn-by-turn routing, custom tile caching, iOS |
Not built chat history UI, encryption, offline audio input |
Not built weather APIs, anything that weakens the core story |
Gemma 4 E2B QAT UD-Q2_K_XL (2.19 GB) runs via llama.rn on a OnePlus 5T (SD835, 6 GB RAM) at about 4.7 tok/s. Release APK builds locally, and the offline download → validate → load → generate path works end to end.
Turn-by-turn routing, custom tile caching, chat history UI, iOS, encryption, weather APIs, and offline audio input are out of scope for this build.