[Feature Request] Native Mobile App (iOS & Android) with Voice Calling

[chefroger]

Feature Description

I'd love to see an official Hermes mobile app for both iOS and Android that supports voice calling — allowing users to have real-time phone conversations with their Hermes AI assistant.

Motivation

Voice communication is the most natural and hands-free way to interact with an AI assistant, especially for:

• Quick questions while driving, cooking, or exercising
• Users who prefer speaking over typing
• Accessibility (visual impairments, difficulty typing)
• Building a more personal, human-like connection with the assistant

Proposed Solution

A native mobile app (React Native or Flutter for cross-platform) that:

1. Voice Calls: Tap-to-call / always-listening wake word → real-time voice conversation with Hermes
2. Text Chat: Full messaging interface mirroring existing platform integrations
3. Push Notifications: Stay connected and receive timely updates
4. Authentication: Secure login linking to the user's Hermes account
5. Audio Pipeline: Uses existing Hermes TTS/STT capabilities (e.g., edge-tts, faster-whisper)
6. Background Mode: Continue conversations when app is in background

Alternatives Considered

• Browser-based PWA: Limited functionality, no true push notifications or background audio
• Third-party VoIP integrations: Adds dependency and complexity; native is cleaner
• WebRTC bridge: Could work but latency and reliability are concerns

Questions for Maintainers

1. Is this on the roadmap?
2. Are there technical docs on the existing voice/audio pipeline?
3. Would you accept a community PR for this?

---

Feature Request | Priority: High | Labels: feature-request, mobile, voice

[kshitijk4poor]

Collected the related work on this to help maintainers triage — turns out most of the pieces already exist in flight, and the remaining gap is really just an umbrella plan.

Existing threads covering parts of this ask:

• Local WebSocket bridge for mobile/local clients — [Feature]: Expose a local WebSocket interface for third-party clients (e.g. CoClaw mobile app) #11712 + PR feat(api-server): add local websocket bridge for agent clients #20564 (active, +756 / -15, generic /v1/ws on the API server, reuses bearer auth + _run_agent path, no client-specific code). This is the protocol-seam half of "native mobile app" and is effectively done once merged.
• Device control from a phone — PR feat: Android device control toolset #1345 attempted a 14-tool Android toolset in core; closed with the direction "great fit as a plugin or optional skill, not core" (v0.3.0 plugin architecture). Author agreed to split into a standalone plugin. That's the authoritative pattern for anything that touches phone sensors.
• Voice calls from/to a phone — [Feature]: AgentPhone voice call gateway platform #11063 (AgentPhone gateway platform, HTTP-webhook model, session-isolated), Feature: Twilio Skill — Phone Calls, SMS, and Voice from the Agent #409 (Twilio skill, placed in optional-skills/). These are voice-channel features, distinct from a companion app.
• Camera primitives — PR feat: add ai-camera skill — turn any camera into AI-powered vision system #13441 (ai-camera skill, already supports phone cameras via IP Webcam URL + existing vision_analyze).

Proposed architecture for the remaining "native mobile app" scope, consistent with the direction set above:

1. Core (already in flight via feat(api-server): add local websocket bridge for agent clients #20564): a stable, authenticated local JSON-frame protocol over WebSocket. Message-send + streaming deltas + session list/get/reset. No device concepts. Versioned (/v1/ws).
2. Capability surface for device-initiated actions (future, plugin-owned): when a client connects with a declared capability set (camera, mic, screen, location, push), the agent learns it can request those as tools. Core only needs a capability-advertisement frame on the existing protocol — no new adapters, no SDKs. Actual device code lives in the mobile app, not in this repo.
3. The mobile apps themselves: separate repos / separate distribution (TestFlight, Play Store), following the pattern raulvidis committed to in feat: Android device control toolset #1345. They talk to the /v1/ws bridge and register capability frames. Keeping them out of the main repo avoids App Store signing, Fastlane, Gradle, and native-toolchain overhead landing in a Python agent repo — and lets the mobile UX iterate independently.
4. Voice calling stays its own track: [Feature]: AgentPhone voice call gateway platform #11063 / Feature: Twilio Skill — Phone Calls, SMS, and Voice from the Agent #409 solve "call me on my phone" via telephony gateways; this issue should stay scoped to "the app on my phone talks to my agent". They're complementary but shouldn't be conflated.

Concrete ask for maintainers: is the plan above consistent with the direction you want? If so, #11911 can probably be reframed as an umbrella tracking issue that (a) waits on #20564, (b) adds a capability-advertisement frame as a small follow-up PR once /v1/ws is merged, and (c) points at an external mobile-app repo as the reference client. I'm happy to open the follow-up PR for the capability frame on top of #20564 once it lands, and to write a short developer-guide/mobile-clients.md documenting the protocol from the app-author side.

[Nsomnia]

node makes it not easy. with glm 4.7 free and women Gemini key switching in opencode earlier this year I tried doing a native apk and it's possible but tons of hurdles.

[konsisumer]

This issue appears to describe the same problem as #15774, #35966. Maintainer might want to consolidate or pick one as canonical.

[HangbinYang]

Hi everyone, I’m the developer of HermesPilot, a native mobile client for Hermes Agent.

It’s designed for people who already run Hermes Agent locally or on their own machine and want a smoother way to access it from their phone.

Download:

• iOS App Store: https://apps.apple.com/app/hermespilot/id6765671330

What it does:

• Connects your phone to your own Hermes Agent through a local companion service called Hermes Link
• Supports pairing by QR code or manual pairing code
• Lets you chat with Hermes Agent from a native mobile interface
• Supports local-network access when available, with relay fallback for remote access
• Keeps multiple Links / machines manageable from the app
• Includes conversation management and mobile-friendly message rendering
• Supports push notifications for background task updates, where available
• Does not replace Hermes Agent itself; it is only a mobile client and connection layer for your existing Hermes setup

Basic setup:

npm install -g @hermespilot/link
hermeslink pair