Taplyne 0.3.0

Taplyne 0.3.0 adds live voice, an iPhone/iPad companion, encrypted remote control, persistent conversations, and saved workflows. Free and open source under AGPL-3.0-only. AI usage is billed by your chosen provider.

Download and start

Download Taplyne.dmg, open it, and drag Taplyne to Applications. The installer includes a START HERE.txt guide. The same app is available as Taplyne.zip. Both support Apple silicon and Intel on macOS 15 or later; the app and DMG are Developer ID signed, notarized by Apple, and stapled.

Follow the getting-started guide. Start with USB and Bluetooth, then choose your own OpenAI API key or Claude Code login in Settings > Agent.

What changed

• Live two-way OpenAI voice, interruption, native dictation and spoken replies. Live replies default to English and switch language on request.
• Native companion for iPhone and iPad, with QR pairing, text, voice, and device-scoped credentials.
• Encrypted relay control across networks, signed runner setup, reconnect handling and rejection of stale commands.
• Encrypted conversation history, task steering, saved workflows, reusable inputs and run history.
• Direct OpenAI BYOK, an optional ChatGPT sign-in flow, improved Unicode input through the remote runner, and more reliable navigation on iPad.

Remote setup and current limits

The Mac download is ready to install. The companion and relay are distributed as source, with setup instructions. Remote mode requires Xcode/Apple development signing, Developer Mode, and your own deployed TLS relay. There is no hosted relay account or App Store/TestFlight companion in this release. The Mac must remain running. If the device restarts or iOS stops its runner, reconnect locally and launch it again.

Physical iPad testing passed unplugged screen capture, Home control, companion chat and a spoken Home command followed by an audible background reply while using an iPhone cellular hotspot and keeping the Mac on Wi-Fi. The signed runner also passed capture and Home on a physical iPhone. This is evidence for those tested devices and network paths; it is not a compatibility claim for every iOS device or carrier.

Live ChatGPT consent, external audio route changes and echo behavior remain unverified. See the verification matrix for the full measured scope.

Verification includes focused server, agent/auth, input, relay and voice tests; both native app builds; provider checks; physical-device acceptance; universal Release build; source secret scans; and Apple notarization. SHA-256 checksums are included below.

Taplyne 0.2.2

Taplyne gives AI agents a real iPhone through local USB screen capture and Bluetooth mouse and keyboard input.

Download Taplyne.dmg, open it, and drag Taplyne to Applications. The app and installer are signed with Developer ID, notarized by Apple, and stapled. The same universal app is available as Taplyne.zip. Both Apple silicon and Intel Macs are supported on macOS 15 or later.

This release makes phone actions faster and more reliable. After each input Taplyne now waits until the screen stops moving before it checks the result, so an action on a screen that is still animating is reported as unverified instead of failing on a half-drawn frame. When a label moves between the moment an agent reads the screen and the moment it taps, Taplyne finds the same label on the same page and taps its new position; a label that is repeated or appears on a different page is refused rather than guessed. The pointer reuses its last verified position instead of re-anchoring on every move, and hover timing, keyboard input and navigation feedback are improved. Delivered input is never repeated automatically.

Verification: 40 server tests including new action timing tests, Bluetooth input safety regressions, universal release build, source secret scan, and Apple notarization of the app and the DMG. Physical input coverage is limited and this release does not claim a device compatibility matrix.

Setup is unchanged from 0.2.1; see the README. Source is licensed under AGPL-3.0-only.

Taplyne 0.2.1

Taplyne 0.2.1 improves pointer tracking around enlarging iPhone icons, restores direct English keyboard input, waits for navigation transitions, and shows useful action errors. Pause and resume no longer report intentional process termination as an agent failure.

Download Taplyne.dmg and drag Taplyne to Applications, or use Taplyne.zip. Both contain the same universal macOS 15+ app, signed with Developer ID and notarized by Apple.

Validation: 36 server tests, pointer/Bluetooth/input and agent regression checks, universal Release build, notarization and installer inspection. Physical checks passed for Search, Settings, General, Back, Home, and fresh English text entry. The complete built-in agent Settings task did not pass: it encountered stale-frame and pointer checks. The subsequent app-opening fix passed automated tests, but its physical confirmation is pending because the phone was in use.

Known limits: Hebrew and emoji paste failed in the latest phone checks even with Handoff enabled. Unicode paste remains experimental and reports uncertainty. OCR supplies visible text and bounds rather than a native accessibility tree. Compatibility across devices and Developer Mode disabled remain unverified. Argent can be used alongside Taplyne through their separate MCP servers; no native Argent backend is included.

Source: AGPL-3.0-only. Personal device captures and original private repository history are excluded from the public source.

Taplyne 0.2.0

Taplyne gives AI agents a real iPhone through local USB screen capture and Bluetooth mouse and keyboard input.

Download Taplyne.dmg, open it, and drag Taplyne to Applications. The app and installer are signed with Developer ID, notarized by Apple, and stapled. The same universal app is available as Taplyne.zip. Both Apple silicon and Intel Macs are supported on macOS 15 or later.

This release adds fresh screen references and closed-loop pointer aiming, local English and optional Hebrew OCR, label taps and bounded scrolling, explicit result verification, Unicode clipboard input, field and form tools, navigation, and human-controlled pause, takeover and resume. MCP and REST share the same action contract. Older coordinate clients need to supply frame references; typing and key presses need them too.

Install the USB setup helper with brew install uv && uv tool install pymobiledevice3. Allow Camera access for USB screen capture, pair Bluetooth, enable AssistiveTouch, and calibrate the pointer. Text entry requires Handoff and the same Apple Account on the Mac and iPhone. The setup guide and optional Hebrew OCR installer are in the README.

An early release: private macOS Bluetooth APIs make device compatibility variable. OCR describes visible text rather than a native accessibility tree. A form stops before pasting when it cannot establish the intended field's focus. Input delivery and task success are reported separately, and uncertain actions are never automatically repeated. Developer Mode is not required by the capture and input architecture; operation with it disabled still needs physical-device validation. Argent remains a separate transport rather than a native Taplyne backend.

Verification: 33 server tests, real English and Hebrew OCR fixtures, Bluetooth and input regressions, universal release build, Mac pause/takeover/resume/tools UI checks, source and release privacy scans, dependency audit, and Apple notarization. Physical input coverage is limited and this release does not claim a device compatibility matrix.

Source is licensed under AGPL-3.0-only. See README, SECURITY.md, and THIRD_PARTY_NOTICES.md for details.