Porter

A native macOS agent with no fixed command list.

If you can do it by seeing the screen and using a Mac, Porter can reason through it too—and it gets better every time.

Porter turns the Mac itself into one conversational interface. Say “Hey Porter” or press ⌘K, describe the outcome, and keep working. Porter binds the request to the exact window in front of you, reasons over what is visible, and either explains it or completes the task with real mouse and keyboard input.

Porter is not limited to a catalog of supported commands, applications, or prewritten workflows. It uses the same general loop a person does: observe the interface, decide what moves the task forward, act, inspect what changed, and adapt. If the screen takes an unexpected path, that path simply becomes the next state to reason about.

That makes Porter’s practical scope open-ended: anything that can be accomplished through visual reasoning and ordinary macOS interaction is in its problem space. The screen supplies the context, natural language supplies the intent, and the operating system supplies the tools. There is no command language to learn, automation graph to build, or connector to configure before Porter can try something new.

This is not a scripted concept. Porter is a working SwiftUI and AppKit application with local wake-word detection, multimodal screen understanding, a continuously adapting computer-use loop, inspectable personal memory, visible agent activity, and explicit human-control boundaries.

One loop, not a list of tricks

Porter does not select from rehearsed demos or a menu of supported tasks. Each request enters the same observe–reason–act loop, which means the next task does not have to resemble anything Porter has done before.

Ask the screen, not a chatbot

Leave a homework problem, chart, document, form, or unfamiliar interface visible and ask a natural question:

“How do I solve question two?”

Porter captures the exact focused window—not the whole composited display—and returns a structured answer in a native, movable panel. The question and the visual context arrive together, so the user does not have to describe what the computer can already see.

Delegate any visible workflow

Describe the end state you want. Porter observes the current interface, chooses one grounded action, performs it, checks the result, and continues. It does not need to know the complete route in advance, because it can revise the route after every observation.

Porter can click, type, scroll, wait through slow interfaces, move between applications, and recover when the screen differs from what it expected. Because it reasons about the interface in front of it, a new application is not an unsupported integration—it is simply a new visual environment to understand.

Turn one sentence into a finished outcome

Ask Porter to apply for a job. It can navigate the hiring workflow, understand the role, use what it knows about the user’s background and preferences, complete the application, adapt to questions it has never seen, and carry the process through from a single request. The user does not have to restate their history on every page or teach Porter the shape of every form.

The important part is not job applications. It is that Porter can combine visual reasoning, computer control, and personal context to complete an unfamiliar end-to-end workflow without that workflow being built into the product first.

Improve with every interaction

Porter adapts both within a task and across tasks. Over time, it can build a rich working model of everything the user chooses to teach it: their identity, background, experience, preferences, routines, recurring decisions, and the way they like work completed. That context compounds. The more Porter does, the less the user has to explain and the more completely Porter can act on a simple outcome.

This memory remains user-owned. What Porter learns is local, visible in the Memory screen, traceable to its source, correctable through explicit instructions, and removable at any time.

Why Porter is different

Product decision	What it changes
The screen is context	Users can refer to “this question,” “that chart,” or “the form” without translating the interface into a prompt.
Reasoning replaces routing	Requests do not have to match a finite intent or supported command. Porter reasons from the current screen and desired outcome.
The operating system is the tool layer	Any visible macOS interface can become part of a workflow without waiting for a purpose-built connector.
Adaptation happens continuously	Every action produces a new observation, so Porter can change course when an application, page, or task behaves differently than expected.
Control is observable	An iridescent edge appears whenever Porter is listening or acting, and the activity timeline shows what happened.
Memory compounds	Porter develops a richer model of the user over time, turning short requests into deeply personalized, end-to-end execution.
The human keeps final authority	A new “Hey Porter” request or physical keyboard/mouse input interrupts control, and user-defined approval boundaries can keep consequential actions manual.
The interface stays native	Porter appears only when needed through voice or a system-wide ⌘K surface, including over full-screen apps.

How it works

“Hey Porter” or ⌘K
          │
          ▼
intent + exact focused window + local preferences
          │
          ▼
multimodal model proposes one typed action or answer
          │
          ▼
Porter grounds the proposal in a fresh macOS observation
          │
          ▼
mouse / keyboard / Accessibility action
          │
          ▼
observe the result, verify progress, and continue

Nothing in this loop assumes a particular task or application in advance. Each action is selected from the current visual state, and every result becomes fresh evidence for the next decision. This is what lets Porter move from a familiar workflow to an unseen one, recover from surprises, and continue improving as its memory accumulates useful context.

Porter never hands unrestricted control to a model. Model output is decoded into typed actions, checked against current state, and grounded against a fresh observation before dispatch. The general loop combines visual coordinates with Accessibility semantics. Additional safeguards do not define what Porter can do; they add stronger guarantees where an outcome is consequential while leaving the underlying capability open-ended.

Porter also avoids observing itself by construction. Its visual pipeline captures one exact target window rather than the composited display, so companion panels do not leak back into the model’s view. External screen recorders can still capture the complete experience for demos.

Built beyond the happy path

Desktop software is messy: applications launch cold, pages load slowly, controls move, windows use different coordinate systems, and model output is not always perfect. Porter is engineered around those conditions rather than assuming a flawless demo environment. Its ability to observe again and change course is the capability—not a fallback bolted onto a scripted path.

The current build is a complete foundation for open-ended personal computer use:

• a native SwiftUI and AppKit application for Apple silicon;
• fully local “Hey Porter” wake-word detection, voice activity detection, and speech recognition;
• a system-wide ⌘K surface that works over full-screen applications;
• multimodal understanding of any focused window through ScreenCaptureKit and Accessibility-tree observation;
• contextual answers and explanations delivered in a movable, resizable native surface;
• open-ended visual computer use through mouse, keyboard, typing, scrolling, waiting, and application control;
• continuous re-observation and replanning across unfamiliar interfaces and multi-application workflows;
• a compounding personal context model capable of carrying the user’s knowledge, history, preferences, and judgment into new tasks;
• persistent local memory with provenance, explicit corrections, usage counts, and forgetting;
• a visible agent-active edge, physical-input interruption, and spoken cancellation;
• an inspectable activity timeline and privacy-safe local model-usage ledger; and
• 341 native tests spanning planning, observation, coordinate grounding, computer use, visual understanding, voice, memory, approvals, and runtime behavior.

Technology

Layer	Implementation
Interface	SwiftUI, AppKit, Liquid Glass
Vision	ScreenCaptureKit, CoreGraphics, Accessibility APIs
Control	Accessibility actions, CGEvent mouse and keyboard input
Local voice	AVAudioEngine, sherpa-onnx, Silero VAD
Reasoning	Multimodal planning and tutoring through Amazon Bedrock
Local state	macOS Keychain, typed preference memory, privacy-safe usage ledger

Run Porter

Porter currently targets Apple silicon, macOS 26, and Xcode 26.

git clone https://github.com/GodlyDonuts/Porter.git
cd Porter
./scripts/build_macos_app.sh --open

The script builds Porter, installs a stable copy at /Applications/Porter.app, and opens it. After granting a macOS permission, reopen that same application from Applications, Spotlight, Launchpad, or the Dock—do not rebuild it.

On first launch:

1. Grant Microphone, Accessibility, and Screen Recording access in Porter’s Permissions screen.
2. Add an Amazon Bedrock API key in Intelligence. The key is stored in the user’s Keychain and is never committed to this repository.
3. Put a question or task on screen, then say “Hey Porter” or press ⌘K.

The hackathon build uses a bring-your-own-key setup. A production version would hide provider configuration behind the product instead of asking a consumer to manage it.

Verify the project

Run the complete local gate:

./scripts/check.sh

Or run the native test suite directly:

DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
  xcodebuild test \
  -project native/PorterApp/Porter.xcodeproj \
  -scheme Porter \
  -destination 'platform=macOS'

Live tests that operate real macOS applications and files are opt-in and live under scripts/test_*_live.sh.

How Codex was used

Codex was an engineering partner throughout Porter’s development—not a one-shot project generator. The working loop was simple: build a vertical slice, test it against a real Mac, bring the failure back into the codebase, and turn that failure into a regression test.

That loop helped:

• trace and evolve a large Swift, SwiftUI, and AppKit codebase;
• implement ScreenCaptureKit and Accessibility observation;
• reconcile the three coordinate systems involved in model images, macOS windows, and CGEvent input;
• evolve the first purpose-built workflows into a general vision-and-action loop;
• diagnose cold launches, slow pages, scrolling failures, stale observations, and malformed model actions;
• build the complete local wake-word and speech-recognition integration;
• design the Command-K answer experience and agent-active screen edge; and
• convert real-world failures into a broad native regression suite.

The product direction came from hands-on testing and deliberate design decisions. Codex compressed the implementation and debugging loops enough to turn an ambitious OS-agent idea into a cohesive working application within the hackathon timeline.

Current build requirements

Porter is a hackathon build rather than a packaged production release. It currently requires the user’s Bedrock key, an ad-hoc local build, macOS permissions, and recent Apple-silicon hardware. These are deployment requirements, not a fixed capability boundary. User-defined safeguards can keep irreversible actions behind explicit approval, and Porter intentionally uses text answers rather than synthetic speech.

Bundled dependency and model notices are retained under third_party. They are part of the voice-runtime distribution and should not be removed from packaged builds.

License

Porter is released under the MIT License. Bundled third-party components remain subject to their respective notices and licenses.