Use Codex without watching it.
CodeByEar is an open-source, voice-first macOS companion for blind and low-vision people who want to create with Codex. The default route assumes no GitHub account and no knowledge of Git or terminals. It turns Codex activity into a few understandable states, explains decisions in plain language, and keeps important actions behind deliberate confirmation.
Current release: functional local prototype for OpenAI Build Week. The source, safe practice, live Codex App Server path, and ad-hoc signed Mac app are implemented. The current source passes 276 tests with 0 failures and 6 intentionally gated live/Keychain skips. The installed-app accessibility harness passes on an unlocked desktop, including first use, one-button work-folder creation, recovery, high-risk blocking, and purpose-first URL reading. The app is not Developer ID signed or notarized. Optional Realtime voice uses separate OpenAI Platform billing; the normal VoiceOver, Voice Control, and Dictation route does not require it.
Once the final release archive and checksum are published, judges and non-developers can use that
unchanged archive without rebuilding. Existing dist/release-* and review-working-tree-* ZIPs are
not submission candidates; only a new archive produced from the final clean commit may be used.
Judges should follow Install the prebuilt Mac app only with the published
checksum. The guide never asks anyone to disable Gatekeeper globally.
The release packager rebuilds the app from the same clean Git commit used for the source archive
and embeds that commit in CodeByEarSourceRevision inside the app's Info.plist. Packaging fails
if the app and source revisions differ.
- Press Return for a safe practice, then choose Create my work folder. CodeByEar creates one private local folder; no GitHub account is created or connected.
- Speak or type an everyday request such as “make a simple page for my shop.” Hear only the current state and what to do next.
- When a decision is needed, hear what can change and approve or reject it. At the end, hear what changed and whether the automatic checks passed.
A raw agent stream is difficult to navigate by sound. CodeByEar does not read every line. It keeps one large current state and one next action on screen and in speech:
- what Codex is doing now;
- whether Codex is working, waiting for a decision, finished, or stopped by an error;
- what can change when approval is needed;
- whether automatic checks actually passed or failed;
- the single useful action to take next.
The first launch detects VoiceOver and offers one Return-key safe practice that cannot change files. Skip practice and continue to easy setup goes directly to the same one-button work-folder setup. Language, text size, and macOS Voice Control remain under Optional setup. Realtime voice is an advanced option under Settings → Connection and is not required. Opening Voice Control uses the native macOS pane; when the user returns, a persistent card explains what to do without pretending that CodeByEar changed a protected Mac setting.
Four controls stay available throughout the app:
- What is happening?
- What should I do?
- Repeat
- Stop speaking
With macOS Voice Control, each visible control can be activated by name. VoiceOver handles navigation and reads CodeByEar's structured output. Inside CodeByEar, Voice Control or Dictation fills the request field. People who already dictate requests in ChatGPT desktop or Codex can keep using those products' own input surfaces; CodeByEar's unique job is making a Codex task's state understandable by ear. Another app's private live event stream is not scraped or controlled.
- Native SwiftUI app for macOS 14+
- Japanese, English, and system-language modes with runtime switching and persistence
- VoiceOver-first safe practice and a direct easy-setup route
- One-action private local work-folder creation with no GitHub account or remote connection
- Voice Control and Dictation input using macOS; no bundled local speech model and no Realtime key required
- Four always-available controls for status, next action, repeat, and immediate speech stop
- Large text from 85% through 160% with reflow instead of clipping
- Plain-language working, decision, completion, failure, and recovery states
- Purpose-first link speech, file-name-first path speech, and technical details only on request
- Contextual Settings guidance that opens the correct tab and identifies the exact control
- Local Codex App Server JSON-RPC connection with GPT-5.6 threads constrained to the selected work folder
- Revalidation of the work-folder device, inode, and owner immediately before a real Codex task starts
- Exclusive private-folder creation, isolated Git configuration, no remotes or hooks, symlink rejection, and no recursive cleanup of uncertain data
- One-time file, action, permission, and network approvals with four risk levels
- Mac owner authentication for high and critical risk; voice and accessibility activation alone cannot authorize them
- Fail-closed handling for unknown or incomplete server requests and actions outside the selected folder
- Deterministic safe practice using the production state, approval, and result surfaces
- Optional OpenAI Realtime speech with interruption, typed tools, separate billing disclosure, Keychain storage, and diagnostics
Requirements:
- macOS 14 Sonoma or later
- Xcode 15 or a compatible Swift toolchain
CodeByEar does not download or run a local speech LLM. The current installed app is about 11 MB and measured about 141 MB resident memory while idle; Codex subprocess memory varies with the coding task. Optional Realtime speech runs remotely, so the default VoiceOver route does not reserve gigabytes for a local model.
From the repository directory:
./script/build_and_run.sh --demoThe script builds dist/CodeByEar.app, installs a development copy at ~/Applications/CodeByEar.app, ad-hoc signs it, and opens the installed copy. Installing before launch avoids macOS protected-folder prompts caused by running an ad-hoc app from a source tree inside Documents.
For a build, launch, and real on-screen-window check:
./script/build_and_run.sh --verifyFor an unattended replay smoke:
./script/build_and_run.sh --demo-auto-approveThe synthetic fixture proposes two file changes, requests an automatic check, and ends with seven recorded passing checks. Safe practice is visibly labeled and is never evidence that a live action ran.
For a reproducible live Codex demonstration, create a separate temporary Git project with one intentional login-validation failure:
./Demo/prepare-live-sample.shThe script copies the temporary path to the clipboard. In CodeByEar's picker press
Command-Shift-G, paste, and press Return; use --show-path only for exact-path review. The tracked
template is never modified, and the live result must be reported from actual App Server events
rather than the replay's fixed counts.
Real work requires an installed and authenticated Codex CLI. Realtime voice is optional. VoiceOver reads the interface and CodeByEar status; macOS Voice Control and Dictation are the built-in speech-input routes.
- Start CodeByEar with
./script/build_and_run.sh. - If VoiceOver is already on, press Return for Start a short safe practice. Otherwise press Command-F5 first. Choose Skip practice and continue to easy setup for the direct route.
- Choose Create my work folder. CodeByEar creates and selects a private local folder without GitHub. If you already have a folder prepared for Codex, choose Choose an existing folder instead.
- In What would you like help with?, speak or type an everyday request and choose Start.
- Hear each decision, then approve or reject it. A physical approval applies once. High-risk requests always require macOS owner authentication.
- Hear what changed and whether the automatic checks passed when Codex finishes.
The no-Realtime route creates no Realtime API usage. VoiceOver handles navigation and output but does not transcribe speech: macOS Voice Control or Dictation provides built-in speech input, and the keyboard is the verified fallback. CodeByEar cannot silently change macOS Voice Control's vocabulary, so unfamiliar names remain visible in the task field for review before execution. The first-run card and Settings → Accessibility open the native Accessibility → Voice Control pane and speak the exact controls. CodeByEar sends Codex phase, approval, risk, and result announcements through VoiceOver. Codex itself still uses the signed-in Codex plan or the Codex CLI's configured authentication. Exact-candidate Voice Control dictation remains listed as an open hardware check rather than a completed claim.
To add OpenAI Realtime speech-to-speech, open Settings → Connection and save an OpenAI Platform API key in Keychain. Realtime transcription receives only a short, fixed list of common developer terms. The vocabulary prompt never adds project names, paths, branch names, or file contents. Saved-key verification starts automatically and, with VoiceOver, pauses behind an explicit Continue to macOS Keychain step. Then run Test live Realtime voice without microphone. This paid check sends only a short text prompt, disables push-to-talk while it runs, and must return real model audio before push-to-talk is considered ready. Next run the local microphone check. With VoiceOver it separates permission and recording into two confirmed steps; recording begins after a tone, lasts two seconds, rejects silence without sending it, and is discarded locally.
The local development script ad-hoc signs each rebuilt app. In the verified development workflow, macOS therefore asked for Keychain approval again after replacing the app. This is not evidence that the key was lost: use the separate macOS password window, or cancel and continue without Realtime. Judges using one unchanged published archive do not rebuild between launches.
ChatGPT and Codex credits do not pay for Realtime API usage. Realtime uses separate OpenAI Platform billing. The app explains this next to the API-key controls and provides direct, semantically labeled links to Platform billing and the selected model's official pricing.
Rates checked on 2026-07-20:
| Model | Text input / output per 1M tokens | Audio input / output per 1M tokens |
|---|---|---|
gpt-realtime-2.1 |
US$4 / US$24 | US$32 / US$64 |
gpt-realtime-2.1-mini |
US$0.60 / US$2.40 | US$10 / US$20 |
Push-to-talk also requests an asynchronous text transcript from gpt-4o-mini-transcribe.
That model is billed separately at US$1.25 per 1 million audio input tokens and US$5 per
1 million text output tokens. The no-microphone diagnostic sends text, so it incurs no
transcription charge and its displayed estimate excludes transcription. Rates were checked on
2026-07-20.
CodeByEar limits each model response to 512 output tokens. If all 512 were billed as output audio, that component would be about US$0.0328 on standard or US$0.0102 on mini. Input, text, conversation context, and reasoning can add cost, so the result screen reports the actual returned token count and a rate-based estimate for the no-microphone check. The OpenAI Platform usage dashboard remains authoritative.
| Action | Shortcut |
|---|---|
| Choose an existing work folder | Command + O |
| Start the request | Command + Return |
| Stop the Codex task | Command + . |
| Connect or disconnect Realtime | Command + K |
| What is happening? | Command + Option + 1 |
| What should I do? | Command + Option + 2 |
| Repeat | Command + Option + 3 |
| Stop speaking | Command + Option + 0 |
| Push to talk | Hold unmodified Space |
| Send voice input | Release Space |
| Stop the voice response | Escape |
| Open Settings | Visible Settings button or Command + , |
| Confirm a high-risk action | Command + Shift + Return |
Modified Space is ignored by the tested keyboard event policy, so VoiceOver's Control + Option + Space remains available. The real VoiceOver process used that command to activate first-run and approval controls without starting push-to-talk in the no-key audit. Connected-Realtime conflict testing and the full rotor/cadence review remain tracked in the verification ledger.
Press Command + F5 to turn VoiceOver on or off. The first-run card and Accessibility Settings
explain the division of responsibility: VoiceOver navigates controls, macOS consent dialogs, and
braille output; CodeByEar supplies concise Codex state, execution truth, approval risk, and one next
safe action. While VoiceOver is running, automatic status updates are posted to VoiceOver instead
of starting a competing Mac voice. The status, choices, repeat, and VoiceOver-help actions stop an
active Realtime reply before announcing, and use VoiceOver or the Mac voice, never both. CodeByEar
also observes VoiceOver's running state so Command-F5 stops an already-playing Mac welcome. It
moves VoiceOver focus when the session reaches a new useful action or blocking state. It does not
run a delayed second focus request: manual VoiceOver movement is preserved, and approval or error
events arriving while Settings is key cannot move VoiceOver back to the session view. The verified
core path is recorded in
docs/VOICEOVER_LIVE_AUDIT.md.
Speech requests actions; it does not execute them directly. Realtime receives a closed set of typed functions. Codex starts in an isolated app-specific home with an attested selected-project filesystem profile. Model-generated tool network is disabled, and only the isolated app temporary directory is writable outside the project. App Server requests pass through the local approval broker.
The broker enforces these invariants:
- approvals apply to one request only;
- there is no “approve everything” path;
- unknown request types and incomplete known approval payloads fail closed;
- permission expansion is high risk;
- high and critical actions cannot be approved by voice;
- the exact selected directory is passed as the runtime workspace root;
- the App Server must attest that the active profile,
cwd, sole runtime root, instruction sources, approval policy, and MCP state match CodeByEar's policy before work starts; - symlink-aware checks reject command, file, and permission requests that escape the selected project;
- decisions are redacted and appended to the local audit history.
String-based risk classification is defense in depth, not a proof that arbitrary shell code is safe. The primary boundaries are the OS-enforced project-only permission profile, isolated Codex configuration, disabled network and external tools, startup attestation, explicit one-time approval, macOS device-owner authentication for high-risk actions, and fail-closed protocol handling. Read docs/security-model.md.
Microphone / keyboard
|
v
OpenAI Realtime API -- typed function calls -- AppStore
|
AgentController
/ | \
state UI risk broker audit
|
Codex App Server
|
selected work folder
Replay events enter the same normalizer, approval UI, and result surfaces as live App Server events. Realtime and Codex have separate lifecycles, so the safe guide works without either service.
See docs/architecture.md and docs/DESIGN.md.
Run the release suite:
./script/verify_release.shInclude the authenticated local Codex round trip when the signed-in Codex account is available:
./script/verify_release.sh --live-codexWith the desktop unlocked and Accessibility access granted to the invoking terminal, include the real app accessibility-tree and keyboard acceptance flow:
./script/verify_release.sh --accessibilityThat isolated audit uses temporary preferences and history. It exercises the first-run structure, the exact-control Settings guide, runtime Japanese-to-English switching, two keyboard approvals, the verified replay result, the high-risk owner-authentication boundary, structured error recovery, semantic link review, safe copy, and warned full copy without touching a real project. It does not replace a human VoiceOver speech-quality or target-user test.
The current app source passes 276 tests with 0 failures and 6 intentionally gated skips during the credential-free phase. The latest authenticated release gate also passed three Codex tests: a GPT-5.6 edit-and-verify turn, an OS-level outside-project read denial, and a full GPT-5.6 turn in which an outside secret never appeared. The combined command stops with exit 77 rather than claiming an accessibility pass when the desktop is locked.
./script/verify_release.sh --live-codex --accessibilityWith a temporary key supplied through a secure process environment, include the paid live Realtime text-to-audio and semantic-URL check:
./script/verify_release.sh --live-realtimeThe command requires CODEBYEAR_LIVE_REALTIME_API_KEY to have already been injected by a secure
environment mechanism. Do not place a key in source, shell history, or documentation. The installed
app's Settings flow is preferred for a person: Keychain may require a one-time macOS approval.
The release suite also creates, updates, reads, and removes a dedicated non-secret test item in the logged-in macOS Keychain. It never reads or replaces the production API-key item. A macOS login session that has been unlocked at least once is therefore required even when the AX audit is omitted.
The regular suite is credential-free. Current evidence and explicitly unverified hardware checks are separated in docs/VERIFICATION.md, with review verdicts in docs/QUALITY_REVIEW.md.
Sources/CodeByEar/
Models/ localized task, approval, and replay state
Services/ Realtime, App Server, audio, safety, and audit logic
Stores/ application and agent-session coordination
Views/ accessible SwiftUI task, approval, history, and Settings UI
Resources/ English/Japanese strings and the deterministic replay
Tests/ deterministic and gated live integration tests
Demo/ replay fixture plus reproducible synthetic live sample
docs/ design, architecture, accessibility, security, and submission evidence
script/ build, verification, and release packaging tools
- The API key is stored in macOS Keychain and is never written to source or preferences.
- Live microphone audio is sent to OpenAI only while Realtime is connected and the user records.
- The no-microphone live check sends a short text prompt and receives paid model audio; it sends no microphone data.
- Codex App Server runs as a local child process in CodeByEar's isolated Codex home. It links only the existing Codex authentication file so the signed-in Codex plan can be used; user configuration, memories, global instructions, plugins, and MCP servers are not loaded.
- Approval history remains in
~/Library/Application Support/CodeByEar/action-history.jsonl. - The prototype has no analytics, advertising SDK, or project-operated telemetry backend.
The direct standard-key Realtime architecture is intended for personal local testing. A generally distributed product should use an authenticated backend to issue short-lived client secrets. Read PRIVACY.md and SECURITY.md.
CodeByEar is licensed under Apache-2.0. Contribution and conduct guidance are in CONTRIBUTING.md and CODE_OF_CONDUCT.md.
Codex was the primary implementation environment for the Build Week work. In a GPT-5.6 Codex thread, the project specification was turned into the native App Server client, normalized agent state, one-time approval broker, bilingual SwiftUI flows, semantic technical-speech formatter, deterministic replay, and macOS accessibility harness. Codex also helped convert review findings into regression tests—for example, preventing voice from enabling automatic startup, keeping the startup decision out of the approval walkthrough, and making spoken choices match the controls currently on screen.
The main technical decisions made with Codex were to consume structured App Server events instead of scraping a terminal, give Realtime only narrow typed functions rather than an arbitrary shell, start Codex with an attested selected-project filesystem profile and model-generated tool network disabled, fail closed on unknown approval requests, and keep recorded evidence visibly separate from live GPT-5.6 execution. Human direction set the target-user needs and safety boundaries. The final guidance candidate passed the clean-HEAD release gate, the documented real-app accessibility checks, and independent static review. Real VoiceOver cadence and rotor review plus target-user acceptance remain explicit gates rather than inferred passes.
GPT-5.6 is also part of the running product: CodeByEar starts a local Codex App Server thread rooted at the exact selected Git folder, and GPT-5.6 performs the live inspection, edit, and verification. OpenAI Realtime is a separate optional speech layer and is not described as GPT-5.6.
Submission materials are prepared in:
- docs/BUILD_WEEK_SUBMISSION_DRAFT.md
- docs/build-week.md
- docs/demo-script.md
- docs/SUBMISSION_CHECKLIST.md
- docs/COMPLETION_AUDIT.md
- docs/VOICE_CONTROL_LIVE_AUDIT.md
Repository publication, release upload, YouTube upload, /feedback, and Devpost submission are external actions reserved for the project owner.
- OpenAI Build Week
- Build Week official rules
- Build Week FAQ
- Realtime conversations
- Realtime WebSocket guide
- GPT-Realtime-2.1 pricing
- GPT-Realtime-2.1 mini pricing
- Transcription and speech API pricing
- Codex App Server
Apache License 2.0. See LICENSE.