Record & Replay

Record a workflow on your screen, then turn it into a reusable Claude Code skill — using a vision model.

An open-source Claude Code plugin that ports Codex's Record & Replay to Claude Code and makes it work cross-platform with Chinese multimodal models. Demonstrate a task once; the plugin captures it, a vision model (MiniMax M3, Qwen-VL, GLM-4V, …) understands what you did, and writes a generalized SKILL.md Claude can replay later. Show it once, skip the script.

Why vision? Codex's original records low-level OS events (macOS accessibility APIs). That's brittle and macOS-only. Capturing screenshots and reasoning over them with a vision model works on Windows, macOS, and Linux, needs no accessibility permissions, and degrades gracefully when UIs change.

---

How it works

  record ──► screen video + mouse/key events ──► stop ──► encode mp4 ──► vision model ──► SKILL.md
  (you demo)        (background, nothing missed)          (batch)         (MiniMax M3 …)    (reusable skill)

1. Record — a background process captures a continuous screen video and records your mouse clicks and keystrokes, so nothing is missed between captures.
2. Stop — finalize the recording; frames are encoded into a single mp4.
3. Generate — the video plus the input-event timeline are sent to the vision model, which returns a structured, parameterized skill rendered into a Claude Code SKILL.md.

Video, not just screenshots. Periodic screenshots can miss a click that happens between captures, and they never show the actual keystrokes. By recording a continuous video and a timestamped log of clicks + keys, the model sees the true workflow — closer to how a human would re-learn the task by watching.

There is also a lighter frames mode (periodic screenshots only) for environments without ffmpeg.

---

Requirements

• Claude Code v2.1.128+ (plugin support).
• Python 3.9+ on your PATH (python --version).
• Python deps: mss, Pillow, requests, imageio-ffmpeg (bundles ffmpeg for video encode), pynput (input events). See Install.
• A vision-model API key (MiniMax, Qwen, GLM, or OpenAI). Test offline without one using the built-in mock provider.

---

Install

1. Add the plugin to Claude Code

From a checkout of this repo (the repo root is the plugin):

claude plugin validate .          # confirm the manifest is valid

Then enable it in a project. In a Claude Code session:

/plugin marketplace add minybear/record-replay     # once published, or use a local path
/plugin install record-replay

For local development you can also launch Claude Code with the plugin directly:

claude --plugin-dir .

2. Install the Python engine deps

pip install -r requirements.txt
# or: pip install mss Pillow requests

Verify everything is wired up:

orr doctor          # run inside a Claude Code session (bin/ is on PATH when enabled),
                    # or call the engine directly:  python -m orr doctor

---

Quickstart

Inside a Claude Code session (project where you've enabled the plugin):

/record-replay:record submit-expense-report

Switch to your browser/app and actually fill in & submit the expense report. Then:

/record-replay:stop submit-expense-report
/record-replay:generate submit-expense-report

You'll get a skill at .claude/skills/submit-expense-report/SKILL.md describing the steps, parameters, and a replay strategy. Reload skills/plugins and Claude can now drive that workflow.

No API key yet? Add --provider mock to generate to run the whole pipeline offline and inspect the output shape.

---

Configure a vision provider

Pick a provider and set a key. MiniMax M3 is the default.

orr config set provider minimax
orr config set providers.minimax.api_key sk-your-key-here

Or use an environment variable (takes effect immediately, no file write):

export MINIMAX_API_KEY=sk-your-key-here

Provider	Default model	Base URL	Env var
minimax (default)	MiniMax-M3	https://api.minimax.io/v1 (global); https://api.minimaxi.com/v1 (China)	MINIMAX_API_KEY
qwen	qwen-vl-max	https://dashscope.aliyuncs.com/compatible-mode/v1	DASHSCOPE_API_KEY
glm	glm-4v	https://open.bigmodel.cn/api/paas/v4	ZHIPU_API_KEY
openai	gpt-4o	https://api.openai.com/v1	OPENAI_API_KEY
mock	—	offline, no key	—

Switch a provider's base URL, model, or key any time:

orr config set providers.minimax.base_url https://api.minimaxi.com/v1   # China endpoint
orr config set providers.qwen.model qwen2.5-vl-72b-instruct

See docs/PROVIDERS.md for per-provider gotchas.

---

Commands

Slash command	Engine command	What it does
/record-replay:record <name>	orr record <name>	Start a background screen capture
/record-replay:stop [name]	orr stop [name]	Stop one (or all) recording
/record-replay:generate <name>	orr generate <name>	Analyze + write SKILL.md
/record-replay:list	orr list	Show recordings and which are active
/record-replay:config …	orr config … / orr doctor	Configure / health check

The plugin also ships a record-replay skill that auto-activates when you ask Claude to "record a workflow", "turn what I did into a skill", etc.

Useful record flags:

--mode auto|video|frames   video = continuous video + input events (recommended); auto = video if ffmpeg else frames
--fps 5                    video capture rate
--no-events                do not record mouse/keyboard (privacy)
--mask-text                record positions + key names, mask typed characters
--interval 2               frames mode: seconds between screenshots

Useful generate flags:

--provider minimax|qwen|glm|openai|mock   override the active provider
--max-frames 20                            cap frames sent to the model
--scope project|user|path:<dir>            where SKILL.md is written
--out <path>                               write to an exact file
--name <skill-name>                        override the generated skill name

---

Generate from an existing video or screenshots

Didn't use the live recorder? Point generate at media you already have:

orr frames expense --from-video recording.mp4 --fps 1   # needs ffmpeg
orr frames expense --from-dir ./screenshots
orr generate expense --provider minimax

---

Configuration reference

Config lives at ~/.orr/config.json (or $ORR_DATA_DIR / $CLAUDE_PLUGIN_DATA). Recordings live under ~/.orr/recordings/<name>/ (frames + manifest.json).

{
  "provider": "minimax",
  "providers": { "minimax": { "api_key": "", "base_url": "https://api.minimax.io/v1", "model": "MiniMax-M3" }, /* … */ },
  "capture": {
    "mode": "auto",            // auto | video | frames (auto=video if ffmpeg else frames)
    "interval": 2.0,           // frames mode: seconds between screenshots
    "fps": 5.0,                // video mode: capture frame rate
    "events": true,            // video mode: record mouse clicks + keystrokes
    "max_frames": 20,          // frames mode: cap sent to the model
    "max_dim": 1280,           // frames mode: downscale longest edge
    "jpeg_quality": 75,        // frames mode: quality sent to the model
    "dedupe_threshold": 0.012  // frames mode: drop near-identical frames
  },
  "skill_scope": "project"    // project | user
}

---

Troubleshooting

Symptom	Fix
No screen capture library	pip install mss Pillow
model did not return parseable JSON	Try another provider/model; inspect <recording>/analysis.raw.txt
HTTP 401/403	Wrong/missing API key — orr config set providers.<p>.api_key …. For MiniMax, a domestic (China) key must use https://api.minimaxi.com/v1 (extra i); the global api.minimax.io returns 401 — orr config set providers.minimax.base_url https://api.minimaxi.com/v1
HTTP 429 / 无可用资源	Provider account out of quota — top up credits
Recording captured 0 frames	The detached capture process was killed by your shell/IDE — run orr record <name> --foreground in a separate terminal, or check <recording>/server.log
Generated skill has 1 frame	The screen barely changed during recording — interact more so frames differ
python3 exits 49 on Windows	That's the Windows Store stub; the orr launcher skips it and uses python automatically
Don't want keystrokes recorded	Use --no-events (no input capture) or --mask-text (positions + key names only). Events stay local and go only to your configured provider
Video mode not used (falls back to frames)	pip install imageio-ffmpeg so ffmpeg is available; check with orr doctor

orr doctor --ping runs a minimal real call against the active provider to confirm the key works end-to-end.

---

How it's built

• Engine (orr/, Python): recorder → frame extractor → vision providers → skill generator. See docs/ARCHITECTURE.md.
• Plugin surface (commands/, skills/, bin/, .claude-plugin/): thin wrappers that invoke the engine.
• Providers are all OpenAI-compatible Chat Completions with image_url (base64) content — adding a new vision model is a one-class change. See docs/PROVIDERS.md.

---

Contributing

PRs welcome. Good first issues: more providers, richer event capture (mouse/keyboard) alongside screenshots, video-recording mode, and replay back-ends (e.g. driving the desktop-computer-automation skill to actually execute generated skills).

License

MIT — see LICENSE.