Skip to content

feat(adapters): convert window-scoped capture sessions in their own pixel space#146

Merged
abrichr merged 2 commits into
mainfrom
feat/capture-window-mode-adapter
Jul 18, 2026
Merged

feat(adapters): convert window-scoped capture sessions in their own pixel space#146
abrichr merged 2 commits into
mainfrom
feat/capture-window-mode-adapter

Conversation

@abrichr

@abrichr abrichr commented Jul 18, 2026

Copy link
Copy Markdown
Member

Consumes openadapt-capture's new window-scoped recording mode (OpenAdaptAI/openadapt-capture#30, merged to capture main) in the recording adapter, so a window-mode session can never be silently mis-converted.

What a window-scoped session gets

All four behaviors from the capture #30 spec are implemented (no guard-only fallback was needed):

  • (a) No pixel_ratio rescaling. CaptureSession.window_capture declares coordinate_space == "window_pixels": action coordinates were translated at capture time into the captured frame's pixel space, so the adapter converts with scale = 1.0. Applying pixel_ratio would double-scale every click — the regression test records with pixel_ratio = 2.0 present and asserts coordinates pass through exactly.
  • (b) Frames as-is. Window-mode video frames are already the client-window viewport that rdp_window replay captures; the frame path is untouched (asserted).
  • (c) Bounds-timeline screening. Every mouse action is validated against the recording's bounds-timeline WindowEvents (state.viewport per bounds/title change, so a mid-recording resize is honored per-action; static window_capture["viewport"] fallback when no timeline rows exist). Fail-safe: refuse loudly, per this module's existing loud-rejection policy — window capture records out-of-window input at out-of-range coordinates instead of clamping, and such an action targeted a different window: dropping it would silently lose a demonstrated action, keeping it would compile a wrong-target step. Error: out-of-window input: <type> at (x, y) (t=...) falls outside the captured window viewport WxH; ... re-record keeping all input inside the captured window.
  • (d) Backend hints stamped. No backend-hints field exists on the recording or bundle today, so per scope this adds additive keys to the ADAPTER's meta.json output only (the compiler ignores unknown meta keys; ir.py untouched): window_capture provenance (target + resolved owner/title, coordinate space) and backend_hints = {"backend": "rdp", "rdp_window": <target owner>, "rdp_window_title": <target title>} — named in BackendConfig terms so replay --backend rdp can resolve the same client window. The hints carry the recorded target substrings (the user's proven-to-resolve intent), not the volatile resolved title. Wiring these hints into the backend factory / CLI defaults is a deliberate follow-up (those files are out of scope here).

Additional guards (refuse loudly, never mis-scale):

  • A window-scoped session declaring an unknown coordinate_space is refused: window-scoped capture session declares coordinate_space=..., which this adapter does not understand (expected 'window_pixels'); converting it could silently mis-scale action coordinates — upgrade openadapt-flow ...
  • A window-scoped session with no viewport metadata anywhere (no timeline rows, no config viewport) is refused — out-of-window input could not be screened.

Non-window sessions are unchanged; test_meta_matches_recorder_contract now pins the exact non-window meta key set so window-mode fields can never leak into it.

Capture pin / unreleased API

The capture extra pin stays openadapt-capture>=0.5.4: window mode is on capture main but not yet on PyPI (latest release is 0.5.4). The adapter reads the new surface defensively — getattr(session, "window_capture", ...) with a fallback to the recording's persisted config["capture_window"] (the same dict the property reads) and getattr access to the bounds-timeline rows. This is also a safety property, not just a CI convenience: a window-scoped session loaded through an older installed capture package is still recognized instead of silently double-scaled.

Tests build real capture sessions (SQLAlchemy recording.db + real video via capture's public writers) rather than fakes, so CI's test job (which installs the capture extra from PyPI) exercises the fallback path for real. The full suite was also run locally against capture main installed from a clone (property path): 21/21 pass both ways. When the next capture release ships, the pin can be bumped in a trivial follow-up.

Tests (8 new)

  • window-mode happy path + double-scale regression (coordinates NOT rescaled despite pixel_ratio=2.0)
  • frames taken as-is (viewport = captured frame size)
  • meta stamps window_capture + backend_hints (target-substring semantics)
  • out-of-window click rejected; out-of-window scroll rejected
  • mid-recording resize honored via bounds-timeline WindowEvents (post-resize click rejected against the new viewport)
  • unknown coordinate_space refused; missing viewport refused
  • non-window meta key-set regression

ruff check, ruff format, lenient whole-package mypy, docs-consistency gate, and tests/test_capture_adapter.py (21 passed) all green locally.

🤖 Generated with Claude Code

@abrichr
abrichr marked this pull request as draft July 18, 2026 19:27
@abrichr

abrichr commented Jul 18, 2026

Copy link
Copy Markdown
Member Author

Pausing this as a draft despite green CI. Review found that the current head can borrow a future viewport for earlier actions, accept mid-recording resize even though the MP4/Flow model is fixed-size, ignore malformed window metadata, round an in-range float outside the viewport, and fail to bind actual frame size to metadata. A corrected local patch is ready at 246d28a. After the 1.14.1 replacement release is verified, rebase onto exact main, apply the equivalent fixes, run protected CI, then reconsider merge.

abrichr and others added 2 commits July 18, 2026 16:08
…ixel space

Consume openadapt-capture's new window recording mode (capture #30):

- Detect CaptureSession.window_capture, with a defensive fallback to the
  recording's config JSON for installed capture versions that predate the
  property — a missed window session would silently double-scale every
  coordinate.
- Skip pixel_ratio rescaling: window-mode action coordinates are already
  in captured-frame pixels (coordinate_space == "window_pixels"); frames
  pass through as-is (already the client-window viewport rdp_window
  replays in).
- Screen every mouse action against the recorded bounds-timeline window
  events (mid-recording resizes honored); out-of-window input refuses
  conversion loudly instead of being dropped or compiled as a
  wrong-target step.
- Refuse loudly on an unknown declared coordinate_space or missing
  viewport metadata (out-of-window input could not be screened).
- Stamp the output meta.json with window_capture provenance plus
  backend_hints (rdp_window / rdp_window_title, the recorded target
  substrings) so replay --backend rdp can resolve the same window;
  additive keys the compiler ignores, nothing in replay reads them yet.
- Non-window sessions byte-identical (meta key-set regression test).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abrichr
abrichr force-pushed the feat/capture-window-mode-adapter branch from a0cb0a7 to 047a4e5 Compare July 18, 2026 20:09
@abrichr

abrichr commented Jul 18, 2026

Copy link
Copy Markdown
Member Author

Replaced the original head with the corrected implementation, rebased onto released 1.14.1 (047a4e5). The adapter now fails closed on mid-record window resize, malformed/mixed window metadata, frame-size mismatch, future-viewport borrowing, and coordinate-rounding drift instead of silently rescaling. Local verification after rebase: 33 focused adapter tests, Ruff, mypy, release consistency, and diff check pass. Keeping this PR in draft until the exact-head full CI matrix is green.

@abrichr
abrichr marked this pull request as ready for review July 18, 2026 21:38
@abrichr
abrichr merged commit 1bd3f55 into main Jul 18, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant