feat(adapters): convert window-scoped capture sessions in their own pixel space#146
Conversation
|
Pausing this as a draft despite green CI. Review found that the current head can borrow a future viewport for earlier actions, accept mid-recording resize even though the MP4/Flow model is fixed-size, ignore malformed window metadata, round an in-range float outside the viewport, and fail to bind actual frame size to metadata. A corrected local patch is ready at 246d28a. After the 1.14.1 replacement release is verified, rebase onto exact main, apply the equivalent fixes, run protected CI, then reconsider merge. |
…ixel space Consume openadapt-capture's new window recording mode (capture #30): - Detect CaptureSession.window_capture, with a defensive fallback to the recording's config JSON for installed capture versions that predate the property — a missed window session would silently double-scale every coordinate. - Skip pixel_ratio rescaling: window-mode action coordinates are already in captured-frame pixels (coordinate_space == "window_pixels"); frames pass through as-is (already the client-window viewport rdp_window replays in). - Screen every mouse action against the recorded bounds-timeline window events (mid-recording resizes honored); out-of-window input refuses conversion loudly instead of being dropped or compiled as a wrong-target step. - Refuse loudly on an unknown declared coordinate_space or missing viewport metadata (out-of-window input could not be screened). - Stamp the output meta.json with window_capture provenance plus backend_hints (rdp_window / rdp_window_title, the recorded target substrings) so replay --backend rdp can resolve the same window; additive keys the compiler ignores, nothing in replay reads them yet. - Non-window sessions byte-identical (meta key-set regression test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
a0cb0a7 to
047a4e5
Compare
|
Replaced the original head with the corrected implementation, rebased onto released 1.14.1 ( |
Consumes openadapt-capture's new window-scoped recording mode (OpenAdaptAI/openadapt-capture#30, merged to capture main) in the recording adapter, so a window-mode session can never be silently mis-converted.
What a window-scoped session gets
All four behaviors from the capture #30 spec are implemented (no guard-only fallback was needed):
CaptureSession.window_capturedeclarescoordinate_space == "window_pixels": action coordinates were translated at capture time into the captured frame's pixel space, so the adapter converts withscale = 1.0. Applyingpixel_ratiowould double-scale every click — the regression test records withpixel_ratio = 2.0present and asserts coordinates pass through exactly.rdp_windowreplay captures; the frame path is untouched (asserted).state.viewportper bounds/title change, so a mid-recording resize is honored per-action; staticwindow_capture["viewport"]fallback when no timeline rows exist). Fail-safe: refuse loudly, per this module's existing loud-rejection policy — window capture records out-of-window input at out-of-range coordinates instead of clamping, and such an action targeted a different window: dropping it would silently lose a demonstrated action, keeping it would compile a wrong-target step. Error:out-of-window input: <type> at (x, y) (t=...) falls outside the captured window viewport WxH; ... re-record keeping all input inside the captured window.meta.jsonoutput only (the compiler ignores unknown meta keys;ir.pyuntouched):window_captureprovenance (target + resolved owner/title, coordinate space) andbackend_hints={"backend": "rdp", "rdp_window": <target owner>, "rdp_window_title": <target title>}— named inBackendConfigterms soreplay --backend rdpcan resolve the same client window. The hints carry the recorded target substrings (the user's proven-to-resolve intent), not the volatile resolved title. Wiring these hints into the backend factory / CLI defaults is a deliberate follow-up (those files are out of scope here).Additional guards (refuse loudly, never mis-scale):
window-scoped capture session declares coordinate_space=..., which this adapter does not understand (expected 'window_pixels'); converting it could silently mis-scale action coordinates — upgrade openadapt-flow ...Non-window sessions are unchanged;
test_meta_matches_recorder_contractnow pins the exact non-window meta key set so window-mode fields can never leak into it.Capture pin / unreleased API
The
captureextra pin staysopenadapt-capture>=0.5.4: window mode is on capture main but not yet on PyPI (latest release is 0.5.4). The adapter reads the new surface defensively —getattr(session, "window_capture", ...)with a fallback to the recording's persistedconfig["capture_window"](the same dict the property reads) andgetattraccess to the bounds-timeline rows. This is also a safety property, not just a CI convenience: a window-scoped session loaded through an older installed capture package is still recognized instead of silently double-scaled.Tests build real capture sessions (SQLAlchemy
recording.db+ real video via capture's public writers) rather than fakes, so CI'stestjob (which installs thecaptureextra from PyPI) exercises the fallback path for real. The full suite was also run locally against capture main installed from a clone (property path): 21/21 pass both ways. When the next capture release ships, the pin can be bumped in a trivial follow-up.Tests (8 new)
pixel_ratio=2.0)window_capture+backend_hints(target-substring semantics)ruff check, ruff format, lenient whole-package mypy, docs-consistency gate, and
tests/test_capture_adapter.py(21 passed) all green locally.🤖 Generated with Claude Code