feat: add gesture telemetry recording overlays#245
Merged
Conversation
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d352b5c7b1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+62
to
+63
| const result = await runCmd('swift', ['-', filePath], { | ||
| stdin: VIDEO_VALIDATION_SCRIPT, |
There was a problem hiding this comment.
Android record stop now depends on isPlayableVideo, which shells out to swift/AVFoundation; on non-macOS hosts this validator is unavailable, so valid pulled MP4s are treated as unplayable and copyAndroidRecordingWithValidation eventually fails with pulled file is not a playable MP4. This breaks Android recording stop in environments like the repo’s Ubuntu Android workflow, so the validation path should be cross-platform or gracefully skipped when the validator runtime is missing.
Useful? React with 👍 / 👎.
Comment on lines
+598
to
+600
| field: 'gestureTelemetry', | ||
| path: recording.telemetryPath, | ||
| fileName: path.basename(recording.telemetryPath), |
There was a problem hiding this comment.
The new telemetry artifact is emitted as field: 'gestureTelemetry' and without localPath, while the response key is telemetryPath; in remote-daemon flows artifact materialization keys off artifact.field and requires a client path, so this artifact is not materialized back and telemetryPath remains a daemon-local path. As a result remote clients cannot reliably download/use the telemetry sidecar from record stop.
Useful? React with 👍 / 👎.
thymikee
force-pushed
the
codex/show-touches-main
branch
from
776d414 to
71ca739
Compare
thymikee
force-pushed
the
codex/show-touches-main
branch
from
71ca739 to
369b02c
Compare
ken0nek
added a commit
to ken0nek/agent-device
that referenced
this pull request
Mar 25, 2026
PR callstack#248 refactored interaction.ts into a thin routing layer delegating to focused command modules. PR callstack#245 (gesture telemetry overlays) unintentionally reverted get/is/scrollintoview back to inline implementations due to a stale branch. This restores the thin router from callstack#248, adapted for callstack#245's handleTouchInteractionCommands, and removes interaction-press.ts and interaction-fill.ts which were made dead code by callstack#245's interaction-touch.ts.
thymikee
pushed a commit
that referenced
this pull request
Mar 25, 2026
PR #248 refactored interaction.ts into a thin routing layer delegating to focused command modules. PR #245 (gesture telemetry overlays) unintentionally reverted get/is/scrollintoview back to inline implementations due to a stale branch. This restores the thin router from #248, adapted for #245's handleTouchInteractionCommands, and removes interaction-press.ts and interaction-fill.ts which were made dead code by #245's interaction-touch.ts.
thymikee
added a commit
that referenced
this pull request
Jun 1, 2026
…ry.swift Behavior-preserving relocation: move the text-entry concern out of the (formerly 1805-line) RunnerTests+Interaction.swift — the repo's most-changed file — into a new RunnerTests+TextEntry.swift. Interaction.swift drops to 1068 lines (-737). Moved verbatim (no logic changes): - value types: TextTypingRepairMode, TextEntryTiming, TextEntryResult, TextEntryTarget - the focus -> type -> verify -> repair pipeline (typeTextReliably + focus orchestration + readiness polling + dropped-char/repair heuristics) - clearTextInput and the text-entry leaf helpers (editableTextValue, isPlaceholderValue, isGenericTextInputLabel, normalizedElementText, moveCaretToEnd, estimatedDeleteCount, keyboardBecameVisible, keyboardElementExists) The whole cluster is text-entry-exclusive (its only callers are within the moved code or the already-internal command entry points), so every symbol keeps its original visibility — no access widened. Shared helpers used by gestures/snapshot/get-text (isKeyboardVisible, visibleKeyboardFrame, textInputAt, textInputCandidatesAt, readableText, ...) stay in Interaction. The new file is auto-included via the project's file-system-synchronized group. This is the safe, cosmetic half of the "TextEntry engine" architecture candidate. A deeper extraction behind a real seam/type is intentionally deferred: it carries the #245 revert risk and wants stronger text-entry e2e coverage first. Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED. Pure relocation, no behavior change.
thymikee
added a commit
that referenced
this pull request
Jun 1, 2026
…ry.swift (#651) Behavior-preserving relocation: move the text-entry concern out of the (formerly 1805-line) RunnerTests+Interaction.swift — the repo's most-changed file — into a new RunnerTests+TextEntry.swift. Interaction.swift drops to 1068 lines (-737). Moved verbatim (no logic changes): - value types: TextTypingRepairMode, TextEntryTiming, TextEntryResult, TextEntryTarget - the focus -> type -> verify -> repair pipeline (typeTextReliably + focus orchestration + readiness polling + dropped-char/repair heuristics) - clearTextInput and the text-entry leaf helpers (editableTextValue, isPlaceholderValue, isGenericTextInputLabel, normalizedElementText, moveCaretToEnd, estimatedDeleteCount, keyboardBecameVisible, keyboardElementExists) The whole cluster is text-entry-exclusive (its only callers are within the moved code or the already-internal command entry points), so every symbol keeps its original visibility — no access widened. Shared helpers used by gestures/snapshot/get-text (isKeyboardVisible, visibleKeyboardFrame, textInputAt, textInputCandidatesAt, readableText, ...) stay in Interaction. The new file is auto-included via the project's file-system-synchronized group. This is the safe, cosmetic half of the "TextEntry engine" architecture candidate. A deeper extraction behind a real seam/type is intentionally deferred: it carries the #245 revert risk and wants stronger text-entry e2e coverage first. Verified: xcodebuild build-for-testing -> TEST BUILD SUCCEEDED. Pure relocation, no behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace recording-time touch visualization with a telemetry-first pipeline.
Improve recording reliability and timing.
screenrecord+ overlay rendering and remove the old screenshot-encoder pathAdd a real verification harness for this feature.
Validation
pnpm typecheckpnpm test:unitpnpm test:smokepnpm build:xcuitestAGENT_DEVICE_RECORDING_E2E=1 node --test test/integration/recording-overlay.test.tsKnown gap:
screenrecordstop path is still less stable than longer recordings.