Skip to content

feat: support iOS simulator transform gesture#586

Merged
thymikee merged 1 commit into
mainfrom
codex/ios-transform-gesture
May 26, 2026
Merged

feat: support iOS simulator transform gesture#586
thymikee merged 1 commit into
mainfrom
codex/ios-transform-gesture

Conversation

@thymikee
Copy link
Copy Markdown
Member

Summary

Add iOS simulator support for gesture transform.

This wires transform to a private XCTest synthesized multi-touch path for pan/scale, applies XCTest rotation for the rotation component, removes the old handler-level decomposition helper, and updates help/docs plus the command-planning coverage.

Closes #582

Validation

Ran pnpm format, pnpm build, pnpm build:xcuitest, pnpm exec vitest run src/utils/__tests__/args.test.ts, pnpm check:quick, git diff --check, and five transform math sanity variations.

Manual verification on iPhone 17 with Metro on 8082: gesture transform 201 421 80 -40 2 35 700 changed the app metrics from x 0, y 0, scale 1.00, rotate 0 with all change flags no to x 57, y -28, scale 1.56, rotate 23 with pan/pinch/rotate all yes. SkillGym was not run per request.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 25, 2026

PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstackincubator.github.io/agent-device/pr-preview/pr-586/

Built to branch gh-pages at 2026-05-26 07:53 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@thymikee thymikee force-pushed the codex/ios-transform-gesture branch from 2a1dda9 to f6ff245 Compare May 25, 2026 18:31
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a1dda9fca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m Outdated
@thymikee thymikee force-pushed the codex/ios-transform-gesture branch 3 times, most recently from 6db01a0 to c3cbc62 Compare May 25, 2026 18:42
Copy link
Copy Markdown
Member Author

@thymikee thymikee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the iOS simulator transform changes. Using XCSynthesizedEventRecord/XCPointerEventPath is a nice step up from the old pan+pinch+rotate decomposition, and the fallback to .unsupported(message) when the private API throws is the right call.

Main concerns:

  1. The implementation is not one continuous two-finger gesture: pan+scale runs through the synthesized path (which ends with liftUpAtOffset:), and then rotation runs as a separate XCUIElement.rotate(...). That's at odds with the iOS smoke-suite contract you added ("Need one continuous two-finger gesture without lifting fingers"). Either:

    • Fold the rotation into the pointer path geometry (the Android helper already does this — interpolate θ(t) and place fingers at (centerX + cos(θ)*radius, centerY + sin(θ)*radius)), or
    • Update the contract + docs to acknowledge that rotation is a follow-up phase.
  2. Hardcoded pinch radius (80 pt) isn't tunable and may not work well on iPad / for scale < 0.5. Either parameterize it or scale it off the app frame.

  3. Robustness for private API drift: explicit nil checks on recordClass / pathClass and on each NSSelectorFromString would give clearer "Xcode version broke this" messages instead of an objc_msgSend crash.

  4. Typedef width: RunnerMsgSendLongLong claims long long, but processID / interfaceOrientation are NSInteger. Works today on LP64 but the typedef should match the selector signature.

  5. Codex bot's claim about the loop ending at t < 1.0 appears to be incorrect (the loop is <=). Suggest resolving that thread.

Nothing here is a hard block; the rotation-as-followup vs. atomic-gesture question is the one most worth aligning on before merge.


Generated by Claude Code

Comment thread ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m Outdated
Comment thread ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m Outdated
Comment thread ios-runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerSynthesizedGesture.m Outdated
Comment thread test/skillgym/suites/agent-device-smoke-suite.ts
@thymikee thymikee force-pushed the codex/ios-transform-gesture branch 2 times, most recently from cf87b48 to 8f9c55b Compare May 26, 2026 07:40
@thymikee thymikee force-pushed the codex/ios-transform-gesture branch from 8f9c55b to fdc928f Compare May 26, 2026 07:53
@thymikee thymikee merged commit ea21793 into main May 26, 2026
19 checks passed
@thymikee thymikee deleted the codex/ios-transform-gesture branch May 26, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explore continuous two-finger iOS transform gesture synthesis

1 participant