Explore continuous two-finger iOS transform gesture synthesis

[thymikee]

Problem

gesture transform is intended to model a composed gesture: pan, scale, and rotate without lifting fingers. Android now has the right shape because the multi-touch helper can inject two pointer streams in one gesture. iOS currently approximates this with sequential public XCTest operations: pan, then pinch, then rotate.

That sequence works for basic verification, but it is not semantically equivalent to a real two-finger transform. Apps using gesture recognizers such as React Native Gesture Handler can observe it as separate gesture lifecycles instead of one continuous transform, and the current implementation cannot express simultaneous translation + scale + rotation.

Goal

Implement an iOS backend for gesture transform that sends two continuous finger paths in one synthesized event:

center(t) = startCenter + t * (dx, dy)
radius(t) = startRadius * scale(t)
angle(t)  = startAngle + t * degrees

fingerA(t) = center(t) + rotatedVector(radius(t), angle(t))
fingerB(t) = center(t) - rotatedVector(radius(t), angle(t))

Both contacts should go down once, move through sampled points, and lift once. Pan, pinch, and rotate should emerge from those two paths rather than being performed as separate procedural commands.

Research Notes

The public XCTest APIs appear insufficient for this exact behavior:

• XCUIElement.pinch(withScale:velocity:) performs a real pinch but does not let us combine translation and rotation into the same gesture.
• XCUIElement.rotate(_:withVelocity:) performs a real rotate but is separate from pinch and pan.
• XCUICoordinate.press(forDuration:thenDragTo:...) exposes one pointer path, but it is not clear that two concurrent calls can run as one gesture; XCTest likely serializes public actions.

There is a promising lower-level XCTest/XCUIAutomation path. In Xcode 26.2's simulator XCUIAutomation.framework, these private classes/selectors are present:

• XCSynthesizedEventRecord
  • initWithName:displayID:interfaceOrientation:
  • addPointerEventPath:
  • synthesizeWithError:
• XCPointerEventPath
  • initForTouchAtPoint:offset:
  • pressDownAtOffset:
  • moveToPoint:atOffset:
  • liftUpAtOffset:
• XCPointerEvent
  • includes event construction helpers such as moveEventWithStartPoint:destination:offset:duration: and related pointer event state.

This maps closely to the model we need: build two XCPointerEventPaths, add both to one XCSynthesizedEventRecord, and synthesize it once.

Proposed Approach

1. Prototype private XCUIAutomation event synthesis in the iOS XCTest runner behind runtime selector checks.
2. Build two sampled touch paths from the existing gesture transform x y dx dy scale degrees [durationMs] inputs.
3. Dispatch both paths in one synthesized event record.
4. If the private classes/selectors are unavailable, return a clear UNSUPPORTED_OPERATION or fall back only where the command contract explicitly allows it.
5. Verify on the React Native test app by asserting the on-screen transform metrics change in one command: x, y, scale, and rotate.
6. Check behavior on both iOS simulator and physical device if possible. If private event synthesis is simulator-only or Xcode-version-sensitive, document that explicitly in capabilities/help.

Acceptance Criteria

• agent-device gesture transform on iOS sends one continuous two-finger gesture, not pan -> pinch -> rotate as separate gestures.
• React Native Gesture Handler test app observes translation, scale, and rotation in the same transform interaction.
• Existing gesture pinch and gesture rotate can either keep using public XCTest APIs or be optionally reimplemented as special cases of the same two-pointer path model.
• Runtime feature detection prevents crashes when private XCTest selectors are unavailable.
• Integration/e2e coverage proves the command changes x, y, scale, and rotate on the test app.
• Help/capability docs accurately describe iOS support and any simulator/device limitations.

Risks / Open Questions

• These APIs are private and may change across Xcode versions.
• We need to confirm whether synthesizeWithError: is callable from the runner process in current simulator and physical-device test contexts.
• Display ID, target process ID, coordinate space, and interface orientation need to be set correctly for reliable injection.
• If this only works on simulator, physical iOS support should remain explicit rather than silently approximating the gesture.