Skip to content

Unify React Native overlay analysis and reliable dismissal #566

@thymikee

Description

@thymikee

Consolidated scope

This issue is the canonical React Native overlay reliability issue and consolidates #565, #566, and #577.

React Native overlay handling is currently split across several places:

  • snapshot presentation decides which overlay accessibility nodes to preserve or suppress
  • snapshot warning output separately detects React Native overlay text
  • react-native dismiss-overlay separately resolves a safe target and sends an action
  • screenshot --overlay-refs does not share the same overlay action model

That split makes the agent experience less reliable: collapsed LogBox banners can have close affordances in different positions, failures can leave agents unsure whether to inspect refs or press visible text, and dismissal can continue without a verified post-action view when stricter automation needs certainty.

Goal

Create one React Native overlay analysis model that all overlay-related callers consume. The model should preserve enough fidelity for both user experience and reliability:

  • overlay kind, such as collapsed LogBox, expanded LogBox, or RedBox
  • severity and reporting metadata when available
  • visible evidence nodes used to identify the overlay
  • safe controls/actions, preferring snapshot refs or selectors over coordinates
  • explicit marking of coordinate fallback as fallback, not the primary target model
  • agent-facing warnings and next-command guidance
  • optional post-action verification state

User experience and reliability requirements

  • Prefer explicit accessibility targets when a close, dismiss, or minimize control is present.
  • Never press the warning body or a generic full-screen warning parent as a dismissal shortcut.
  • Use coordinate fallback only when the layout is confidently recognized as a safe collapsed banner affordance.
  • Handle non-LTR and non-trailing close targets; do not assume the close affordance is always on the trailing edge.
  • Keep the default react-native dismiss-overlay fast path: dismiss once, then tell the agent to take a fresh snapshot -i -c before continuing.
  • Add an explicit stricter verification mode, for example --verify, for automation that must confirm the overlay is gone before continuing.
  • Report verified: true only after a fresh post-dismiss snapshot no longer detects the overlay.
  • When dismissal is ambiguous, unsafe, or the overlay remains visible, return a useful warning or failure payload with the next action for the agent.
  • Failure guidance should point agents to screenshot --overlay-refs rather than pressing the banner body when no safe target is found.

Proposed direction

Model React Native overlays as snapshot analysis artifacts, for example snapshot.overlays[] or snapshot.issues[], with safe actions attached to each detected overlay. Then make these callers consume the same analysis result:

  • iOS snapshot presentation preserves or suppresses overlay nodes through the overlay analysis module
  • snapshot warnings are generated from the shared analysis
  • react-native dismiss-overlay consumes the shared safe-action model
  • verified dismissal performs a post-action snapshot before reporting final verified success
  • screenshot --overlay-refs annotates overlay targets consistently with the command's safe-action model

Acceptance criteria

  • One module owns React Native overlay recognition vocabulary and safe-action selection.
  • Snapshot presentation, warning generation, react-native dismiss-overlay, and screenshot --overlay-refs do not duplicate overlay heuristics.
  • Collapsed LogBox dismissal prefers a concrete close or dismiss node from the snapshot when available.
  • Collapsed LogBox dismissal remains safe for the existing iOS-style banner plus non-trailing or RTL-like close targets.
  • Coordinate fallback is used only when the layout is confidently recognized as a safe collapsed banner target.
  • Full-screen warning parents are not exposed as generic app targets.
  • RedBox behavior still prefers Minimize over Dismiss.
  • react-native dismiss-overlay keeps the current fast behavior by default and does not claim verified dismissal without a post-action snapshot.
  • A verification option is available for stricter automation, for example --verify.
  • Verified success reports verified: true only when a fresh post-dismiss snapshot no longer detects the overlay.
  • Ambiguous, unsafe, or still-visible overlays preserve a useful failure or warning payload with the next action for the agent.
  • Failure output tells agents to use screenshot --overlay-refs rather than pressing the banner body when no safe target is found.
  • Tests cover collapsed LogBox, expanded LogBox, RedBox, no-overlay app copy, existing iOS-style collapsed banner behavior, non-trailing or RTL-like close target behavior, verified success, and still-visible overlay behavior.

Scope guard

This is a React Native overlay reliability issue, not a generic overlay framework for every platform or native modal type. Keep changes focused on RN overlay recognition, safe dismissal actions, user-facing recovery guidance, and verification.

Blocked by

None - can start immediately

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions