Held Keys and Modifier Chords

This release expands input automation with held keys, complete Defold key coverage, and modifier chords.

• Special keys can now be held for up to 60 seconds using hold with /input/key or Python’s game.key().
• All named dmHID::Key values are supported, including punctuation, symbol, keypad, lock, and system keys.
• Clicks, drags, drag paths, pointer sessions, and key presses now accept up to four chord modifiers. This enables actions such as Shift-click, Ctrl-drag, and Ctrl-Z.
• Modifiers are pressed before the primary action and released afterward, matching the ordering observed from physical keyboard input.
• Held keys remain active for their full requested duration, including with short controller leases.
• Invalid hold values, unsupported modifiers, and modifier chords with literal text now fail clearly instead of silently producing different input.
• Input receipts expose requested_duration and modifier_count, while /health advertises input.key>=2 and input.modifiers.
• Long gesture visualizations remain visible for their configured duration after the final injected point.
• The example project now displays held SPACE timing and visibly confirms an H + SHIFT + ALT chord.

Existing key taps and unmodified gestures require no changes. The Python wrapper checks the required capabilities before using held keys or modifiers.

API error envelopes now include the precise logical status in error.status. Because of a limitation in Defold’s embedded HTTP server, some logical errors are transported as HTTP 500; raw API clients should read error.status. The Python wrapper handles this automatically.

A big thank you to @jdbence for contributing held-key input, complete named-key coverage, and modifier chords!

Faster, Smaller Screenshots

This release makes screenshot-heavy automation substantially faster while greatly reducing disk usage.

• Native screenshots now use fast PNG compression instead of occupying roughly the same space as raw RGBA pixels.
• In the 2560×1440 beta benchmark, median capture time decreased from 189.6 ms to 24.7 ms—about 7.7× faster.
• The benchmark image decreased from 14.07 MiB to about 90 KiB. Compression ratios will vary with image content.
• Screenshots remain standard PNG files compatible with existing image tools.
• Downscaled captures using resolution_multiplier remain supported and use fast compression.

No API changes or migration steps are required.

Fix #12

Reliable Commands and Accurate Input Visualization

This release makes larger automation requests reliable and ensures click and drag debug visuals appear at the coordinates actually delivered to the game.

• start_command() now supports larger payloads without failing because the data was encoded into the request URL.
• Input, marker, video recording, and Metal capture operations use the same reliable request transport.
• Click and drag debug visuals remain aligned with screen coordinates when games use custom cameras, viewports, or render targets.
• Debug visualization requires no changes to the game project or render script and only draws when visualization is enabled.
• The raw Python request API now documents json_body= while continuing to support the existing json= argument.

No migration steps are required.

Fix #10

Faster Engine Connections

This release makes project.connect_engine() significantly faster when reconnecting to an already-running healthy game, especially on Windows.

• Reuses validated cached engine connections without repeatedly reading the Defold Editor console.
• Reads the editor console only once when engine rediscovery is required.
• Uses the IPv4 loopback address for editor communication, avoiding slow localhost resolution on affected systems.
• Continues to reject stale or reused engine ports and safely refreshes connections after engine restarts.

No API changes or migration steps are required.

Fix #8

Elements, Safer Input, and Automatic Updates

This release introduces Automation Bridge HTTP API v2 and Python wrapper 3.0, with safer element input and easier future upgrades.

What’s new

• Passing an Element to click(), or two Element objects to drag(), now verifies their runtime identities. If an element ID has been reused, the wrapper raises engine.StaleElementError instead of sending input to the wrong element.
• key() now accepts normalized names such as M, SPACE, KEY_ENTER, and {KEY_ESCAPE}. Unsupported names fail before input is queued.
• type_text() now always treats braces and other characters as literal UTF-8 text.
• project.update_automation_bridge() can update the project to the latest stable release or an explicitly pinned version. It updates game.project, fetches dependencies through Defold, replaces the complete project-root Python wrapper, and rolls back on failure.
• A Python migration guide is now included with the wrapper.

Migration from 2.0.x

Extension 2.1.0 and Python wrapper 3.0 must be updated together. Wrapper 3.0 communicates only with HTTP API v2.

2.0.x	2.1.0
/automation-bridge/v1	/automation-bridge/v2
/nodes and /node	/elements and /element
Response fields nodes and node	elements and element
Capabilities nodes and node	elements and element
Element IDs beginning with n:	Element IDs beginning with e:
automation_bridge.nodes	automation_bridge.elements; preferably use engine.Element and engine.Bounds

The documented high-level Python helpers—including element(), elements(), maybe_element(), element_by_id(), click(), drag(), and type_text()—retain their names.

For the initial upgrade, replace the complete Python wrapper directory instead of merging files, restart Python, and follow automation-bridge-python/MIGRATION.md. Future upgrades can use project.update_automation_bridge().

Python automation fixes

This release improves Python-based automation workflows:

• The Python wrapper can now be copied directly into a Defold project root and used with PYTHONPATH=automation-bridge-python.
• Fixed waiting for an input to reach the accepted state when using its input ID.
• Fixed trace replay so recorded request data, including click coordinates, is preserved.
• Successful asynchronous requests now return HTTP 200, removing the WARNING:DLIB: Unsupported status code: 202 message.
• Updated setup instructions and examples for both copied wrappers and source checkouts.

Reliable Editor Rebuilds

Fixes Automation Bridge becoming unavailable after subsequent builds from the Defold Editor.

• Keeps the Automation Bridge endpoint available across in-process engine reboots.
• Prevents the profiler route from shadowing Automation Bridge requests.
• Correctly recognizes cached ports when the engine identity changes during a same-process reboot.

Release 2.0.0

Highlights

• Increased the minimum supported Defold version from 1.12.4 to 1.13.1.
• Added deterministic FIFO input execution with lifecycle receipts and safe cancellation.
• Added an opt-in Lua synchronization API for events, state, commands, input acknowledgements, and semantic annotations.
• Added snapshot-safe scene queries, coordinate conversion, frame correlation, and atomic screenshots.
• Added native H.264 MP4 recording on macOS and Windows without FFmpeg.
• Added Metal .gputrace capture on macOS.
• Redesigned the Python wrapper around explicit editor and engine namespaces.
• Added editor discovery, build workflows, preview rendering, console access, debugger control, references, and typed preferences.
• Added structured traces, visual comparisons, generated gestures, bounded runtime logs, and improved profiling.

Breaking Python API changes

• The package root now exports only automation_bridge.editor and automation_bridge.engine.
• The Python package version is now 2.0.0; the native extension reports 1.3.0. The HTTP API remains v1.
• Replaced AutomationBridgeClient.from_project(...) with:
  • editor.open_project(...)
  • project.build_and_run()
  • project.clean_build_and_run()
  • project.connect_engine()
  • engine.connect(port)
• Renamed public scene terminology from Node to Element.
• Renamed:
  • nodes() → elements()
  • node() → element()
  • maybe_node() → maybe_element()
  • by_id() → element_by_id()
  • wait_for_node() → wait_for_element()
  • observe_node() → observe_element()
  • format_nodes() → format_elements()
  • wait_for_ack() → wait_for_input_acknowledgement()
  • recording → video_recording
• Removed compatibility aliases and direct get(), post(), put(), delete(), post_json(), and put_json() helpers. Use request(...) for raw endpoint access.
• Removed optional; use supports(...) for optional capabilities and require(...) for mandatory capabilities.
• Moved pixel operations under game.visual, gesture generation under game.gestures, recording under game.video_recording, Metal capture under game.metal_capture, and profiling under game.profiler.
• click(), drag(), and drag_path() wait for native release by default.
• type_text() and key() return after acceptance by default unless another wait state is requested.
• Public input helpers return typed native receipts instead of estimating completion with sleeps.

Deterministic input and pointer control

• Serialized clicks, drags, drag paths, pointers, and keys through one native FIFO.
• Added controller ownership using client_id, session_id, request_id, and renewable leases.
• Added lifecycle receipts with:
  • accepted, started, released, cancelled, and failed states
  • queue position
  • accepted/start/release frames and timestamps
  • requested and actual duration
  • device and pointer identity
  • engine instance, scene sequence, and request correlation
• Added input status, pending-queue inspection, cancellation, session flushing, and controller configuration.
• Added safe release of held mouse buttons, touch contacts, and keys when an action is cancelled, a lease expires, or a script is interrupted.
• Added explicit auto, mouse, and touch device selection without injecting the same gesture through multiple devices.
• Added capability reporting for supported input devices.
• Added stale-scene protection through expected_scene_sequence for node-targeted input.
• Added continuous drag paths with:
  • sampled and linear segments
  • quadratic and cubic Bézier curves
  • per-segment durations and easing
  • pre- and post-gesture holds
  • 2–128 control points
  • visualization of the actual injected path
• Added leased low-level pointer sessions with open, move, hold, up, and cancellation operations.
• Added structured InputReceipt, InputController, PointerSession, and interruption-scope APIs in Python.
• Added bounded structured JSON bodies for input mutations to prevent long query strings from being truncated.

Application synchronization

• Added the debug-only automation_bridge Lua module.

• The application API is opt-in through:

  [automation_bridge]
  application_api = 1
  event_capacity = 256

• Added structured application events with:

  • monotonic sequence numbers
  • explicit cursors
  • bounded retention
  • long polling
  • overflow detection without silently skipping lost events

• Added revisioned published state with exact-name lookup and long polling for newer revisions.

• Added registered application commands with:

  • strict JSON arguments
  • command IDs
  • pending, running, completed, failed, cancelled, and timed-out states
  • result and error records
  • cancellation of pending commands

• Added explicit application acknowledgement of native input through automation_bridge.acknowledge_input(...).

• Added semantic scene annotations through automation_bridge.annotate(...):

  • automation_id
  • localization_key
  • role

• Added timeline markers with native monotonic timestamps and optional recording-clock correlation.

• Added typed Python Event, EventStream, StateSnapshot, overflow, and command-timeout APIs.

• Event streams resolve "now" before an action begins, preserve unmatched events, and report buffer overflow explicitly.

• Added Python helpers for events, state waits, commands, markers, and full input acknowledgements.

Scene inspection and synchronization

• Added scene_sequence and engine_frame correlation to scene, node, screen, input, screenshot, and observation results.
• Added snapshot IDs distinct from persistent engine instance identity.
• Added game-object instance identifiers, allocation generations, logical IDs, and first-observed scene sequences.
• Fixed unsafe collection-root identity handling that could dereference uninitialized instance data during health discovery.
• Added server-side selectors for:
  • exact type, name, text, and URL
  • path, kind, instance ID, and logical ID
  • semantic automation ID, localization key, and role
  • visible, enabled, bounded, and visible-and-enabled state
• Added case-sensitive substring matching when requested.
• Added pagination with offsets/cursors, truncation state, next cursors, and complete match counts.
• Added active-collection and selector diagnostics.
• Added stale-snapshot rejection before resolving input targets.
• Zero is now accepted as a valid stale scene-sequence sentinel and returns stale_scene instead of bad_request.
• Added Python element appearance, disappearance, count, and multi-frame observation helpers.
• Added clearer selector errors with nearby candidate diagnostics.

Coordinates, frames, and resizing

• Defined top-left coordinate spaces for:
  • window
  • client area
  • drawable display pixels
  • backbuffer
  • viewport
  • normalized viewport
  • configured project content
• Added /coordinates/convert for conversions between advertised native spaces.
• Added /frame for native engine-frame and scene-sequence observation.
• Expanded screen metadata with named rectangles, offsets, project scale, scene sequence, and engine frame.
• Resize operations now distinguish:
  • already correct
  • successfully resized
  • resize requested but not yet observed
  • unsupported resize
• Added portrait and landscape helpers that retain the last known window size.
• Added frame waits with timeouts calculated from the requested frame count.
• Frame-wait failures now report initial and final frames, whether frames advanced, lifecycle stage, and best-effort health data.

Atomic screenshots and visual checks

• Replaced screenshot file-size polling with retained capture IDs and explicit pending, complete, and failed states.
• Added post-render after_frames scheduling.
• Added frame, scene-sequence, dimensions, path, failure reason, and SHA-256 metadata.
• Screenshots are written to temporary files and atomically renamed before completion is published.
• Standardized screenshot rows, scene bounds, pointer input, and coordinate conversion on top-left orientation.
• Fixed vertically inverted screenshots on macOS by preserving Defold’s already normalized readback row order.
• Added runtime screenshot scaling from 0.01 through 1.0, using bilinear downscaling while retaining the native source path.
• Added dependency-free PNG comparison with normalized mean absolute error.
• Added:
  • region comparisons
  • expected-image assertions
  • stable-frame waits
  • region-change waits
  • typed visual observation results

Native video recording

• Added /recording/capabilities, /recording/status, /recording/start, and /recording/stop.
• Recording runs inside the Defold process and does not require FFmpeg or another external recorder.
• Added native H.264 MP4 recording on macOS 15+ using ScreenCaptureKit.
• macOS recording:
  • automatically selects the Defold process window
  • captures the undecorated game content
  • supports application audio
  • supports resizing and 1–60 FPS
  • excludes title bars, shadows, and other applications
  • removes the rounded bottom-window safety area to produce rectangular video
• Added native H.264 MP4 recording on Windows 10 version 1903+ using Windows Graphics Capture, Direct3D 11, and Media Foundation.
• Windows recording:
  • selects the largest visible Defold-owned window
  • crops to the Win32 client area
  • supports resizing and FPS control
  • temporarily disables Windows 11 rounded corners and restores them afterward
  • is video-only and explicitly rejects audio=true
• Added synchronous finalization and output validation.
• Fixed the Win...

Better bounds detection

• improve clicking by better bounds calculation

1.0.0 - Release 🎆

This is the first release that covers most of the difficult issues. It still uses many private Defold APIs, so engine updates may break it. These private APIs will be iteratively replaced with proper DMSDK APIs as the needed APIs are introduced.