Repository navigation
Simulator Control — an iOS Simulator plugin for DSH #10189
terri-yaki
started this conversation in
Show Your Plugins!
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Simulator Control — iOS Simulator plugin for DSH
Unofficial community project, independently built and maintained. MIT licensed.
Repository: https://github.com/terri-yaki/dsh-simulator-control
Tested with: DSH 0.2.1-rc.2 · macOS 27.0.1 · Xcode 27.1 · iOS 27.0 simulator runtime · AXe 1.8.0
DSH agents can write and build iOS apps, but they cannot normally see a simulator or touch it. I built a Host + Client bundle that closes that loop: the agent gets 16 simulator tools, and you get a live Simulator tab in the right sidebar to watch and click along.
The screenshot below is from the plugin itself mid-flow:
sim_testlaunched Settings, navigated with accessibility selectors, andsim_screenshotcaptured the result.What it does
sim_devices,sim_boot,sim_shutdown,sim_doctorsim_app— list / find builds in DerivedData / install / run / launch / terminate / uninstall / container pathsim_screenshot(attaches the PNG when the model accepts images),sim_describe_ui(compact accessibility tree: labels, ids, values, point frames)sim_tap(label / id / value / coordinates, with waits),sim_type,sim_swipe,sim_button(home, lock, side, Siri, Apple Pay)sim_open_url(deep links),sim_env(appearance, status-bar override, location, privacy, pasteboard, open Device Hub),sim_logssim_videoH.264 clips;sim_testmulti-step flows withassertwaits, per-step PASS/FAIL and a final attached screenshotAuto-discovery and boot. With no booted simulator, the tools boot the most recently used iPhone on the newest runtime (or
DSH_SIM_DEFAULT_DEVICE), and say so in the result;sim_devicesandsim_doctorshow the auto-boot target first. Several booted simulators are common on a dev Mac, sodevice(name or UDID) disambiguates.App lookup without paths.
sim_app findlists simulator.appbuilds from~/Library/Developer/Xcode/DerivedData;sim_app run --bundle_idinstalls the newest matching build when needed and launches it.Side panel. A right-sidebar tab type shows the live screen (~1 fps, downscaled server-side), with a device picker, Boot/Shut down, Home, Refresh, click-to-tap on the exact point, and a text field + Send. Host routes live under
/simulator-control/{state,frame,action}and apply the Web GUI's Connection trust checks (no session → 401).Install
Profile is patched by the plugin manager; nothing to hand-edit. After installing (or updating) the Client half, refresh the page once to pick up the new tab type.
Optional environment tunables read at plugin load:
DSH_SIM_DEFAULT_DEVICE,DSH_SIM_AUTOBOOT=0,DSH_SIM_BOOT_TIMEOUT,DSH_SIM_AXE.Why AXe instead of the common recipes
Simulator.appis gone (Device Hub replaced it).idbis awkward to install now, and CGEvent/AppleScript clicking needs macOS Accessibility grants.This plugin is deliberately a thin, auditable wrapper:
xcrun simctlfor lifecycle/screenshots/logs/video, AXe for UI inspection and input, and one canonical implementation per action shared by the individual tools,sim_test, and the side panel.What I verified
The bundle ships a 37-check E2E (
node scripts/e2e.mjs) that boots a simulator, drives Settings through label/id selectors, types, scrolls, finds + installs + runs an app from DerivedData, exercises environment toggles, runs passing and failingsim_testflows, records a video, checks the side-panel routes (including 401 without a trusted session), and asserts no simulator wallpaper-service crashes appeared during the run. It passes on a disposable iPhone 17 simulator, and the side-panel tab was confirmed live in the GUI.Notes worth sharing (mostly Xcode 27 simulator behavior)
sim_describe_uiand retry. Also, touch works while system HID (Home/lock, keyboard) can drop out on some Device Hub states — the plugin verifies Home and falls back to a deterministic dismissal instead of claiming success.sim_doctornow reports recent PosterBoard crashes. If a device is already in that loop, erasing it in Device Hub is usually the fix.--labelfails on duplicates;sim_describe_uiprints accessibility ids as#id.simctl pbcopycan be overridden by the macOS clipboard; the tool reports the observed value.sim_typesends HID keycodes, so a non-Latin layout (e.g. Zhuyin) reinterprets them; switch the simulator keyboard to English before typing.Questions for the maintainers / plugin authors
@deepseek-ai/schemasteryfor aConfigexport) failed withERR_MODULE_NOT_FOUND. I fell back to environment variables. Is importing shipped packages from profile-installed bundles supported? If so, what is the correct pattern, and should plugin tunables preferConfig+ the patch layer over env vars?ctx.sidebarRightTabs.registerplus a keyed body onsidebar.right.pane.tabwith a guide entry. Is that the intended public path for third-party panels, and is there stability guidance for thesidebarRight/sidebarRightTabssurface?Happy to turn any of the verified behavior into docs or an upstream example if useful.
License: MIT · AXe is a separate MIT project installed via Homebrew (not vendored).
All reactions