feat: integrate opt-in SOCKS5 tunnel into WebDriverAgent - #29
Merged
Conversation
…mework build script Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ev YAML builder Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…action Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # WebDriverAgent.xcodeproj/project.pbxproj
Every NetworkExtension call from these handlers failed after exactly FBSocks5PreferencesTimeout (10s) with "Cannot load the VPN preferences: timed out", on both free and paid teams. Without .onControlQueue, FBWebServer serves a route via dispatch_sync(automationQueue) -> dispatch_sync(main), so the handler body runs on the main thread inside a dispatch_sync'd main-queue block. NETunnelProviderManager delivers loadAllFromPreferences completions on the main queue, which cannot be drained while that sync block is spinning FBRunLoopSpinner, so the completion never ran and the spinner always timed out. The device log showed "NetworkExtension: Loading all configurations" with no result line following it. Serving these routes on the connection's own queue lets the main queue drain the completion. /stats drops from 10.058s to 0.033s, and connect now brings the tunnel up on device (verified on iPhone SE / iOS 26.5 with traffic traversing an external SOCKS5 proxy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…firmed
The consent tap logged success while the alert stayed on screen, so
saveToPreferences never completed and connect failed with "Timed out
saving the VPN configuration. The consent alert was confirmed".
It diverged from FBBroadcastManager's dismissal tap in three ways, each
of which mattered:
- It synthesized the event through the runner, while reading the button
frame out of SpringBoard's coordinate space. The record is stamped
with the receiver's interface orientation, so the tap could land
somewhere else entirely. Synthesize via the system app instead.
- It used the blocking variant inside a bounded spin. That
acknowledgement can take the full event-synthesis timeout margin when
the system sheds the event, which the spin's own deadline cannot
interrupt. Dispatch fire-and-forget instead.
- It latched after one apparently successful dispatch. A dispatched tap
is not a landed tap, so a shed event meant the alert stood forever.
Keep re-attempting while the alert is observed, paced by a 1s cooldown
so a re-tap cannot land during the dismissal animation.
Verified on iPhone SE / iOS 26.5 against a fresh install: the first tap is
shed and the retry 1.09s later lands, the alert clears, and connect
returns connected in 2.2s.
The XCUI access also has to reach the main thread now that these routes
are served off it, so FBWebServer exposes the automation funnel it already
used for main-queue-served routes. Hopping straight to the main queue
would let the block run inside another handler's run-loop spin, which is
the reentrancy that funnel exists to prevent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three distribution gaps left the tunnel target unbuildable for everyone who is not working in a recursive clone: - The npm files allowlist shipped Scripts/build.sh but none of the inputs it now needs, so an installed appium-webdriveragent could not build WDA at all: the runner build exited on the missing engine source and Xcode referenced absent target files. Ship WebDriverAgentTunnel, the engine sources and the two new scripts. - build-hev-socks5-tunnel.sh keyed its up-to-date stamp on the submodule's Git SHA, which does not exist in the npm tarball. Fall back to hashing the engine source tree, which keys the stamp on what is actually being compiled either way. - No workflow checked out submodules, so every TARGET=runner job would hit build.sh's hard failure before reaching Xcode. The publish job needs them too, otherwise it uploads a tarball with an empty ThirdParty. Also add the tunnel paths to the WDA Tests trigger, so a change confined to WebDriverAgentTunnel/, ThirdParty/, .gitmodules or either new script cannot merge without runner build coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verify the SOCKS5 path before reporting connected. hev does not dial the proxy until tunneled traffic creates a session, so the provider reported startup success as soon as the engine thread was spawned: NetworkExtension reached NEVPNStatusConnected, the endpoint answered connected: true, and every packet was blackholed. The provider now performs a SOCKS5 greeting (plus username/password sub-negotiation when credentials are configured) before installing any routes, holds startup pending until the engine has survived its initialization window, and cancels the tunnel if the engine later exits on its own. connectWithURI: additionally stops as soon as the session settles back to disconnected instead of spinning out the caller's deadline - an unreachable proxy now fails in ~2.5s with an accurate message rather than after 40s claiming success. Serialize the SOCKS5 lifecycle. .onControlQueue leaves these handlers on their own per-connection queues and FBWebServer only funnels the non-control routes, so a disconnect could return while an in-flight connect went on to start the tunnel, and two connects could race different proxy configurations through stop/save/reload/start. All three entry points now run on one serial lifecycle queue, which also gives stats a consistent snapshot. Honor the connect timeout across every wait. loadAllManagers:, waitUntilStopped: and the post-save reload each restarted a fixed 10s stage, and both spinners floored their budget at one second, so a one-second request could block for half a minute. Every wait now gets whatever is left of the caller's deadline, capped by its own stage limit, and refuses to start once the budget is gone. Match the VPN consent alert before tapping. An app-wide system.buttons[@"Allow"] query would select any other SpringBoard prompt that happened to be up, silently granting an unrelated permission. The button is now located inside a two-button alert, the way FBBroadcastManager anchors its own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build the engine from the normal Appium launch path. The tunnel target links a gitignored xcframework, so a fresh clone or npm install fails immediately with "There is no XCFramework found"; Scripts/build.sh only covers the CI and bundling flows. This cannot be fixed inside the Xcode project - xcodebuild resolves XCFramework references while computing the build graph, so neither a run-script phase on the target nor an aggregate target it depends on ever runs (both were tried and still failed). Build it from XcodeBuild.start instead, which is the path that was reported. Stop IPv6 from bypassing the tunnel. Only IPv4 routes were installed, so on a dual-stack device every AAAA-reachable destination kept using the real egress while the tunnel was up. The provider now claims IPv6 as well, with a /128 exclusion for an IPv6 proxy. hev has no IPv6 path, so such traffic is dropped rather than leaked - failing closed, at the cost of IPv6-only destinations being unreachable while connected. Verified that normal browsing still works, falling back to IPv4 through the tunnel. Identify the consent alert by more than its button count. Two buttons plus an "Allow" label also describes other system permission prompts, so the matcher could grant an unrelated permission. It now anchors on the alert's own text as well. Make the startup acknowledgement atomic. An engine exiting between the settle wait and the flag being set left the exit callback suppressing cancellation while startup reported success, advertising a connected tunnel over a dead engine. Both paths now decide under one lock. Let a disconnecting tunnel settle before reconfiguring it. An immediate retry after a timed-out connect, or a connect racing an external VPN stop, would rewrite and start the manager while its previous stop was in flight. Disable SIGPIPE on the preflight socket. A proxy closing the connection mid-handshake raised SIGPIPE on Darwin and terminated the extension: the engine's process-wide ignore has not been installed during the preflight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_static runs `make clean` in the shared submodule checkout, so two preparations racing from one checkout - two WDA sessions starting at once - could delete each other's archives mid-compile or while libtool was merging them, giving nondeterministic failures or a corrupt xcframework. Take a cross-process lock, and hold it across the stamp check as well as the build: the loser then re-reads the stamp the winner just wrote and exits early instead of rebuilding what already exists. The lock is a mkdir, which is atomic on every filesystem this runs on, and records its owner pid so a lock left behind by a killed build is reclaimed rather than blocking for the full timeout. Verified with two concurrent invocations (one builds, the other waits and then reports up-to-date) and with a stale lock naming a dead pid. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Returning from a timed-out saveToPreferences does not cancel it, so NetworkExtension could still persist the manager afterwards. A follow-up operation running before that landed would load no manager, build a second one with the same provider id, and leave two persisted configurations. ownManagerIn: returned whichever came first, so disconnect could pick the disconnected duplicate and report success while the real tunnel kept running - the VPN stayed up with nothing left pointing at it. Two changes, because either alone still leaves a hole: - Record the in-flight save and have the next locked connect/disconnect wait for it to settle before loading. Blocking here is safe from the lifecycle queue: the completion is delivered on the main queue, which that queue does not occupy. The wait is bounded by the caller's own deadline and clears the fence either way, so one stuck save cannot block every later call. The timed-out branch deliberately leaves it armed; every path past it has the completion in hand and disarms it. - Make ownManagerIn: prefer a manager that is not idle. Duplicates can still appear transiently, and a stale one must never shadow a running tunnel when disconnect asks which manager is ours. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Start the timeout before queueing. The deadline was created inside the locked body, so time spent waiting for another operation to release the lifecycle queue did not count: a connect with timeout:1 could sit behind a 30s connect and then be handed a fresh one-second budget. The wrapper now stamps the deadline before enqueueing and passes it in. Keep the mutation and its reported stats in one transaction. connect and disconnect released the lifecycle queue before the endpoint asked for stats separately, so an overlapping operation ran in between and a successful connect could answer connected:false, or a disconnect could answer with a newly connected tunnel. Both now return the snapshot taken while they still held the lock. Verified with overlapping requests: each response describes its own transaction. Mirror hev's authentication method selection in the preflight. hev_socks5_client_write_auth_methods always offers exactly one method - username/password only when BOTH fields are set, no-auth otherwise - while the probe offered both whenever a user was present. A no-auth-only proxy with credentials, or a user-without-password URI, therefore passed preflight and reported connected:true even though every real session is rejected. Bound the quit request itself. hev_socks5_tunnel_quit() can block waiting for event_fds[1] when the engine has not initialized yet or is exiting concurrently, and the semaphore timeout was only evaluated afterwards, so stopAndWait: did not actually bound that path and the NetworkExtension stop callback could hang forever. The quit is now issued off the caller's thread so the deadline covers the request as well as the exit. Let the stamp decide whether the engine needs rebuilding. Skipping the script whenever the xcframework directory merely existed meant a stale output - from an older checkout, a submodule bump, a script revision, or a failed partial build - was linked indefinitely. The script already exits cheaply when its SHA stamp is current. Reclaim stale build locks without deleting a live owner. Two builders could read the same dead pid and both rm -rf, so the slower one removed the lock the faster one had just acquired, putting both inside the shared make clean/build. Reclamation now claims the directory by rename - atomic, so exactly one builder wins - and re-confirms the pid before deleting. Verified with three concurrent builders against one stale lock: exactly one reclaimed it, one built, the others waited and saw the fresh stamp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both preference waits folded "the completion never arrived in time" and "the completion reported a failure" into FBSocks5TunnelManagerErrorInternal, which responseWithTunnelManagerError: surfaces as an unknown error. A plain deadline miss therefore did not produce the documented timeout response, and the message read "Cannot load the VPN preferences: timed out" - an internal fault describing a timeout. Split the two on the initial load and on the post-save reload: no completion means FBSocks5TunnelManagerErrorTimeout, and the internal classification is reserved for an actual error handed back by NetworkExtension. Verified on device: a connect with timeout:0.2 now answers timeout / "Timed out reloading the saved VPN configuration" where it previously answered unknown error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The WebDriverAgentTunnel appex was a hard target dependency of WebDriverAgentRunner, so every runner build - plain xcodebuild, the Appium driver, build.sh, CI - required the hev-socks5-tunnel submodule and the engine xcframework, and every device build had to sign the appex. That blocked merging to master for users without an Apple developer subscription. The tunnel is now opt-in via dedicated schemes: - Drop the runner -> tunnel PBXTargetDependency; the default WebDriverAgentRunner(-nodebug) schemes build no appex, need neither the submodule nor the engine, and no longer carry the embed post-action (so a stale appex in shared DerivedData can never sneak into a default build). - Add WebDriverAgentRunnerTunnel(-nodebug) schemes that build the appex alongside the runner and keep the embed post-action. - build.sh: TARGET=tunnel_runner selects the tunnel scheme and is the only target that prepares the hev engine. - xcodebuild.ts: drop prepareSocks5Engine - the driver always builds the default scheme, which no longer links the engine. - connect now fails fast with 'unsupported operation' when the appex is not embedded in the host app, instead of a confusing NEVPN error (unit-tested via FBSocks5TunnelManager isTunnelExtensionEmbeddedInBundle:). - CI: submodule checkout only where still needed (wda-tests build job, npm publish packing the vendored engine sources); new iOS_Tunnel_Build jobs keep the appex + engine + embed path covered. Verified: fresh-DerivedData default-scheme build succeeds with the xcframework absent and embeds no tunnel appex; the tunnel scheme embeds it with the rewritten bundle id; UnitTests pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stats deadline)
- Abort provider startup after a concurrent stop: stopTunnelWithReason:
and the startup continuation now transition {isStopping, runner}
under one lock, and the runner is started inside the section that
publishes it. A stop that lands while the probe or
setTunnelNetworkSettings is pending previously found no runner,
completed as a no-op, and the continuation then started a late hev
engine in a provider the system considered torn down.
- Try every resolved proxy address before failing: the pre-flight now
probes all A/AAAA records in order (IPv4 first, deduplicated) and
keeps the first that completes the SOCKS5 handshake. Unreachable
endpoints move on to the next record; protocol/credential rejections
still fail immediately since they come from the server itself. The
probe's connect() is now bounded by the 8s probe deadline via a
non-blocking connect + poll (SO_SNDTIMEO never covered connect, so a
filtered endpoint could previously hang the start for the ~75s SYN
retry window - per address, once several are tried).
- Keep the connect success snapshot within the caller's timeout: the
stats round trip to the extension is now capped at the remaining
connect deadline (and skipped when exhausted, falling back to zero
counters) instead of a fixed extra 3s.
Two further findings from the same round were already resolved by the
opt-in change: the release workflows no longer need the hev
xcframework (they build the default tunnel-free schemes), and
prepareSocks5Engine no longer exists in lib/xcodebuild.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-wda # Conflicts: # WebDriverAgentLib/Routing/FBWebServer.m
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WebDriverAgentTunnelpacket-tunnel extension backed by vendoredhev-socks5-tunnel/mobilerun/socks5/connect,/disconnect, and/statswith credential redaction and strict URI/timeout validationValidation
WebDriverAgentRunnerTunnelgeneric iOS device build withCODE_SIGNING_ALLOWED=NOWebDriverAgentRunnerTunnelgeneric iOS simulator buildWebDriverAgentRunnerTunnelgeneric iOS build withWDA_PRODUCT_BUNDLE_IDENTIFIER=io.appium.customwdaWebDriverAgentLib_watchOSgeneric watchOS buildnpm run buildnpm test— 53 passingnpm run lintgit diff --checkValidation limits
CODE_SIGNING_ALLOWED=NO