test: measure oracle liveness suite-wide; pin the one dead-path oracle - #1679
Merged
Conversation
Size Report
Startup median (7 runs, lower is better):
Top changed chunks: no changes in the largest emitted chunks. |
Member
Author
|
Reviewed exact head |
thymikee
added
the
ready-for-human
Member
Author
|
CI handoff note on head Generated by Claude Code |
Member
Author
|
CI classification for run 31212551654: infrastructure/aborted shard, not a code failure. The kernel-errors job began its mutation step but was terminated without a completed step, log, failure envelope, or report artifact; all nine sibling shards passed. The aggregate correctly rejected the incomplete 9/10 shard set rather than reporting a mutation regression. Re-run the missing shard/workflow to restore coverage; this does not change the code-review readiness verdict. |
- docs/agents/oracle-negation-spike.md: assertion-negation sweep over 677 test files (5,687 verdicts). Zero vacuous tests: all 150 negation survivors decompose into assert.rejects-validator artifacts (112), helper-oracles (31), in-file-fake breakage (6), and one conditional oracle. Records the companion mock-coupled coverage-uniqueness numbers and the follow-ups they motivate (diff-scoped mutation gate, provider seam closures, transcript provenance). - watchos-sentinel: the non-watchOS test's only assertion sat in a catch block that never fires (tvOS interactor creation succeeds), so no assertion executed on the observed path. Pin creation success instead. Red-run proof: the old shape survived the negation sweep; the new shape fails under it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
…tubs Adds withFakeAdb to test-utils: a scripted in-process AndroidAdbProvider installed through the production withAndroidAdbProvider seam — the same scope the daemon installs per request — replacing PATH-stub shell scripts that spawn a real subprocess per adb call. No PATH mutation, no spawns, no real subprocess waits. Converts settings.test.ts (15 tests, 23ms; waiver said "waits real settings-apply poll time") and notifications.test.ts (2 tests, 9ms). Assertions move from args-log regex greps to structural checks on the recorded call list; the fake receives device-scoped args with the -s serial pair stripped, so serial routing is enforced by the scoped provider matching device.id instead of asserted per call. Remaining PATH-stub files convert next; their contention-retry waiver entries lift together with the conversions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
10 PATH-stub cases move to withFakeAdb through the production provider scope; the 2 tests that already inject an executor directly are unchanged. Cross-invocation shell STATE_FILE state becomes a closure boolean; args-log regex asserts become structural checks on recorded calls. 12/12 green at 386ms — the residue is dismissAndroidKeyboard's two fixed 120ms retry sleeps, not stub subprocess waits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
…vider The adb half of every case moves to withFakeAdb through the production provider scope; installs take the documented exec-shaped fallback (exec(['install','-r',...])), matching what the PATH stub saw minus the serial pair. bundletool/zip/unzip stay real or PATH-stubbed — they run via runCmd outside the adb seam, so this file remains in the serialized subprocess-stub lane with its waiver reason to be corrected from adb to bundletool. 13/13 green at ~130ms; no case enters a retry/poll loop. Conversion note: manifest identity's `unzip -p` failure is silently swallowed (readZipEntry catch -> undefined, aapt fallback) — an invisible degradation path worth a future explicit diagnostic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
9 PATH-stub cases move to withFakeAdb; the 3 tests already injecting providers directly are unchanged. Chunked shell-input assertions become ordered deepEqual on the recorded calls; never-called negatives and call-count checks preserved 1:1. 12/12 green. File time drops to 2.2s, all of it production sleeps: verifyAndroidFilledText unconditionally waits its [0,150,350]ms verification cadence even when the first inspection matches, so each fill verification pass costs ~500ms with an instant fake. A budget-derived cadence there (testing.md pattern 1) would put this file near 25ms; flagged as follow-up rather than changed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
Three test-utils extractions applied across the six converted files:
- assertRejectsAppError collapses the hand-rolled AppError code+message
rejection validator (10 sites here; ~30 more repo-wide can adopt it
incrementally). Validators asserting details or multiple differently-
flagged regexes stay explicit on purpose.
- withFakeAdb gains a `provider` option for extra capabilities
(snapshotHelperArtifact, reverse, ...), replacing input-actions'
nested re-scoping bridge.
- withFakeAdb now mirrors the local executor's contract: a scripted
nonzero exit throws androidAdbResultError unless the call site passed
allowFailure. Provider-scoped exec bypasses exec.ts's throw-on-close-
failure, so returning {exitCode:1} took a different production path
than the PATH-stub `exit 1` these fakes replaced. All 75 tests hold
under the corrected semantics.
Also swaps settings' inline emulator DeviceInfo literals for the shared
ANDROID_EMULATOR fixture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
settings, notifications, device-input-state, input-actions, and app-lifecycle-open no longer stub binaries on PATH or spawn subprocesses, so their contention mechanism is gone: they leave CONTENTION_RETRY_FILES and, through the derived SUBPROCESS_STUB_TESTS constant, the serialized subprocess-stub project (17 -> 12 files). app-lifecycle-install stays with its reason corrected: adb is now in-process, but bundletool stays PATH-stubbed and zip/unzip spawn for .aab packaging paths. Full unit suite green at the new membership: 638 files, 5,724 tests, with the five files running at unit-core's default parallelism. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
- app-lifecycle-open: the missing-package launch failure returns
{stderr, exitCode: 1} and lets withFakeAdb's throw path produce the
production-shaped androidAdbResultError instead of hand-modeling the
thrown AppError — the drift the helper exists to eliminate.
- withScriptedAdb deleted: the six converted files were its only
callers, and a live PATH-stub export invites new tests back into the
serialized lane this batch shrank. withMockedAdb stays (dispatch and
runtime-hints tests still stub other binaries).
- android-snapshot-helper gains androidSnapshotHelperScriptResponse so
the version-probe detection and versionCode reply have one source of
truth; input-actions' local copy delegates to it.
- withFakeAdb's provider option becomes a distributed Omit over the
AndroidAdbProvider union, so touch without gestureViewport is a
compile error at the fake's boundary (planted and verified) instead
of a TypeError inside production gesture planning.
- spike-doc re-run checklist restores wider than the codemod globs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
Fallow's dead-code gate flagged it: scripts are always passed as inline lambdas, so only FakeAdbResponse needs a name at the barrel. The type stays exported from fake-adb.ts where the withFakeAdb signature uses it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
…ATH stubs withFakeAppleTool mirrors withFakeAdb for the Apple seam: a scripted provider installed via the production withAppleToolProvider scope, flat simctl/devicectl invocations recorded exactly as the PATH-stub shell scripts saw them, throw-on-nonzero fidelity matching exec.ts unless the call site passed allowFailure, and the canned `simctl privacy help` listing served by default (the block withMockedXcrun injected into every script). screenshot-status-bar.test.ts converts as the exemplar: 3/3 green at 9ms with deepEqual call-sequence assertions replacing the args-log regexes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
All withMockedXcrun scripts and hand-rolled PATH stubs move to
withFakeAppleTool; args-log regexes become structural call assertions
(exact deepEqual where order is deterministic, presence checks where
the 5s simulatorBootedMemo TTL makes boot-probe order test-dependent).
12 hand-rolled AppError validators collapse into assertRejectsAppError.
54/54 green; file test time 1172ms -> ~400ms with no test over 201ms.
Five .ipa install tests keep a minimal PATH stub for unzip only:
install-artifact.ts:112 and install-source.ts:438 call runCmd('unzip')
directly, outside the Apple tool provider seam — the file therefore
stays in the serialized subprocess-stub lane with its waiver reason
corrected from xcrun to unzip.
Also observed: getSimctlPrivacyServices caches per PATH+simulatorSetPath
and simulatorBootedMemo keys on deviceId|setPath, so neither cache
accounts for the provider scope — worked around per test, follow-up
worthy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
- interactions, simulator, screenshot, physical-device-screenshot,
devicectl, and screenshot-status-bar leave CONTENTION_RETRY_FILES:
the first five stopped stubbing PATH binaries in earlier refactors
(measured 3-64ms per file, no subprocess activity), and
screenshot-status-bar now injects through the fake tool provider.
apps.test.ts stays with its reason corrected to the unzip PATH stub
(xcrun is in-process; install-artifact.ts:112 / install-source.ts:438
call runCmd('unzip') outside the Apple seam). Serialized lane 12 -> 6.
- oxfmt: fake-apple-tool.ts and contention-retry.ts were pushed
unformatted (local check piped through tail masked the failure).
- fallow complexity: the three fake-script arrows in apps.test.ts drop
under threshold via shared predicates (isSimctlMainScreenScale,
isSimctlScreenshot, isDevicectlDevice), which also deduplicate the
screenshot pair.
Full unit suite green at the new membership: 638 files, 5,724 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf
thymikee
force-pushed
the
claude/unit-test-assessment-mocking-79uihi
branch
from
August 8, 2026 06:07
dcd0c9c to
ff1aaf1
Compare
Member
Author
|
Re-reviewed exact head ff1aaf1: code review clean; remains ready for human review. Range-diff versus the previously reviewed head is patch-equivalent. The fake ADB/Apple tools use the production provider scopes, preserve nonzero/allowFailure behavior, retain direct unzip PATH-stub serialization, and replace shell-log checks with structural call/result assertions. The watchOS sentinel now executes an unconditional non-watchOS oracle. Test-infrastructure-only, so no live-device evidence is required. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test files (5,687 verdicts). Zero vacuous tests: all 150 negation
survivors decompose into assert.rejects-validator artifacts (112),
helper-oracles (31), in-file-fake breakage (6), and one conditional
oracle. Records the companion mock-coupled coverage-uniqueness numbers
and the follow-ups they motivate (diff-scoped mutation gate, provider
seam closures, transcript provenance).
block that never fires (tvOS interactor creation succeeds), so no
assertion executed on the observed path. Pin creation success instead.
Red-run proof: the old shape survived the negation sweep; the new shape
fails under it.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019S5sZnmPn4A9Ct7sTJdfAf