Skip to content

test: assert the specific error code instead of any failure (#1781 B4) - #1790

Open
thymikee wants to merge 1 commit into
mainfrom
test/1781-b4-assert-right-reason
Open

test: assert the specific error code instead of any failure (#1781 B4)#1790
thymikee wants to merge 1 commit into
mainfrom
test/1781-b4-assert-right-reason

Conversation

@thymikee

@thymikee thymikee commented Aug 17, 2026

Copy link
Copy Markdown
Member

Summary

Umbrella #1781, item B4 "assert the right reason". The repo's typed-error rule: failures are AppError with a code from the closed KNOWN_APP_ERROR_CODES set (packages/kernel/src/errors.ts). This PR converts the 20 test assertions across the repo that instead accepted ANY failure — bare expect(...).toThrow(), bare assert.throws(fn), bare assert.rejects(p) — into assertions on the specific code (or, where the propagated error is genuinely opaque, an identity assertion with a comment explaining why).

Re-derived the site list with real paren matching (multi-line matchers excluded, .not.toThrow() excluded) rather than trusting the hand-picked examples in the issue; the exact count came out to 20, not 21 (one of the originally-cited examples turned out to already assert a specific matcher).

Added a synchronous assertThrowsAppError(fn, {code, message?}) sibling to the existing assertRejectsAppError helper in src/__tests__/test-utils/app-error.ts, exported via the test-utils index. packages/provider-limrun and packages/provider-webdriver have no test-utils dir and cannot import from src/, so those sites use vitest's expect(...).toThrow(expect.objectContaining({ code })) or an inline assert.rejects(p, matcherFn) instead — no cross-package imports, no new re-export surface.

Sites converted

  • packages/provider-limrun/src/app-log-runtime.test.ts:153-155 — bare .toThrow() x3 → UNSUPPORTED_OPERATION
  • src/daemon/__tests__/app-log.test.ts:39 — bare .toThrow() → message match (plain Error, not AppError, from verified-file's identity check; message differs by operation)
  • src/daemon/__tests__/resumable-upload-range.test.ts:13 — bare assert.throws(fn)INVALID_ARGS. Also fixed line 19's assert.throws(fn, value): a documented Node.js gotcha where a string second argument is the assertion failure message, not a matcher, so it was equally bare in effect.
  • packages/provider-webdriver/src/webdriver-client.test.ts:229 — bare assert.rejects(p) → asserts the raw AbortSignal.timeout() rejection's name, since the transport re-throws it unwrapped (not an AppError)
  • src/daemon/handlers/__tests__/session-device-claims.test.ts:129,151,174 — bare assert.rejects(p) x3 → identity assertions; each test's point is device-claim rollback/retention behavior around an opaque mocked upstream failure, not any particular error shape
  • src/platforms/android/__tests__/settings.test.ts:109 — bare assert.rejects(p)UNSUPPORTED_OPERATION
  • src/platforms/android/__tests__/snapshot.test.ts:1071,1342 — bare assert.rejects(p) x2 → COMMAND_FAILED + message
  • src/platforms/android/__tests__/touch-helper-session.test.ts:526 — bare assert.rejects(p)COMMAND_FAILED, wrong-protocol message
  • src/platforms/apple/core/__tests__/runner-command-retry.test.ts:472,527,550,762,881,1016 — bare assert.rejects(p) x6 → COMMAND_FAILED with the recovery-path-specific details/message
  • src/platforms/apple/core/__tests__/runner-transport.test.ts:61 — bare assert.rejects(p) → identity assertion; fetchWithTimeout does not wrap fetch() failures into an AppError

No repo-wide scanner/lint rule added (explicitly out of scope per #1781); no test loosened.

Follow-up surfaced

src/utils/app-log-files.ts:43 and src/utils/verified-file.ts:35,110 throw plain Error instead of AppError, which is why app-log.test.ts:39 had to become a message match instead of a code assertion. This is a typed-error-rule gap in product code, out of scope for this PR. Filing as its own issue, referencing #1781 B4.

Test plan

  • npx vitest run on all 10 touched test files — 153 tests passed
  • pnpm run typecheck — clean
  • npx oxlint --deny-warnings on touched files — clean

Converts the 20 test assertions across the repo that accepted ANY
failure (bare `expect(...).toThrow()`, bare `assert.throws(fn)`, bare
`assert.rejects(p)`) into assertions on the specific AppError `code`
each test is actually about, or — where the propagated error is
genuinely opaque (a mocked upstream failure whose identity, not its
shape, is the point) — identity assertions with a comment explaining
why.

Added a synchronous `assertThrowsAppError(fn, {code, message?})`
sibling to the existing `assertRejectsAppError` helper in
src/__tests__/test-utils/app-error.ts, exported via the test-utils
index, for the two src/ sites that needed it.
packages/provider-limrun and packages/provider-webdriver have no
test-utils dir and cannot import from src/, so those sites use
vitest's `expect(...).toThrow(expect.objectContaining({ code }))` or
an inline `assert.rejects(p, matcherFn)` instead.

Sites converted:
- packages/provider-limrun/src/app-log-runtime.test.ts:153-155
  (bare `.toThrow()` x3 -> `UNSUPPORTED_OPERATION`)
- src/daemon/__tests__/app-log.test.ts:39 (bare `.toThrow()` ->
  message match; plain Error, not AppError, from verified-file's
  identity check)
- src/daemon/__tests__/resumable-upload-range.test.ts:13 (bare
  `assert.throws(fn)` -> `INVALID_ARGS`); also fixed line 19's
  `assert.throws(fn, value)`, a documented Node.js gotcha where a
  string second argument is the failure message, not a matcher, so
  it was equally bare in effect
- packages/provider-webdriver/src/webdriver-client.test.ts:229 (bare
  `assert.rejects(p)` -> asserts the raw AbortSignal.timeout()
  rejection's `name`, since the transport re-throws it unwrapped)
- src/daemon/handlers/__tests__/session-device-claims.test.ts:129,
  151, 174 (bare `assert.rejects(p)` x3 -> identity assertions; each
  test's point is device-claim rollback/retention around an opaque
  mocked upstream failure)
- src/platforms/android/__tests__/settings.test.ts:109 (bare
  `assert.rejects(p)` -> `UNSUPPORTED_OPERATION`)
- src/platforms/android/__tests__/snapshot.test.ts:1071, 1342 (bare
  `assert.rejects(p)` x2 -> `COMMAND_FAILED` + message)
- src/platforms/android/__tests__/touch-helper-session.test.ts:526
  (bare `assert.rejects(p)` -> `COMMAND_FAILED`, wrong-protocol
  message)
- src/platforms/apple/core/__tests__/runner-command-retry.test.ts:472,
  527, 550, 762, 881, 1016 (bare `assert.rejects(p)` x6 ->
  `COMMAND_FAILED` with the recovery-path-specific details/message)
- src/platforms/apple/core/__tests__/runner-transport.test.ts:61
  (bare `assert.rejects(p)` -> identity assertion; fetchWithTimeout
  does not wrap fetch() failures into an AppError)

No repo-wide scanner/lint rule added (explicitly out of scope per
#1781); no test loosened.
@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 2.26 MB 2.26 MB -1.1 kB
JS gzip 744.4 kB 744.0 kB -490 B
npm tarball 863.7 kB 862.7 kB -906 B
npm unpacked 3.01 MB 3.01 MB -2.6 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.1 ms 28.0 ms -0.1 ms
CLI --help 67.4 ms 66.4 ms -1.0 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/prepare-kind.js -891 B -306 B
dist/src/selector-runtime.js +245 B +72 B
dist/src/interaction2.js +15 B 0 B

@thymikee

Copy link
Copy Markdown
Member Author

Request changes before merge:

  • The PR body says 20 matcherless assertions, but its own per-site list totals 21 (4 toThrow, 2 assert.throws, 15 assert.rejects). Please correct the count and scope wording.
  • B4 is specifically making failure-reason assertions load-bearing. docs/agents/testing.md requires red evidence for a regression pin, but the PR/discussion only show green runs. Deliberately mutate a representative expected code and an identity exemption (or otherwise make each matcher class reject its wrong failure), run the affected tests, and quote the red results in the PR; then restore and report green.

The changed assertions themselves appear honest on review. CI is green for the relevant test gates; the only failure is CodeQL Java/Kotlin failing while GitHub returned 503 during SARIF upload, which is infrastructure-unrelated to this TypeScript-only test change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant