Skip to content

bug(testing): imageAttachment corpus test fails locally once a cited Claude session is rotated #684

Description

@Juliusolsson05

Found while working on #681. Unrelated to that change — reproduces on a clean main.

Expected

npm test passes on the author's machine, and core-tier *.test.ts files do not depend on the developer's home directory.

Actual

FAIL |unit| src/providers/shared/renderer/protocols/media/imageAttachment.test.ts
  > corpus-wide invariants > every fixture is traceable to a census row and a real session

AssertionError: atp-codex-image-inside-claude-transcript cites a missing session:
/Users/<user>/.claude/projects/-Users-<user>-Desktop-Development-klay/c0c60d3d-….jsonl

The project directory still exists; that individual session JSONL does not. Claude Code rotates and removes session transcripts as a matter of course, so this is expected local behavior, not corruption.

Mechanism

imageAttachment.test.ts:296-301 resolves CORPUS_ROOTS from homedir() and asserts existsSync on the absolute source path each fixture records:

const CORPUS_ROOTS = {
  claude: join(homedir(), '.claude', 'projects'),
  codex: join(homedir(), '.codex', 'sessions'),
}

if (existsSync(CORPUS_ROOTS.claude) || existsSync(CORPUS_ROOTS.codex)) {
  const rooted = path.startsWith(CORPUS_ROOTS.claude) || path.startsWith(CORPUS_ROOTS.codex)
  if (rooted) {
    expect(existsSync(path), `${f.$fixture.id} cites a missing session: ${path}`).toBe(true)
  }
}

The outer guard means CI never runs the assertion — no corpus roots there, and the baked-in absolute paths would not match a CI homedir anyway. The failure is therefore invisible in CI and permanent on the machine the fixtures were recorded on, which is the worst distribution: the person most likely to run the suite is the only one who sees it, and it never blocks a merge.

Why this is worth an issue rather than a quick patch

docs/testing/standard.md is explicit:

npm test MUST be safe on a clean machine with no credentials, no personal configuration, and no network access. Loading an ambient .env, reading a developer's home directory, or launching an installed provider is live behavior and requires an explicit opt-in variable in addition to the test:live command.

This assertion is live behavior inside a deterministic-tier file. The intent behind it is good — the comment says it wants to "give an opinion where the evidence is available, and say nothing where it is not" — but the evidence it reaches for is mutable state outside the repository, so "available" is not a stable property.

Impact

Low severity, ongoing friction: a permanently red suite on the author's machine trains people to ignore a red suite, which is the expensive part.

Suggested direction

Either move the traceability assertion behind the test:live opt-in, or drop the existsSync check and keep verifying only what the fixture itself carries (census row, fingerprint, recorded payload size) — which is self-contained and cannot rot.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions