test(ai-action): prove packed CLI and Action runtime

[kunaldhongade]

Summary

• install the real @submuxhq/codedecay@0.3.5 tarball into an independent npm project and execute its dist/index.js outside the workspace
• cover packed ai and ai preflight behavior, Codex plus alternate profiles, refs, cwd/output, all task filters, invalid input, check states, fail gates, and a deterministic repair loop
• execute the literal composite Action shell blocks from action.yml with a recording fake CLI instead of relying on string-presence assertions
• prove exact AI argv forwarding, mode-specific exclusions, missing CLI diagnostics, final exit propagation, and best-effort summary/comment behavior
• add no production code and no hosted runner dependency

OSS-first decision

act is not installed in the contributor environment and would add a Docker/runtime dependency for shell logic that can be executed directly. The focused harness parses the maintained action.yml, substitutes declared inputs, and runs the exact Bash step with GitHub-compatible environment variables. It does not copy the argument-building logic.

Test design

Packed CLI

• npm pack plus fresh external npm install
• no workspace:, link:, or file: dependency protocols
• isolated real Git repositories and explicit base/head commits
• passing, failing, skipped, and safety-blocked configured checks
• persisted bundle before verification and risk-gate exits
• real Node test after a deterministic missing-test repair, followed by packed CLI re-analysis

GitHub Action

• exact argv and cwd recording from the real Run CodeDecay Bash block
• AI profile/refs/cwd/output/task/requirements/check and fail-gate forwarding
• unsupported flag exclusion for analyze, redteam, and agent modes
• CLI exit 17 propagation from agent mode
• summary and PR-comment rendering remain best effort on the same failure
• unsupported mode, unsupported format, and missing CLI diagnostics

A/B fault injection

Temporary production mutants were applied one at a time and restored before commit:

• removed AI --with-checks forwarding: exact argv test failed
• swallowed the final CLI exit status with || true: failure propagation test changed from exit 17 to 0 and failed
• changed packed AI default profile from Codex to generic: installed-tarball test failed

Validation

• focused adjacent matrix: 6 files, 40/40 tests passed
• full suite: 122 files, 641/641 tests passed
• pnpm run lint
• pnpm typecheck
• pnpm build, including all packages and VitePress docs
• pnpm --filter @submuxhq/codedecay pack --dry-run
• pnpm test:child-repo-e2e: 36/36 assertions passed, including tarball install, Chromium, MCP, Action simulation, differential checks, and agent convergence
• pnpm demo:end-user: passed
• pnpm eval:pr-safety: 2/2 scenarios passed
• CodeDecay self-redteam: merge risk 15, security 0, no missing-test findings; only findings are expected test-only ratio heuristics

Closes #667

[github-actions]

CodeDecay PR Check

Lead catch: No direct high-signal catch found

CodeDecay did not find a strong direct regression or weak-test signal in this run.

Risk: Medium · Merge 16/100 · Decay 41/100 · Security 0/100

This score is conservative pending stronger direct evidence. Treat structural-only signals as review guidance, not proof of a regression.

Full CodeDecay report

CodeDecay Report

Overall risk: Medium

Score	Value
Merge risk	16/100
Decay risk	41/100
Security risk	0/100

Findings	Count
High	2
Medium	2
Low	4

Changed Files

• packages/cli/test/built-cli-ai.test.ts added (+435/-0)
• packages/github-action/test/action-runtime.test.ts added (+253/-0)
• packages/github-action/test/helpers/action-runtime.ts added (+104/-0)

Likely Impacted Areas

• Low Tests (test): packages/cli/test/built-cli-ai.test.ts, packages/github-action/test/action-runtime.test.ts, packages/github-action/test/helpers/action-runtime.ts

Language And Parser Coverage

• Source files classified: 3
• Fully supported parser files: 3
• Limited files: 0
• Unsupported files: 0

Merge Risk Breakdown

• Score: 16/100
• Raw score before dampeners: 21/100
• Adjusted score before severity cap: 16/100
• Highest contributing severity: Low
• Evidence mode: heuristic-only

Top contributors:

• +9 Change size (structural): Changed lines amplify review cost across 3 file(s).
• +4 Test area changed (heuristic): packages/cli/test/built-cli-ai.test.ts touches a test area and should be reviewed for regression impact.
• +4 Test area changed (heuristic): packages/github-action/test/action-runtime.test.ts touches a test area and should be reviewed for regression impact.
• +4 Test area changed (heuristic): packages/github-action/test/helpers/action-runtime.ts touches a test area and should be reviewed for regression impact.
• +0 Architecture note applies (memory-context): Untrusted architecture context: CLI is the published surface: The public npm package is @submuxhq/codedecay and the binary is codedecay. Internal workspace packages are implementation details.

Dampeners:

• -5 Heuristic-only dampener: Merge risk stays conservative until direct evidence exists.

Notes:

• Heuristic-only merge risk is capped at 54/100 until direct evidence exists.
• Untrusted memory context is visible but contributes 0 score until trusted evidence corroborates it.

Decay Risk Breakdown

• Score: 41/100
• Raw score before dampeners: 55/100
• Adjusted score before severity cap: 41/100
• Highest contributing severity: High
• Evidence mode: heuristic-only

Top contributors:

• +18 Large test change relative to source change (heuristic): packages/cli/test/built-cli-ai.test.ts adds 435 lines of tests for 0 source additions.
• +18 Large test change relative to source change (heuristic): packages/github-action/test/action-runtime.test.ts adds 253 lines of tests for 0 source additions.
• +10 Large test change relative to source change (heuristic): packages/github-action/test/helpers/action-runtime.ts adds 104 lines of tests for 0 source additions.
• +9 Change size (structural): Changed lines amplify review cost across 3 file(s).

Dampeners:

• -14 Heuristic-only dampener: Decay stays conservative until direct evidence exists.

Notes:

• Heuristic-only decay is capped at 54/100 until direct evidence exists.

Security Risk Breakdown

• Score: 0/100
• Raw score before dampeners: 0/100
• Adjusted score before severity cap: 0/100

Security Matcher Coverage

• Changed source files scanned: 0
• Security candidates found: 0
• Skipped files: 0

Test Evidence

• Mode: heuristic-only
• Sources: none
• Notes:
• No runtime coverage artifact was found. Test audit remains heuristic-only.

Untrusted Memory Context

• Project invariant may be impacted (packages/cli/test/built-cli-ai.test.ts:1): Untrusted memory context: invariant "Output must be actionable" applies to this change. Redteam reports and agent bundles should say what behavior to verify, which test proof is weak or missing, and what task a coding agent should perform.
• Architecture note applies (packages/cli/test/built-cli-ai.test.ts:1): Untrusted architecture context: CLI is the published surface: The public npm package is @submuxhq/codedecay and the binary is codedecay. Internal workspace packages are implementation details.

High Risk Findings

• Large test change relative to source change (packages/cli/test/built-cli-ai.test.ts:1): packages/cli/test/built-cli-ai.test.ts adds 435 lines of tests for 0 source additions.
• Large test change relative to source change (packages/github-action/test/action-runtime.test.ts:1): packages/github-action/test/action-runtime.test.ts adds 253 lines of tests for 0 source additions.

Medium Risk Findings

• Large test change relative to source change (packages/github-action/test/helpers/action-runtime.ts:1): packages/github-action/test/helpers/action-runtime.ts adds 104 lines of tests for 0 source additions.

Low Risk Findings

• Test area changed (packages/cli/test/built-cli-ai.test.ts:1): packages/cli/test/built-cli-ai.test.ts touches a test area and should be reviewed for regression impact.
• Test area changed (packages/github-action/test/action-runtime.test.ts:1): packages/github-action/test/action-runtime.test.ts touches a test area and should be reviewed for regression impact.
• Test area changed (packages/github-action/test/helpers/action-runtime.ts:1): packages/github-action/test/helpers/action-runtime.ts touches a test area and should be reviewed for regression impact.

Recommended Checks

• Flow check (CLI release smoke): Run built CLI smoke tests
• Flow check (CLI release smoke): Run package dry-run
• Flow check (CLI release smoke): Run published-package or tarball demo before release
• Flow check (GitHub Action dogfood): Check cwd/output/fail-on behavior
• Flow check (GitHub Action dogfood): Confirm CodeDecay dogfood check runs on the PR
• Flow check (GitHub Action dogfood): Verify action inputs match docs
• Flow check (Pull request redteam review): Check weak or missing test proof
• Flow check (Pull request redteam review): Keep deterministic evidence separate from AI suggestions
• Flow check (Pull request redteam review): Review edge cases and agent fix tasks
• Flow check (Pull request redteam review): Run codedecay redteam against the PR diff
• Run project command: Full validation (pnpm install && pnpm run lint && pnpm typecheck && pnpm test && pnpm build)
• Run project command: Package dry-run (pnpm --filter @submuxhq/codedecay pack --dry-run)

Notes

CodeDecay is deterministic and local-first. This report was generated without telemetry, API keys, LLMs, or model calls.

---

Found by CodeDecay - deterministic, local-first, no telemetry.