AI Scan — an AI-assisted security review for a GitHub repository

A scanner tells you what it found. A review tells you what to do.

Paste a public GitHub repository URL. The service clones it, finds real vulnerabilities, secrets, and configuration risks, ranks them by how much they actually matter in that repository, and returns a review: a verdict, the handful of things that matter, an ordered list of what to do next, and — the part nobody ships — an explicit list of what you can safely ignore, with reasons.

---

Quick start

pnpm install
cp .env.example .env     # optional — the app runs with no configuration at all
pnpm dev                 # http://127.0.0.1:3200

Open http://127.0.0.1:3200, paste a repo, watch it work.

pnpm test        # 64 tests
pnpm typecheck
pnpm build && pnpm start

No API key is required. With ANTHROPIC_API_KEY set, the review is written by Claude. Without it, a deterministic rules engine writes it instead. The result screen always tells you which one you got. See The AI layer.

Two repositories worth trying:

Repository	What it demonstrates
OWASP/NodeGoat	272 findings collapsed into 5 actions — grouping, prioritisation, deferral
tj/commander.js	0 findings, grade A — the clean-bill path

Scans take about 10s on the rules engine and about 50s with Claude, nearly all of it the single triage call.

---

The product decision

The customer said they didn't want a raw scanner report. Taking that seriously is the whole assignment, so I started from what a raw report actually fails to answer.

A scanner gives you facts:

CVE-2024-45590  body-parser  1.18.3  DoS       high      direct, runtime
GHSA-8cb4       form-data    2.1.4   weak RNG  critical  transitive
GHSA-8cb4       form-data    2.0.0   weak RNG  critical  transitive
… × 272

That list is true and nearly useless. It sorts by a number that describes the CVE in the abstract, not by anything about your codebase. So the product answers three questions instead:

1. What matters? — Severity is a property of a vulnerability. Risk is a property of your repository. A critical in a package that is installed but never imported ranks below a high in the HTTP client on your request path. The scan proves this on NodeGoat: body-parser (high, imported, direct, runtime) scores 84 and outranks five critical findings scoring 60, because nothing in the repository imports those packages.

2. What do I do? — Findings are collapsed into work. One form-data upgrade closes ten advisories, so it is one action with a copy-pasteable command — not ten rows demanding ten decisions.

3. What can I ignore? — Of NodeGoat's 272 findings, 232 land in "Safe to defer", split by reason: development-only dependencies that never ship, packages the source never imports, and everything scoring below the action threshold. Telling someone what not to work on is the difference between a tool that adds work and one that removes it.

The result screen leads with a verdict and an action list. The raw findings table still exists — behind a toggle, where it belongs.

Nothing is allowed to fall through the cracks. nextActions and deferred form a partition: every finding is either something you act on or something consciously set aside, and deferUnreferenced in src/triage/index.ts enforces that in code after triage runs. topRisks deliberately overlaps both — it's a highlight layer, not a third bucket. That distinction matters more than it sounds; see How I used coding agents.

---

The risk model

riskScore (0–100) is a pure function in src/scoring/score.ts. No clock, no randomness, no network — the same findings always produce the same ranking.

Severity sets the baseline; context moves it:

Signal	Effect	Reasoning
Base: critical / high / medium / low	70 / 55 / 35 / 18	Starting point only
Development-only dependency	−22	Doesn't ship to production (still runs in CI, so a discount, not a dismissal)
Declared directly in package.json	+8	You control the version — you can actually fix it
Transitive dependency	−4	Fixing it means moving something else
Imported by this repo's source	+15	The vulnerable code is genuinely reachable
Installed but never imported	−12	Present in the tree, but nothing here calls it
A fixed version exists	+6	Best effort-to-benefit ratio available
No published fix	−5	Nothing to act on today
Committed credential	+20	Exploitable the moment anyone reads the repo
Credential in an .example file	−30	Almost certainly illustrative

Every finding carries riskFactors — the same reasoning in plain language — which feeds both the UI ("why that score") and the model prompt.

The posture grade is driven by the worst contextual risk, not the finding count. Fifty low-risk notices are not worse than one live credential, and a grading scheme that says otherwise trains people to ignore the grade.

---

The AI layer, and why the model is not allowed to do arithmetic

The split is deliberate:

scanners ──▶ deterministic scoring ──▶ ranked findings ──▶ triage ──▶ review
             (pure, unit-tested)                            (Claude
                                                          │ or rules)

Code computes the ranking. The model explains it. A security tool that returns a different answer when you re-run it is not a security tool, so prioritisation lives in tested code. What the model does well — reading a pile of advisories, spotting that eight of them share one root cause, and writing two sentences a tired engineer can act on — is what it's asked to do.

Concretely:

• The prompt sends scored findings and forbids inventing a finding, a CVE, or a version. Ids returned by the model are validated against the real finding set and dropped if unknown (src/triage/index.ts).
• The response is pinned to a Zod schema via structured outputs (messages.parse + zodOutputFormat), so the pipeline never parses loose prose or defends against a malformed shape.
• Adaptive thinking is on at effort: 'low' — enough for judgment, and it halves scan time versus medium with no loss of grouping or structure.
• If the call fails — or no key is set — triage falls back to the rules engine. A scan the user already waited for should end in a readable review. Losing the narrative is a degradation; losing the review is a failure.

The fallback isn't a stub. It groups by root cause, emits real upgrade commands for the repo's actual package manager, and separates urgent from deferrable using the same scores the model would have seen. What it can't do is explain why in language specific to your codebase — which is precisely the gap worth paying a model for, and why the UI names the author of every review.

---

API

Method	Path	Purpose
POST	/api/scans	Start a scan. 202 with the scan, which begins queued.
GET	/api/scans/:id	Full scan state — status, progress, findings, review.
GET	/api/scans/:id/events	SSE live progress. The UI uses this.
GET	/api/scans/:id/report	The review alone. 409 until triage finishes.
GET	/api/scans	Recent scans.
GET	/api/health	Liveness, plus which triage engine is active.

curl -X POST localhost:3200/api/scans \
  -H 'content-type: application/json' \
  -d '{"repoUrl":"OWASP/NodeGoat"}'

curl -N localhost:3200/api/scans/<id>/events   # live progress
curl localhost:3200/api/scans/<id>/report      # the review

repoUrl accepts what people actually paste: full HTTPS URLs (with or without a /tree/main tail), SSH remotes, and owner/repo shorthand.

---

Architecture

POST /api/scans → 202
      │
      ▼  in-process queue (concurrency 2)
  1. resolve    normalize URL → GitHub API: exists? public? size? branch?
  2. clone      git clone --depth 1 --filter=blob:none, timeout + cleanup
  3. collect    ├── deps      package.json + pnpm/npm/yarn lockfile
  │             ├── osv       OSV.dev /v1/querybatch → real CVEs
  │             ├── secrets   8 high-signal credential patterns
  │             └── config    committed .env, no lockfile, install hooks
  4. reach      does this repo's source actually import the package?
  5. score      DETERMINISTIC → riskScore 0–100  ← pure, unit-tested
  6. triage     Claude (schema-pinned) │ rules fallback → Report
  7. cleanup    always, including on failure

src/scoring/score.ts is the risk model, src/triage/ is the review layer, and src/plugins/scanning.ts is the only file that picks concrete implementations. public/ is the UI (no build step); tests/ holds 64 tests.

Why real scanning, not mocks. The brief allowed a mocked scan. Mocking the scanner would have meant mocking the exact thing the product is about — you can't demonstrate that context beats severity using findings you invented to prove it. OSV.dev is free and unauthenticated, so real CVE data cost no setup burden for whoever runs this.

Why these seams. ScanStore and JobQueue are interfaces with in-memory implementations, constructed in exactly one plugin. Swapping in Postgres and BullMQ means implementing two interfaces and changing one file. The seam is the point; the implementation is a take-home.

Reading the lockfile, not the manifest. A range like ^1.2.0 can't be matched against a CVE's affected versions — only a resolved version can. It also catches transitive dependencies, which is where most real vulnerabilities live (NodeGoat: 1,092 resolved packages from a manifest listing ~20).

Resolve before cloning. Repository size, visibility, and existence are checked via the GitHub API first, so a private, missing, or 2 GB repository is rejected without ever touching the disk.

---

Technical decisions

Fastify + fastify-type-provider-zod. One Zod schema per route gives both the runtime 400 and the compile-time type of request.body. Hand-rolling that means a manual parse plus a hand-written interface that can drift from it — an ugly place to lose correctness in a tool whose thesis is determinism. Pino and a catch-all error boundary come free.

SSE drops to reply.raw. Fastify's serialization adds nothing to an event stream, so the framing stays explicit — about fifteen lines. The client falls back to polling if the stream drops, because the scan is still running server-side and the result is worth recovering.

Node 24, ESM, TypeScript 7. .env is read with Node's built-in process.loadEnvFile() rather than a dependency.

Functions, not classes. The store, the queue, and the failure type are factory functions closing over their state; ScanStore and JobQueue stay interfaces so the seams survive. No this, no inheritance, one way to construct anything.

No frontend framework. One HTML file, one stylesheet, ~250 lines of vanilla JS. This is a backend assignment; a React build pipeline would have signalled effort in the wrong place.

---

Honest limitations

Things I'd want a reviewer to know I know:

• Reachability is a heuristic, and it's the weakest assumption here. It's a text scan for import specifiers, not a module graph. It can't see dynamic imports or re-exports, and it treats a package imported once identically to one imported everywhere. Most importantly, declared_only means your source doesn't import it — not that it's unused: a transitive dependency still runs, just via the package that pulled it in. That −12 is what pushes 228 of NodeGoat's findings into "safe to defer", so this one assumption carries more weight than any other number in the model. It's the first thing I'd harden, and the prompt now explicitly forbids the model from describing those packages as "unused" or "build tooling".
• Secrets are scanned in the working tree, not git history. A credential committed and later deleted is still exposed and still won't be found here. Fixing it properly means --filter=blob:none no longer suffices.
• There's no allowlist, so test fixtures get flagged. I found this by scanning this repository with itself: it reported the credential-shaped strings in tests/secrets.test.ts. Values vendors publish in their own docs (AWS's AKIAIOSFODNN7EXAMPLE) and structurally impossible keys (a PEM block with a 8-character body) are now ignored — note that's an exact-value match, never a substring one, since suppressing anything containing "EXAMPLE" is the bug described below. But a well-formed fake is indistinguishable from a real credential, so a scanner of this kind needs a .aiscanignore. That's the next thing I'd add.
• CVSS scores are approximated from the vector string rather than computed by a real CVSS library, and only the top 40 advisories get detail fetched — the rest are grouped into "safe to defer" by score alone, never individually triaged.
• Storage is in-memory. Scans die with the process.
• No auth, no rate limiting, no multi-tenancy. Out of scope at three hours, but the first three things needed before this faces anyone.
• Only npm. The collectors are ecosystem-shaped, so Python or Go would mean new collectors behind the same Finding interface — deliberately, that's the only part that would need to change.

What I'd do next, in order: persist scans (Postgres) → auth + rate limiting → real reachability via the module graph → scan git history for secrets → more ecosystems.

---

How I used coding agents

I built this with Claude Code, and the honest summary is that it was fastest where the work was mechanical and needed the most supervision exactly where the product thinking lived.

What worked well. Collectors were near-ideal agent work: lockfile parsing across three formats, the OSV batch client, the credential regex set, and the SSE plumbing were all specified-then-generated, and I mostly reviewed rather than rewrote. The test suite was similarly fast to produce once I'd decided what the invariants were. Boilerplate — types, config parsing, error plumbing — essentially disappeared as a cost.

Where I overrode it. The scaffold, for one: I asked for "bare minimum Fastify" and got the full fastify-cli generate template — ts-node, c8, concurrently, a compile-then-restart dev loop, CommonJS, npm scripts in a pnpm project. I kept the autoload convention and threw the toolchain out. Accepting generated scaffolding wholesale is how projects acquire dependencies nobody chose.

The risk model I didn't delegate at all. The weights in score.ts, the decision that reachability outweighs directness, and grading on worst-risk rather than finding count are product judgments. An agent will happily produce a plausible scoring function, and plausible is not the same as defensible.

Four bugs worth naming, because they're the ones that taught me something about building on a model rather than about TypeScript:

1. A security scanner failing open. The secret scanner suppressed lines containing "example" to filter placeholders. AWS keys literally end in EXAMPLE (AKIAIOSFODNN7EXAMPLE), so it silently discarded real credentials — the worst possible failure direction. I wrote the test expecting it to pass; it failed. The fix was to look for placeholder wording in the text around the credential, never in the credential itself.
2. The model obeyed an instruction I'd got wrong. My prompt said every finding must appear "exactly once across topRisks, nextActions, and deferred". So once Claude highlighted the committed private key as a top risk, it was forbidden from also listing it under the action that fixes it, and the UI rendered "closes 0 findings" on the most important action in the review. The model was correct; my data model wasn't. topRisks is a highlight layer that must overlap the partition, not a third bucket. When an LLM does something odd, the prompt is the first suspect.
3. The differentiator silently disappeared at scale. Only the top 40 findings go into the prompt, so on NodeGoat the model couldn't reference the other 232 — and "Safe to defer", the whole point of the product, rendered empty on exactly the repositories where it mattered most. The fix belongs in code, not the prompt: deferUnreferenced sweeps anything the reviewer never mentioned into reasoned groups. Never ask a model to guarantee an invariant you can enforce deterministically.
4. A latency fix quietly degraded security advice. Dropping output_config.effort to low halved scan time (95s → 49s) with no loss of structure or grouping — but the remediation for the committed private key silently weakened from "rotate it and purge it from git history" to git rm --cached plus a .gitignore entry, which leaves the key fully recoverable. What lower effort costs you is domain knowledge the model no longer pauses to retrieve, not reasoning. So the fix was a prompt rule stating that deleting a committed credential is never the whole fix — encode the knowledge instead of paying tokens to rediscover it.

The pattern. Agents are strong wherever correctness is checkable — parsers, clients, plumbing, tests — and need a firm hand wherever a decision encodes an opinion. I also used the model against my own work: I asked it to argue against weighting reachability so heavily, which surfaced the CI-still-runs-dev-dependencies point and turned the dev-dependency discount from a dismissal into −22 rather than −40.

One meta-note: I had Claude Code read Anthropic's own API reference before writing the triage call rather than working from memory, which is why it uses messages.parse with structured outputs instead of the hand-rolled JSON parsing I'd have written from a stale recollection of the SDK.