perf(hs): edge-cache /hs/code/* and /hs/heading/* to reduce middleware load

[SoapyRED]

Summary

Phase 0 + Phase 1 + Phase 1.5 of the scraper-defence-r1 sprint, landed together. Phase 1.6 / Phase 2 / Phase 3 are wall-clock blocked (24h + 48h windows) and become Soap-actionable follow-ups.

Phase 0 — scraper signature audit

See docs/audit/scraper-signature-2026-05-14.md (committed in this PR).

Headline finding: a sustained scraper is active. 7-day window pulled via Vercel MCP get_runtime_logs filtered to source=edge-middleware, query=ScrapeGuard, statusCode=429:

• ≥100 [ScrapeGuard] 429 log lines captured (MCP query timed out before exhausting matches)

• All from IPs in the 216.* /8 range

• All on /hs/code/*, /hs/heading/*, /hs/chapter/*, /hs/section/*

• Sustained ~5-second interval (scripted enumeration, not a real user)

• Sample 10 minutes from window-close:

  19:12:45  GET /hs/code/843691     429
  19:12:35  GET /hs/heading/2003    429
  19:12:30  GET /hs/heading/2007    429
  19:12:25  GET /hs/heading/2829    429
  19:12:20  GET /hs/heading/0803    429
  19:12:10  GET /hs/section/xv      429
  ...
  

ScrapeGuard is doing its job — every request 429'd. The cost is the Redis INCR per request. At sustained ~5s/req from one source, that's ~17K wasted Redis commands/day.

Vercel WAF events (/v1/security/firewall/events) returned 0 actions over the same 7-day window. The 9 existing firewall rules (Tencent AS132203, 43.172/16+43.173/16, AhrefsBot/SemrushBot/etc UA blocks, HS code rate-limit) did NOT match this scraper.

Data-access gaps documented honestly:

Sprint request	Status
Top 20 UAs	❌ middleware log line doesn't include UA; no raw access log accessible from sandbox
Top 20 source IPs / ASNs	⚠️ partial — 216.* /8 confirmed, finer resolution truncated by MCP UI render
Country distribution	❌ same blocker
Upstash Redis baseline	❌ KV_* sensitive-typed in Vercel prod env, not retrievable from sandbox

Probed (all returned 404): /v3/projects/.../logs, /v1/projects/.../runtime-logs, /v2/logs, /v9/logs, /v1/observability/access-logs, /v2/configurations/log-drains. /v2/integrations/log-drains returned 200 but empty (no drains configured). No vercel CLI in sandbox.

Phase 1 — firewall rules added: 0

The sprint brief's evidence thresholds (≥1000 reqs from one UA, ≥500 reqs from one ASN at ≥80% scraper-bait paths) cannot be evaluated given the data-access gaps above:

• UA thresholds need UA strings → middleware doesn't log them
• ASN thresholds need full IPs → MCP truncates IP at 216...
• Adding a 216.0.0.0/8 block would cover ~16M IPs across many ARIN ASNs (residential, corporate, cloud) — too broad to be evidence-based

Honest call: stop short of speculative rules. The application-layer ScrapeGuard is catching the scraper. Phase 1.5 (below) reduces the cost of catching it rather than trying to add another firewall layer with insufficient signature data.

Phase 1.5 — next.config.ts cache headers + FAULT 5

Adds explicit Cache-Control to /hs/code/* and /hs/heading/* via next.config.ts headers():

public, max-age=300, s-maxage=86400, stale-while-revalidate=604800

Both routes already had export const revalidate = 86400 (ISR) — this surfaces the cache strategy at the config level so it's tunable + curl-visible.

What this changes on the wire:

• Cold serve: unchanged (middleware → page render → cached at edge)
• Warm cache hit: response served from Vercel edge without re-rendering the page server component (React tree + lib/calculations/hs lookup skipped)
• The application-layer ScrapeGuard rate limiter STILL runs on every request (Next.js middleware runs before cache lookup). Phase 1.6 measures whether page-render savings reduce overall Redis-INCR volume enough; if not, Phase 2 (generateStaticParams()) ships.

FAULT 5 checklist — applied

Check	Status
CHANGELOG.md entry under 2026-05-14	✅ added
lib/changelog-data.ts entry (renders on /changelog via monthGroups())	✅ added
/changelog page renders the entry	✅ verified by reading the monthGroups() function — entries are sorted by isoDate and grouped by month; new May 14 entry surfaces under "May 2026" group
No new URLs (no sitemap or IndexNow action needed)	✅ no new URLs
No siteStats.ts changes	✅ none
No nav changes	✅ none
No API contract changes	✅ response bodies unchanged; only response headers added

Smoke test posture

scripts/smoke-test.mjs will be run against the Vercel preview URL after this PR opens. The smoke covers API endpoints — it does NOT specifically assert Cache-Control on /hs/* pages, but it does assert no 5xx regression. Post-merge, the deliverable is a curl -I https://www.freightutils.com/hs/code/<sample> showing the new Cache-Control header.

Files

File	Type
docs/audit/scraper-signature-2026-05-14.md	new (Phase 0 audit)
next.config.ts	extended (added hsPageCachedPaths mapping to Cache-Control)
CHANGELOG.md	new entry under 2026-05-14
lib/changelog-data.ts	new entry at top of entries[]

Hard-rule compliance

• ✅ No force-push (PR merged via gh pr merge)
• ✅ lib/data/test_airlines.js not touched (no modification needed this sprint)
• ✅ Conventional commits (docs(audit): + perf(hs):)
• ✅ TaskStop on every backgrounded shell (none spawned this sprint — all Bash calls were foreground)
• ✅ No new data fabricated, no LLM-knowledge claims, no fictional sources

Soap to-do post-merge (time-blocked phases)

• Phase 1.6 — 24h observation window: Pull Upstash Redis commands/day from the Upstash or Vercel Storage dashboard 24h after this PR merges. Compare to the pre-merge baseline. If commands/day is < 3K average target, Phase 2 is OPTIONAL — skip and proceed to Phase 3.
• Phase 2 (conditional) — full generateStaticParams() for HS pages: Only fire if Phase 1.6 shows the cache-header path didn't get under 3K commands/day. Adds ~6,940 HS code params + ~5,224 heading params at build time. Watch the build time budget — abort if it crosses 20 minutes.
• Phase 3 — 48h verification window: Pull Redis commands/day + Sentry quiet check + spot-check 5 /hs/code/* URLs for x-vercel-cache: HIT. Write a final status note to docs/audit/scraper-signature-2026-05-14.md with baseline vs post-fix percent reduction.
• Unblock the Phase 0 data-access gaps for the next iteration: Soap can export the Vercel Dashboard log explorer (Chrome path — has full IP + UA + ASN columns) OR install a Log Drain to a 3rd-party (Better Stack, Logtail), OR extend middleware.ts ScrapeGuard's console.warn to log req.headers.get('user-agent') so the next sprint can do evidence-based UA blocking.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
freighttools	Ready	Preview, Comment	May 14, 2026 7:33pm