v0.2.0 — Per-engine fan-out, service binding, status column
A correctness + architecture pass driven by an end-to-end debugging session on the production fork. Every fix is described in terms of what the prior version got wrong.
Added
migrations/0004_response_status.sql:status(ok/failed/skipped) anderror_messagecolumns onprompt_responses, plus an index onstatus. Failed engine calls used to writeraw_response='ERROR: ...'rows that downstream scoring treated as real zero-mention hits. They're now explicitstatus='failed'rows that aggregates exclude.POST /admin/run-engine— single-engine handler./admin/run-liveself-fetches into it once per engine so each engine runs in its own worker invocation with its own free-plan 50-subrequest budget. Idempotent on(run_id, engine).POST /admin/cleanup-failed-runs— one-shot deletion of legacy polluted rows that were written before the status column existed. Idempotent, optionalbrand_idfilter.bulkCacheGethelper insrc/db.ts— one D1 read for N prompt hashes instead of N separate reads.persistEngineRunhelper insrc/db.ts— one batched D1 write at the end of an engine run instead of per-prompt inserts + cache puts + per-chunk run updates.EnginePromptResultinterface — engines collect their per-prompt results in memory and flush in one batch.runEngineInProcessinsrc/engines.ts— the in-process dispatcher used by/admin/run-engine.Fetcherservice binding (env.SELF) inEnvandEnginesEnv, plus"services"block inwrangler.example.jsonc. The per-engine fan-out goes throughenv.SELF.fetch()because a public-URL fetch back to your ownworkers.devhostname trips Cloudflare's "Worker called itself" guard (error 1042) and never lands.SELF_URLenv var inwrangler.example.jsonc— the cron path needs to know the worker's canonical URL since it has no inbound request to derive an origin from.
Changed
- Each engine client (
src/openai.ts,anthropic.ts,perplexity.ts,gemini.ts,ai-overviews.ts) now uses the bulk pattern: hash all prompts up front,bulkCacheGetonce, run LLM fetches in CONCURRENCY=5 chunks, collectEnginePromptResult[], flush viapersistEngineRun. Per-invocation subrequest count drops from ~89 to ~26. getLatestCompletedRunanchors onEXISTS(prompt_responses with status='ok')instead ofruns.status='completed'. Partially-finished runs still surface their data instead of silently disappearing from MCP tool output.check_visibility,compare_competitors, andget_content_gapsuserun.completed_at ?? run.started_atso they accept in-progress runs.get_visibility_historyandget_citationsSQL queries filterpr.status = 'ok'in the join/where clause so failed/skipped rows can't pollute aggregates./admin/run-engineINSERT OR IGNOREs its runs row before calling the engine — D1's cross-region replication may not have caught up with/admin/run-live'sINSERT INTO runsby the time the self-fetch lands, and a missing parent row would crash the engine's batched insert withFOREIGN KEY constraint failed.runEnginesaccepts an optionalrequestargument and falls back tonew URL(request.url).originwhenSELF_URLisn't set, so HTTP-triggered code paths work even without the env var.
Fixed
- Subrequest-cap row loss: pre-fix, every engine deterministically wrote 10 rows out of 20 because chunk 3 hit the 50-subrequest cap. Now writes complete in one batch under the cap.
- Silent row drops from failed Gemini / AI Overviews calls: pre-fix the catch block's
insertPromptResponse(failed)was itself a D1 subrequest that could fail, leaving zero rows for the prompt. The bulk batch holds failure results in memory and writes them in the same batch as the successes. - Visibility tools dropping engines whose runs got stuck at
in_progress: relaxedgetLatestCompletedRunsurfaces them.