Point Barnacle at a site, describe the user flow in plain English, and run three recon commands. Barnacle drives a real browser through the flow, captures every API call, replays them with plain HTTP to prove which ones work without a browser, probes rate-limit ceilings, and then generates a complete plugin — Zod schemas inferred from captured JSON, load-bearing headers, rate-limit ceiling, hot-path HTTP client, and Stagehand browser fallback. Register the plugin in one line; Barnacle handles sessions, retries, fallback routing, audit persistence, and response envelope wrapping.
Stagehand drives a real browser through your described user flow. Its only job is
to trigger the site's network traffic — not to extract DOM data. While it clicks,
a response listener wiretaps every API call to disk. Once that recon run is done,
a separate script replays those captures via plain fetch() — no browser, no AI —
to prove the endpoints work standalone. The surviving queries and headers become
committed constants. In production, the runtime hits those endpoints directly:
fast, cheap, deterministic. The browser only re-engages if the direct path breaks.
A nightly smoke test tells you the moment a contract drifts. When it fires, you re-run the same recon command you ran the first time and diff the captures. Human involvement is one recon run up front and a small PR when things change.
| Phase | What runs | What you get |
|---|---|---|
| 1 — Browser recon | pnpm run recon:browser |
Every API call the site makes, captured to <run-dir>/graphql/*.json |
| 2–3 — HTTP replay + probing | pnpm run recon:http |
Proof each endpoint works without a browser; rate-limit ceiling; static fixtures |
| 4 — Plugin generation | pnpm run recon:generate |
A complete plugin: Zod schemas, headers, Bottleneck config, hot-path client, Stagehand fallback |
| 5+ — Runtime | pnpm start |
Direct HTTP hot path, automatic browser fallback, nightly smoke test, drift detection |
You will get asked why not just use the browser for every request, or scrape HTML, or reverse-engineer endpoints by hand. Here is the honest comparison:
| Approach | Cost/req | Latency | Fragile to UI | Fragile to API | Effort |
|---|---|---|---|---|---|
| Browser on every request | High | 5–15 s | Medium | Low | Low |
| HTML screen scraper | Low | Low | High | Low | Medium |
| Manual DevTools recon | Low | Low | Low | High (human redo) | High (ongoing) |
| HAR replay (mitmproxy) | Low | Low | Medium | High | Medium |
| Recon → codify → direct HTTP + fallback (this) | Low | Low | Low (re-runnable) | Low (fallback covers) | Medium, front-loaded |
The browser-on-every-call approach uses Steel minutes and Anthropic tokens on every production call — orders of magnitude more expensive at scale. HTML scrapers break on every UI redesign, and the API response usually carries richer data than what the UI renders anyway. Manual DevTools recon is exactly what this pipeline automates, but committed and re-runnable. Front-loaded recon work buys an integration as cheap as direct HTTP, as robust as a browser fallback, and maintainable in a way none of the hand-rolled options are.
Every new site follows the same pipeline (Phases 0–6). The only human-authored input is the flow definition you write once in Phase 0. After that, the scripts run unattended — recon captures, HTTP replay proves endpoints, the generator writes the plugin. When the site changes months later, you re-run the same command and diff the captures. Human time is front-loaded to one recon run and a small PR.
Commit the flow steps to a file first — this makes recon re-runnable in one command without retyping it. When the site changes and you need to re-run recon months later, you git pull and run the same command you ran the first time:
# src/sites/my-site/recon-flow.json
["click the Electronics category filter", "open the first product result"]# Preferred: load flow from committed file
pnpm run recon:browser -- \
--url https://example.com \
--flow-file src/sites/my-site/recon-flow.json
# Or inline (ephemeral — must be re-typed each recon run):
pnpm run recon:browser -- \
--url https://example.com \
--flow '["click the Electronics category filter", "open the first product result"]'
# For sites whose API paths don't match /graph, /api/, /graphql, /v1/, or *.json:
pnpm run recon:browser -- \
--url https://example.com \
--flow-file src/sites/my-site/recon-flow.json \
--capture-all
# Capture page-load XHRs only (no interaction — useful for pure GET-style SPAs):
pnpm run recon:browser -- --url https://example.comDrives a real Stagehand + Steel browser through your flow. Captures are wired via a single CDP session-level listener (page.getSessionForFrame().on(...)) — Stagehand V3 enables the Network domain internally, so attaching our Network.requestWillBeSent / responseReceived / loadingFinished listeners on the main session catches every response, including the early ones that fire before any page-level handler could be wired.
Captures every network call matching /graph, /api/, /graphql, /v1/, or *.json to <run-dir>/graphql/<NNN>-<phase>-<operationName>.json — one file per call, diffable and greppable. Use --capture-all for sites with non-standard API paths; it captures every response, producing more noise but missing nothing. Omitting both --flow and --flow-file runs zero interaction steps and captures only the network activity that fires during page navigation — useful for pure GET-style SPAs that fetch everything they need on load.
Each step runs through a self-healing cascade (act → observe + act → observe + act with ignoreSelectors → LLM rephrase) verified by network-counter delta or URL change. The script's main() attempts up to two global flow replans before giving up; terminal failures dump a diagnostic bundle to <run-dir>/step-failures/. See docs/playbook.md#1c--self-healing-cascade for the full design.
Total runtime: 20–40 minutes for a typical flow (longer if healing or replans fire), fully unattended.
Every artifact — captures, cookie-jar snapshots, step-failure dumps, DOM dumps — is rooted under one run-scoped directory resolved once at startup: <run-dir> defaults to /tmp/recon/<runId>, where <runId> is a timestamp + random suffix generated per process (e.g. 20260718-120326-a1b2). Set RECON_RUN_ID to pin a deterministic runId (e.g. for tests or replaying a known run) and RECON_OUT_DIR to override the base directory runs are rooted under. This keeps concurrent or repeated runs from intermixing files — the startup log line prints both the resolved runId and out=<run-dir>.
Alongside the network captures, every run snapshots the browser's complete cookie jar (via CDP Network.getAllCookies, which returns the whole-browser jar regardless of the current page's URL — unlike document.cookie or Page.getCookies, it also sees HttpOnly cookies) at each phase boundary: the initial goto, immediately before each flow step (pre-step), immediately after each flow step completes (post-step), and once more at run completion (run-complete).
Snapshots land in <run-dir>/cookies/<NNN>-<label>-<phase>.json — one file per boundary, using the same zero-padded chronological index convention as the network captures. <label> is the boundary kind (goto, pre-step, post-step, run-complete); <phase> is the current step's slugified instruction (e.g. click-the-apply-button), or home before the first step starts.
Each file is a JSON object:
{
"label": "post-step",
"phase": "click-the-apply-button",
"stepIndex": 2,
"timestamp": "2026-07-18T12:34:56.789Z",
"cookies": [
{
"name": "_acme_attr",
"value": "abc123",
"domain": ".acme.example",
"path": "/",
"expires": 1234567890,
"size": 20,
"httpOnly": true,
"secure": true,
"session": false,
"sameSite": "Lax"
}
]
}Field reference (mirrors CDP's Network.Cookie type verbatim — no remapping between capture and disk):
| Field | Meaning |
|---|---|
name / value |
The cookie's name and value. |
domain |
Scope, e.g. .acme.example (all subdomains) vs. apply.acme.example (exact host) — the detail needed to tell a click-domain cookie from an apply-domain cookie. |
path |
Cookie path scope. |
expires |
Raw CDP epoch-seconds number; -1 means a session cookie (also reflected in session: true). Not reformatted — read it as CDP reports it. |
size |
Cookie size in bytes, as reported by CDP. |
httpOnly / secure |
Standard cookie flags. |
session |
true for a session cookie (no persistent expiry). |
sameSite |
"Strict" | "Lax" | "None" | null — null when the cookie doesn't set the attribute. |
If the CDP call fails, the file still writes but with an empty cookies array and an error string field carrying the failure message — cookie telemetry is best-effort and never aborts the run.
Diffing what a phase established: to isolate what a specific traversal (e.g. a tracking-click redirect) minted, diff its post-step snapshot against the pre-step snapshot for the next step — cookies present in the later file but absent from the earlier one were established during that step:
diff <(jq -S .cookies <run-dir>/cookies/004-post-step-click-the-apply-button.json) \
<(jq -S .cookies <run-dir>/cookies/005-pre-step-fill-in-your-name.json)Cookies actually sent per request: the jar snapshot shows what's available, not what's sent. Each network capture in <run-dir>/graphql/ separately carries the outgoing Cookie header in its requestHeaders (recovered via CDP's Network.requestWillBeSentExtraInfo, since requestWillBeSent omits it by design) — cross-reference that capture's requestHeaders.Cookie against a jar snapshot to see which of the available cookies a given request, e.g. the application submit, actually sent.
Caveat on Set-Cookie: response captures fold responseReceivedExtraInfo headers (which is where Set-Cookie actually appears — responseReceived omits it) into responseHeaders as a flat Record<string, string>. CDP does not guarantee multiple Set-Cookie values on one response stay distinguishable once folded into that shape — if a single response mints more than one cookie, treat the jar snapshot (not the response capture's Set-Cookie header) as the source of truth for what actually landed.
pnpm run recon:httpReplays every capture via plain fetch() — no browser, no AI — to prove endpoints work standalone. Every replay returning 200 proves the browser is unnecessary for production. Also runs GraphQL introspection, auxiliary fixture detection (static JSON to commit as fixtures), and a rate-limit probe at 1→3→5 rps (run last — if it triggers a ban, all captures are already saved). Results land under the run-scoped root resolved by resolveReconRunDir() — /tmp/recon/<runId>/replays/ by default, rooted elsewhere via --out-dir <path> or RECON_OUT_DIR.
See docs/playbook.md for the full troubleshooting decision matrix when replays fail.
pnpm run recon:generate -- --site-id my-siteReads every artifact from Phases 1–3 — <run-dir>/graphql/*.json (captures), <run-dir>/replays/*.json (replay results), <run-dir>/replays/rate-limit.json (probe findings), <run-dir>/aux/*.json (static fixtures), and src/sites/my-site/recon-flow.json — and writes a complete plugin to src/sites/my-site/. Pass --run-dir <path> to read a specific run's artifacts instead of the most recently modified run root under /tmp/recon (or RECON_OUT_DIR, if set):
contract.ts— Zod schemas inferred from captured JSON, load-bearing headers, Bottleneck ceiling, andexecuteHttp/executeimplementationsflows/browser-flow.ts— Stagehand fallback wired to yourrecon-flow.jsonstepsindex.ts— barrel exportfixtures/— any static JSON found by the auxiliary probe, already copied in
Then review the generated files: trim UI-only fields from the GraphQL query, narrow any z.unknown() entries in the schema you care about, and verify the headers. If you need to regenerate after making changes to the recon flow, pass --force.
Barnacle cannot know what your site's forms mean. "Select the departure port from the Country dropdown" and "…select the test candidate's country" are the same sentence shape; only you know that the first is a search facet and the second is your caller's address. So the vocabulary is yours to supply, with --vocabulary:
// src/recon/my-vocabulary.ts
import type { ReconVocabulary } from "@enricai/barnacle/recon/vocabulary";
export const vocabulary: ReconVocabulary = {
// Only a step naming the SUBJECT may splice off a bare dropdown — a dropdown
// step carries no quoted constant, so the label alone cannot tell your
// caller's data from a filter that happens to say "Country".
subject: /\b(the\s+)?(test\s+)?customer'?s\b/i,
// Labels that must stay literal even when a table row matches.
exclusions: [/\bbilling contact\b/i],
// Ordered label → payload field. First match wins, so specific precedes broad.
table: [
[/\bfirst name\b/i, "FirstName"],
[/\bcity\b/i, "City"],
],
};pnpm run recon:generate -- --site-id my-site --vocabulary ./src/recon/my-vocabulary.ts- The specifier follows the same rule as
BARNACLE_PLUGINS: a leading.or/is a filesystem path; anything else resolves from yournode_modules. The module may exportvocabularyor a default. --vocabulary nonefor a site that splices no caller data at all (read-only inventory, search, pricing). Explicit, so it can't happen by accident.- Every regex must be free of the
g/yflags and every field name must be a valid JS identifier — the loader rejects both. A stateful regex matches only every other step; a non-identifier emitspayload.<name>as a syntax error. - Run the generator with a
.tsvocabulary undertsx, or point it at compiled.js— plainnodecannot import TypeScript before v22.18.
Omitting
--vocabularydisables splicing entirely — every step's recon constant is emitted literally, with no caller-data substitution.
Where --vocabulary matches instruction prose, --form-schema names the JSON keys the generator reads out of an ATS's form-definition responses when recovering field ids, option ids, and submitted values. The engine ships no vendor's format; a site whose ATS exposes a form definition declares its keys with --form-schema:
// src/recon/my-form-schema.ts
import type { ReconFormSchema } from "@enricai/barnacle/recon/form-schema";
export const formSchema: ReconFormSchema = {
fieldIdKey: "fieldId", // UUID-valued field identity
fieldNameKeys: ["code", "label"], // code, then human label — code preferred
fieldOptionsKey: "options",
optionIdKey: "optionId", // option id, inside the options array
optionValueKey: "optionLabel", // option label, inside the options array
responseValueKey: "submittedValue", // submitted free value
responseOptionIdKey: "submittedOptionId", // submitted option reference
};pnpm run recon:generate -- --site-id my-site --form-schema ./src/recon/my-form-schema.ts- The specifier follows the same rule as
--vocabulary: a leading.or/is a filesystem path; anything else resolves from yournode_modules. The module may exportformSchemaor a default. --form-schema nonefor a site with no ATS form definition (a search API, a cruise site) — same as omitting it. "none" is the explicit form.- Wire keys anchor
"key":"uuid"markers, so they may be any non-empty string without a quote or backslash — the JS-identifier rule that governs vocabulary field names does not apply here. fieldNameKeysmodels two roles: the first key is a machine code (PascalCased directly), the second is a human label (run through the section-heading heuristic). Supply one key for a label-only ATS, or two for one that exposes both. Additional keys are unused.- Omit
--form-schema(or passnone) and ATS form-key recovery does not run — the engine hardcodes no vendor's wire format. A site whose ATS exposes a form definition must supply one to recover its option fields. See issue #57.
If the site asks screening questions, tell the generator which payload field answers each one — the same reasoning applies, it cannot know what your site asks:
| Env var | Default | Description |
|---|---|---|
RECON_QUESTION_KEYWORDS |
{} |
JSON object mapping a payload field name to the keywords that identify its question. A question must match at least 2 keywords to map. |
RECON_TELEMETRY_URL_PATTERNS |
(empty) | Comma-separated extra URL fragments to treat as analytics noise, on top of the built-in list. Put your site's trackers here. |
RECON_QUESTION_KEYWORDS='{
"VisaSponsorship": ["visa", "sponsor"],
"RelatedToEmployee": ["related", "employee"],
"CanPerformJobFunctions":["perform", "job functions", "duties"]
}' pnpm run recon:generate -- --site-id my-siteAny question that matches no field is logged by prompt rather than skipped — an unmapped required question is how a generated plugin ends up submitting nothing for it. Read those warnings and either add keywords or accept that the question goes unanswered. Malformed JSON logs a warning and is treated as empty, so a typo cannot kill a recon run mid-flight.
Optionally generate the human-readable findings doc alongside:
pnpm run recon:summarize -- --site-id my-siteWrites docs/my-site-recon.md with: endpoints found, replay status, rate-limit ceiling, header frequency table, and hazards (Akamai, Cloudflare). Without --site-id, the default output path is docs/target-recon.md. Accepts --run-dir <path> the same way recon:generate does.
See Plugin Authoring Guide → Register the plugin below for plugin registration options.
See Plugin Authoring Guide → Wire up the nightly smoke test below for the CI step that runs the smoke test nightly.
See docs/playbook.md for the full detection ladder and maintenance loop.
The dashed deploys edge is the human-in-the-loop step (the contract PR merges and ships to Runtime). The solid orange edge from smoke-test.ts back into Phase 1 is the self-healing loop: when the contract drifts, recon reruns unattended (~20–40 min) and the next PR is a diff of captures, not a hand-rewrite. See docs/architecture.md for the design rationale behind each lane.
A site plugin is a single TypeScript module that satisfies SitePlugin<TInput, TOutput>
from src/site-plugin.ts. Core registers built-in plugins via BUILTIN_SITE_PLUGINS in
src/plugins/discover.ts; out-of-tree plugins are loaded at startup via BARNACLE_PLUGINS.
interface SitePlugin<TPayload, TResult> {
meta: SitePluginMeta;
// Optional direct-HTTP hot path — no browser, no LLM tokens, millisecond latency.
// Core tries this first; falls back to execute() on HttpSchemaError / HttpBotChallengeError / HttpServerError.
executeHttp?: (
payload: TPayload,
context: SitePluginContext
) => Promise<SitePluginResult<TResult>>;
// Browser fallback — Stagehand + Steel session, acquired from the pool by core.
execute(
payload: TPayload,
session: BrowserSession,
context: SitePluginContext
): Promise<SitePluginResult<TResult>>;
// Async work is supported. Note: NOT called on CaptchaError or EmptyResultsError —
// p-retry skips onFailedAttempt for AbortError, so those abort paths bypass this hook.
onRetry?: (error: ScraperError, attempt: number) => void | Promise<void>;
}| Field | Type | Purpose |
|---|---|---|
siteId |
string |
Stable key used for routing (/v1/<siteId>/run) and audit rows |
displayName |
string |
Human-readable label for logs and Swagger docs |
bodySchema |
ZodTypeAny |
Request body schema — core validates before calling execute() |
responseSchema |
ZodTypeAny |
Success response schema — drives Swagger output shape |
routeOverride? |
string |
Override the full route path (legacy compatibility only) |
defaultBaseUrl? |
string |
Fallback base URL when config.scraper.siteBaseUrls[siteId] is absent |
taskTimeoutMs? |
number |
Override the pool's 60-minute per-task hang ceiling for this plugin only — set when the site's normal latency is well below the default and a faster failure is preferable |
maxAttempts? |
number |
Override the retry policy's default of 3 attempts (including the first try). Without this, the per-run ceiling is 3 × taskTimeoutMs; set to 1 so taskTimeoutMs is the real per-run cap |
apiVersion? |
string |
Semver range targeting a plugin API version (e.g. "^1.0.0"); core disables the plugin on a major-version mismatch. Absent means "accept any version." |
extraRoutes? |
readonly SitePluginExtraRoute[] |
Extra non-run routes (OTP trigger, resume, etc.) that core registers as authenticated Fastify routes at startup. See SitePluginExtraRoute in src/site-plugin.ts. |
onShutdown? |
() => Promise<void> |
Optional cleanup for background work the plugin launched fire-and-forget, awaited during graceful shutdown so in-flight work is not abandoned and sessions are not leaked. Mirrors the engine's own drain functions. Bounded by a per-plugin timeout, so a hanging drain cannot stall shutdown. Module plugins only — config-only *.plugin.json manifests are pure JSON and cannot declare a function. |
pnpm run recon:generate produces this structure automatically. Use createRateLimitedJsonClient() for REST endpoints that send Chromium client-hint headers (the common case) and createGraphqlClient() for GraphQL endpoints — recon:generate selects the right one based on what it captured. The skeleton below illustrates the REST hot-path pattern; for GraphQL sites, recon-generate uses createGraphqlClient instead. A GraphQL target with a single captured query inlines it as a constant; a GraphQL target whose captures form a multi-operation mutation sequence (a submission flow) instead gets the same state-threaded, multi-step executeHttp REST submission flows get.
// src/sites/my-site/contract.ts
import { z } from "zod/v4";
import { createRateLimitedJsonClient } from "@/scraper/rate-limited-json-client";
import type { BrowserSession } from "@/scraper/session";
import type { SitePlugin, SitePluginContext, SitePluginResult } from "@/site-plugin";
import { runMySiteBrowserFlow } from "@/sites/my-site/flows/browser-flow";
// Generated: Zod schemas inferred from captured JSON — tighten z.unknown() fields as needed.
const MySiteResponseSchema = z.object({ data: z.object({ items: z.array(z.object({ id: z.string() })) }) });
const MySitePayloadSchema = z.object({ query: z.string().min(1) });
type MySitePayload = z.infer<typeof MySitePayloadSchema>;
type MySiteResponse = z.infer<typeof MySiteResponseSchema>;
// Generated: rate-limit ceiling (5 rps) + Chromium hints + site-specific headers from recon.
// Use createHttpClient() directly only when you need manual Bottleneck or header control.
const httpClient = createRateLimitedJsonClient({
minTimeMs: 200,
userAgent: "Mozilla/5.0 ...",
secChUa: '"Chromium";v="..."',
platform: "Linux",
extraHeaders: {
"Content-Type": "application/json",
Accept: "application/json, */*",
},
schema: MySiteResponseSchema,
});
export const mySitePlugin: SitePlugin<MySitePayload, MySiteResponse> = {
meta: {
siteId: "my-site",
displayName: "My Site",
bodySchema: MySitePayloadSchema,
responseSchema: MySiteResponseSchema,
defaultBaseUrl: "https://my-site.com",
},
// Hot path: direct HTTP — no browser, no LLM tokens.
async executeHttp(payload: MySitePayload, context: SitePluginContext): Promise<SitePluginResult<MySiteResponse>> {
const data = await httpClient(`${context.baseUrl}/api/search`, {
method: "POST",
body: JSON.stringify({ query: payload.query }),
});
return { data };
},
// Browser fallback: Stagehand + Steel — invoked automatically when hot path fails.
async execute(payload: MySitePayload, session: BrowserSession, context: SitePluginContext): Promise<SitePluginResult<MySiteResponse>> {
const raw = await runMySiteBrowserFlow(session.stagehand, context.baseUrl, payload.query);
return { data: raw };
},
};SitePluginResult accepts an optional auditPayload field alongside data:
return {
data: responseData,
auditPayload: { query: payload.query, resultCount: responseData.items.length },
};When auditPayload is present, core writes it — not data — to the submission-envelope telemetry record. Use this to strip PII or large blobs from the audit trail while keeping the full response in the API reply. When absent, data is written as-is.
Core has no opinion on what a reconciliation join key is named or how it's
shaped — that's site-specific vocabulary (an attribution vendor's click ID,
a job-reference composition rule, whatever the site needs). A plugin that
needs its submission and beacon-fire telemetry to be joinable back to its own
attribution provider declares an optional extractJoinKeys hook on its
SitePlugin:
export const myPlugin: SitePlugin<MyPayload, MyResponse> = {
extractJoinKeys: (payload) =>
payload.someVendorClickId ? { vendorClickId: payload.someVendorClickId } : null,
// ...
};dispatch() (src/plugins/loader.ts) calls this once per submission,
resolving extractJoinKeys(payload) from the inbound payload alone — core
never inspects the result's contents. A plugin with no reconciliation needs
simply omits extractJoinKeys.
extractJoinKeys only ever sees the payload a plugin received up front, so
it has no way to attach a field the plugin only discovers during the run
— a token minted mid-flow, a value read from the page after navigation, a
value observed on a response. For that, call the mid-run attach point,
context.telemetry.addJoinKeys(), from anywhere inside execute() or
executeHttp():
async execute(payload: MyPayload, session, context: SitePluginContext) {
const mintedToken = await readTokenFromPage(session);
context.telemetry.addJoinKeys({ mintedToken });
// ...
},context.telemetry is a per-dispatch RunTelemetry accumulator
(src/lib/telemetry/run-telemetry.ts), constructed fresh for every
dispatch by buildPluginContext (src/plugins/loader.ts) alongside
recordBeaconOutcome below. Successive addJoinKeys() calls within the
same run merge, later calls winning on key collision. Once the plugin call
resolves — on both the success and error paths — dispatch() snapshots the
accumulator and merges it over the earlier extractJoinKeys(payload)
result, run-discovered keys winning on collision, before stamping the
combined bag onto the submission envelope's and beacon-fire record's
joinKeys field. joinKeys stays null only when neither source ever
produced anything.
A config-only *.plugin.json manifest can reach
context.telemetry.addJoinKeys() only through the same spec.httpModule
escape hatch documented below for context.recordBeaconOutcome —
executeHttp(payload, context) receives the same SitePluginContext, so an
httpModule can call it exactly like execute() does above; the
manifest's declarative browser flow cannot, since runHealingFlow is
data-driven with no imperative call site for either seam to live in.
Declaring extractJoinKeys also opts the plugin out of core's automatic
TrackingUrl fire. If the site returns a post-submission click-tracking
URL, declare it on the plugin's bodySchema by composing JobTrackingSchema
(src/lib/job-tracking.ts) — MySitePayloadSchema.extend(JobTrackingSchema.shape).
When a plugin has no extractJoinKeys, dispatch() fires that TrackingUrl
itself via fireTrackingClick, site-agnostically, after a successful submit.
When a plugin does declare extractJoinKeys, core assumes the plugin fires
its own post-submit tracking navigation (e.g. because the click and apply
navs must share one browser session for a vendor's device-cookie
attribution to work) and skips its own fire — firing both would open two
independent sessions against the same URL.
By default a self-managing plugin's beacon-fire telemetry is stuck at
beaconStatus: "skipped", since core has no visibility into a navigation the
plugin drives itself. To report the real outcome, call
context.recordBeaconOutcome — passed on SitePluginContext alongside
baseUrl/logger/requestId, bound to this run — from execute(),
executeHttp(), or an extra-route handler:
import type { SitePlugin, SitePluginContext } from "@enricai/barnacle/site-plugin";
export const myPlugin: SitePlugin<MyPayload, MyResponse> = {
extractJoinKeys: (payload) =>
payload.someVendorClickId ? { vendorClickId: payload.someVendorClickId } : null,
async execute(payload, session, context: SitePluginContext) {
const t0 = Date.now();
const fired = await runMySiteBeaconNav(session, payload.TrackingUrl);
await context.recordBeaconOutcome({
beaconStatus: fired ? "fired" : "failed",
joinKeys: { vendorClickId: payload.someVendorClickId },
trackingUrl: payload.TrackingUrl,
durationMs: Date.now() - t0,
});
// ...
},
};Core binds the run's requestId and the plugin's own siteId for you, so
recordBeaconOutcome's input carries only beaconStatus ("fired" |
"failed" — "skipped" stays an engine-owned outcome), the opaque joinKeys
bag (same shape returned from extractJoinKeys), and optional trackingUrl/
durationMs. It never throws — a telemetry-sink hiccup cannot fail the
request. A fired/failed line recorded this way outranks the automatic
skipped line for the same requestId when the two are folded together
(see Telemetry & LLM judging). A plugin that
never calls it keeps today's unchanged skipped default. Import
BeaconOutcomeInput from @enricai/barnacle/site-plugin if you want to type
the input object explicitly — that's the published subpath an out-of-tree
plugin resolves against its own node_modules; in-tree code under src/
uses the @/site-plugin alias instead.
A config-only *.plugin.json manifest can reach context.recordBeaconOutcome
only through the spec.httpModule escape hatch — executeHttp(payload, context) receives the same SitePluginContext a module plugin's does, so an
httpModule can call it exactly like execute() does above. The manifest's
declarative browser flow cannot: runHealingFlow is data-driven, with no
imperative call site for a call like this to live in. One consequence to know
before adopting it: buildConfigPlugin never synthesizes extractJoinKeys,
so a config-only plugin is never managesOwnTracking — when the response
carries a TrackingUrl, core still fires it itself via fireTrackingClick,
and a manifest-recorded fired/failed line for that requestId ranks
equal to core's own line under beaconRank(), so the fold resolves by write
order (last line wins) rather than the manifest's line automatically
outranking core's. Only when no TrackingUrl is present — so core's own
write is the skipped default — does the manifest's recorded line
deterministically outrank it.
If Phase 3b (auxiliary fixture detection) found static JSON endpoints (markets, currencies, labels), recon:generate copies them to src/sites/<id>/fixtures/. Load them at module init via loadFixture() — zero per-request overhead, fails fast on deploy if the fixture is missing or stale:
import { z } from "zod/v4";
import { loadFixture } from "@/scraper/fixtures";
const MarketsSchema = z.array(z.object({ id: z.string(), name: z.string() }));
// Loaded synchronously at module init. Throws at startup if file is missing
// or shape drifted — surface fixture breakage on deploy, not on the first request.
const markets = loadFixture("my-site", "markets.json", MarketsSchema);See docs/playbook.md — Phase 3b for how fixtures are detected and when to use them.
Out-of-tree (recommended for operator-owned plugins): point BARNACLE_PLUGINS at the compiled plugin module — no core edits required:
BARNACLE_PLUGINS=./plugins/my-site/dist/index.js pnpm startBarnacle validates the export at startup and registers POST /v1/my-site/run automatically. See the Out-of-tree plugins env var table for BARNACLE_PLUGINS_STRICT and BARNACLE_PLUGINS_DIR. A copyable, runnable template lives at examples/plugins/hello-site/.
Config-only (no TypeScript, no compile step): a browser-flow plugin can be a single JSON manifest. Point BARNACLE_PLUGINS at a *.plugin.json file, or drop manifests into a directory named by BARNACLE_PLUGINS_CONFIG_DIR:
BARNACLE_PLUGINS=./plugins/acme-jobs.plugin.json pnpm start
# or, for directory-drop discovery of every *.plugin.json:
BARNACLE_PLUGINS_CONFIG_DIR=./plugins pnpm startThe manifest wears the Kubernetes-style apiVersion / kind / metadata / spec envelope, declares its request/response/extract shapes as JSON Schema, and lists the browser flow as data (the same self-heal step format the recon toolchain authors). Core reads it at startup and registers POST /v1/acme-jobs/run — no per-site code. A site needing the direct-HTTP hot path can reference a compiled executeHttp module via spec.httpModule. A copyable manifest lives at examples/plugins/acme-jobs.plugin.json.
The JSON Schema converter accepts a deliberately small subset — object, string, number, integer, boolean, array (with items), string enum, and required — and rejects anything else (e.g. pattern, minLength, $ref, format constraints) at load time. Flow steps interpolate request values with {{ .request.FieldName }}; a reference to a field the request schema does not declare fails loudly, while an optional declared field the caller omits splices as an empty string.
In-tree (bundled built-ins only): push to BUILTIN_SITE_PLUGINS in src/plugins/discover.ts:
import { mySitePlugin } from "@/sites/my-site";
import { BUILTIN_SITE_PLUGINS } from "@/plugins/discover";
BUILTIN_SITE_PLUGINS.push(mySitePlugin as SitePlugin<unknown, unknown>);Core registers POST /v1/my-site/run automatically at startup.
Add a step to .github/workflows/smoke.yml:
- name: Run smoke test — my-site
if: steps.check-secrets.outputs.skip == 'false'
run: |
pnpm run smoke -- \
--site my-site \
--payload '{"query":"test"}' \
--host "$SMOKE_HOST" \
--fallback \
--response-schema src/sites/my-site/contract.ts
env:
API_KEY: ${{ secrets.SMOKE_API_KEY }}
SMOKE_HOST: ${{ secrets.SMOKE_HOST }}
NODE_ENV: production--response-schema points to a module whose default export is a Zod schema. The smoke test validates the full response body against it — not just the envelope shape — so any schema drift on the data payload fails the pipeline immediately.
--fallback additionally runs a second request via the Stagehand browser path. This catches Stagehand cache staleness: if the page DOM changed and the cached selector now points at the wrong element, the hot-path test passes but the fallback test fails — alerting you before the fallback is invoked in production.
When the smoke test fails: re-run pnpm run recon:browser → diff <run-dir>/graphql/*<operationName>*.json against src/sites/<id>/contract.ts → update query / headers / Zod schema → ship. See docs/playbook.md for the full maintenance loop and change severity table.
dispatch() (src/plugins/loader.ts) tries executeHttp() first. Which errors trigger the browser fallback and which don't:
| Hot-path error | Status | Triggers browser fallback? | Reason |
|---|---|---|---|
HttpSchemaError |
Any | Yes | Response shape drifted; browser may still work |
HttpBotChallengeError |
401 / 403 | Yes | Residential proxy IP may get through |
HttpServerError |
5xx | Yes | Server-side outage; recovery strategy is the same |
HttpRateLimitError |
429 | No | A 429 means the configured rps ceiling is too high. Routing to the browser path would just hit the same ceiling and waste a Steel session. The right response is to lower the Bottleneck minTime in contract.ts and re-deploy. |
HttpUrlLockedError |
429 | No | A plugin's classifyResponseBody detected a terminal resource-lock sentinel — the URL is locked at the target's end. Neither a retry nor a browser session can succeed; the caller must back off and surface a "retry later" state. |
UnknownScraperError |
Any | No | Transient network failure or unclassified non-JSON response. createHttpClient retries up to 2 times internally; if all attempts fail, the error propagates as ScrapeFailureError. |
getCachedResponse() checks the LRU cache first. On a miss, getOrCreateInFlight() registers a promise in an inFlight map before awaiting it — meaning concurrent identical requests all await the same upstream call rather than fanning out. First caller wins; all others coalesce onto its promise.
Cache key: <endpoint>:<sha256(canonical payload)[:32]> — the endpoint is a literal prefix; the hash covers only the canonical payload. Object key order and primitive array element order are normalized so {a:1,b:2} and {b:2,a:1} hit the same entry. Default TTL: 15 minutes (CACHE_TTL_MS). Max entries: 1000 (CACHE_MAX_ENTRIES). Only successful responses are cached; errors propagate and never poison the cache.
runWithSession() (src/scraper/pool.ts) queues tasks through a p-queue bounded by SESSION_POOL_SIZE (default: 3). Sessions are created on demand — not pre-warmed — so Steel billing stays proportional to actual traffic.
Per-task hang ceiling: each queued task races against TASK_TIMEOUT_MS (src/scraper/pool.ts, 60 minutes by default). A hung execute() — frozen CDP connection, infinite network wait — converts to SessionTimeoutError, which the retry policy handles by tearing down the broken session and creating a fresh one. The default is sized for long browser flows; shorten per-plugin via SitePluginMeta.taskTimeoutMs. This is a hang-recovery floor, not a p99 latency budget.
Retry policy: withScraperRetry (src/scraper/retry.ts) uses p-retry with factor: 2, minTimeout: 500ms, maxTimeout: 5000ms, randomize: true, and default maxAttempts: 3. EmptyResultsError, CaptchaError, and StepVerificationError short-circuit retries (abort after the first attempt — a deterministic verification failure won't resolve by re-running the whole flow); SessionTimeoutError triggers a session restart before every retry attempt, not just the first.
Graceful shutdown: drainPool() is called during graceful shutdown — SIGTERM/SIGINT triggers app.close(), which fires Fastify's onClose hook, which calls drainPool(). It pauses new intake, waits up to 20 seconds for in-flight tasks to close their Steel sessions, then resolves. Without this, process exit leaves live sessions billing until Steel's own timeout.
createBrowserSession() (src/scraper/session.ts) picks a random desktop viewport per session from: 1280×720, 1366×768, 1440×900, 1920×1080. A fixed pixel size is an easy bot-detection fingerprint; rotating it makes sessions harder to cluster.
By default, Stagehand calls the Anthropic API directly (ANTHROPIC_API_KEY). Set USE_BEDROCK=true to route through AWS Bedrock instead:
USE_BEDROCK=true
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
BEDROCK_MODEL=us.anthropic.claude-sonnet-4-6[1m] # defaultThe [1m] suffix selects the 1-million-token context variant on Bedrock. Both paths run Stagehand with serverCache: true (server-side action cache to skip LLM inference on replay) and selfHeal: false (recon-browser owns its own verify-and-retry cascade; see src/scraper/session.ts for the rationale).
When using Anthropic directly (not Bedrock), the model is controlled by STAGEHAND_MODEL (default: anthropic/claude-sonnet-4-6).
GET /readyz returns readiness status plus per-site drift-detection metrics exposed by src/scraper/metrics.ts:
{
"status": "ready",
"checks": {
"database": { "ok": true },
"scraperCredentials": { "ok": true },
"scraperPool": { "ok": true, "detail": "depth=0" }
},
"stats": {
"scraperPool": { "size": 0, "pending": 0, "concurrency": 3 },
"cache": { "size": 12, "max": 1000, "inFlight": 0 },
"metrics": {
"my-site": {
"hotPathSuccess": 4821,
"fallbackActivations": 3,
"rateLimitRejections": 0,
"p95LatencyMs": 187
}
}
},
"telemetry": {
"currentRunFile": "/path/to/.barnacle/events/run-123.ndjson",
"currentRunFileSizeBytes": 4096,
"orphansRecovered": 0
},
"heal": {
"my-site": { "verdict": "SUCCESS", "bestPassRate": 0.95, "reportPath": "heal-out/my-site/healing-my-site.md" }
}
}What rising fallbackActivations means: the hot path is failing and the browser fallback is absorbing traffic. Cost and latency rise while error rate stays flat — users don't notice yet, but you will on your bill. This is your signal to re-run recon.
p95LatencyMs is reservoir-sampled (Vitter's Algorithm R, capped at 1000 samples) over actual upstream round-trips. Cache hits are excluded — they're memory reads and must not bias the upstream latency signal.
See docs/playbook.md for the full detection ladder.
Barnacle writes three append-only NDJSON files alongside its metrics:
| File | Default path | Purpose |
|---|---|---|
| LLM call samples | .barnacle/calls.ndjson |
One line per LLM/Stagehand call; feed to the judge:llm and slm-self-heal skills |
| Run event stream | .barnacle/events/<runId>.ndjson |
Per-run event stream written by the event-stream subsystem; path surfaced in /readyz telemetry.currentRunFile |
| Submission reconciliation records | .barnacle/submissions.ndjson |
One line per dispatch submit outcome or beacon-fire event; the durable, queryable join-key record — see Submission record schema below and GET /v1/submissions under Endpoints |
Every line in .barnacle/calls.ndjson is a JSON object with these fields (source: src/api/schemas/telemetry.ts):
| Field | Type | Description |
|---|---|---|
callId |
string |
UUID generated per call |
callType |
string |
Which LLM call site produced this sample — see table below |
model |
string |
Model identifier string passed to the SDK |
systemPrompt |
string | null |
System-prompt text, or null when absent |
userContent |
string |
Full user-turn content |
responseContent |
string | null |
Raw response text, or null on SDK error |
parsedOk |
boolean |
Whether the response was successfully parsed into the expected schema |
inputTokens |
number | null |
Input token count from SDK usage metadata |
outputTokens |
number | null |
Output token count from SDK usage metadata |
latencyMs |
number | null |
Wall-clock latency of the SDK call in milliseconds |
success |
boolean |
Whether the call site considered the call successful end-to-end |
ts |
string |
ISO-8601 timestamp at write time |
callType is a stable string constant defined in src/lib/telemetry/call-types.ts:
callType |
Source | When emitted |
|---|---|---|
recon-rephrase |
src/scripts/recon-browser.ts |
Attempt-5 rephrase inside the recon-browser step-healing cascade — the ai-SDK model (Anthropic-direct or Bedrock-backed) is asked to reword the failing step |
recon-replan |
src/scripts/recon-browser.ts |
Global replan after a step terminally fails — Claude rewrites the remaining flow tail |
recon-flow-patch |
src/scripts/recon-heal.ts |
Patch proposal from the recon-flow-patch-generator during the recon-heal self-healing loop |
llm-prompt-patch |
src/scripts/llm-heal.ts |
Patch proposal from the llm-call-patch-generator during the llm-heal self-healing loop |
Every "submit"-kind line in .barnacle/submissions.ndjson is a JSON object
validated against submissionEnvelopeSampleSchema
(src/lib/telemetry/submission-capture.ts, an alias of submitRecordSchema
in src/lib/telemetry/reconciliation-record.ts):
| Field | Type | Description |
|---|---|---|
kind |
"submit" |
Discriminates this record from a "beacon" conversion-event record sharing the same sink; defaults to "submit" so lines written before this field existed still parse. |
siteId |
string |
Which plugin handled the request — the cohort dimension for reconciliation. |
requestId |
string |
Fastify-issued correlation ID; joins a later "beacon" record to this one by matching requestId. |
joinKeys |
Record<string, unknown> | null |
Opaque, plugin-owned reconciliation join keys: the plugin's extractJoinKeys hook resolved once from the inbound payload, merged with anything the plugin attached mid-run via context.telemetry.addJoinKeys() (run-discovered keys win on collision — see Reconciliation join keys above); null when neither source produced anything. |
inboundPayload |
unknown |
The request body the caller posted, unredacted. |
status |
"submitted" | "error" |
Submit outcome. |
auditPayload |
unknown |
The plugin's SitePluginResult.auditPayload, or data when absent; null on errors. |
errorMessage |
string | null |
Failure message on errors; null on success. |
durationMs |
number |
Total dispatch wall time in milliseconds. |
ts |
string |
ISO-8601 timestamp at write time. |
session |
{ id, provider, ip, ipCapturedAt } | null |
Identity and outbound IP of the Browserbase session that served this run; null on the direct-HTTP hot path where no session is ever acquired. See Submission-envelope sink for capture details. |
A "beacon"-kind record shares the same sink to record a later, independent
beacon-fire outcome for the same requestId. Core writes one itself — either
a fired/failed line once fireTrackingClick resolves, or a skipped line
when there is no TrackingUrl to fire or the plugin declared
extractJoinKeys — but a plugin managing its own tracking nav can also emit
one directly via context.recordBeaconOutcome (see
Reconciliation join keys above).
See Submission-envelope sink
for the full schema and the GET /v1/submissions read path.
# Stream all LLM call samples as they arrive
tail -f .barnacle/calls.ndjson | jq '.'
# Filter to a specific call type
tail -f .barnacle/calls.ndjson | jq 'select(.callType == "recon-rephrase")'
# Show only failures
tail -f .barnacle/calls.ndjson | jq 'select(.success == false) | {callId, callType, latencyMs}'
# Token usage summary by call type
jq -s 'group_by(.callType) | map({callType: .[0].callType, totalInputTokens: map(.inputTokens // 0) | add, totalOutputTokens: map(.outputTokens // 0) | add, n: length})' .barnacle/calls.ndjson
# Tail the current run event stream (path from /readyz telemetry.currentRunFile)
tail -f .barnacle/events/<runId>.ndjson | jq '.'All variables are read once at process start. Required variables cause the process to exit on missing values; optional ones have safe defaults.
| Variable | Default | Required | Purpose |
|---|---|---|---|
APP_NAME |
barnacle |
No | Application name used in logs |
NODE_ENV |
development |
No | development / production / test |
PORT |
3000 |
No | HTTP listen port |
HOST |
0.0.0.0 |
No | HTTP listen address |
LOG_LEVEL |
info |
No | Pino log level (debug, info, warn, error) |
| Variable | Default | Required | Purpose |
|---|---|---|---|
API_KEYS_HASHED |
"" |
Yes (prod) | Comma-separated bcrypt hashes of plaintext bearer tokens. See Generating an API key. |
DEV_BYPASS_AUTH |
false |
No | Skip auth entirely. Local dev only — never set in production. |
| Variable | Default | Required | Purpose |
|---|---|---|---|
STEEL_API_KEY |
— | Yes | Steel account API key. Required for all browser automation. |
ANTHROPIC_API_KEY |
— | Yes (if not using Bedrock) | Anthropic API key for Stagehand's LLM calls. |
STAGEHAND_MODEL |
anthropic/claude-sonnet-4-6 |
No | Stagehand model. Use the anthropic/ prefix — Stagehand 2.x's model map is stale and the prefix routes through AI-SDK's fallback path. |
SCRAPER_PROXY_TYPE |
residential |
No | residential (paid Steel tiers) or none (free tier — Steel rejects useProxy=true on hobby plans). |
SCRAPER_SOLVE_CAPTCHA |
true |
No | Enable Steel's built-in CAPTCHA solver. Requires a paid plan; set false on the free tier. |
SESSION_POOL_SIZE |
3 |
No | Maximum concurrent Steel browser sessions. |
SCRAPER_MIN_ACTION_DELAY_MS |
500 |
No | Minimum delay between scraper actions (ms). Jitter applied on top. |
SCRAPER_MAX_ACTION_DELAY_MS |
1500 |
No | Maximum delay between scraper actions (ms). |
STAGEHAND_API_TIMEOUT_MS |
120000 |
No | Anthropic SDK request timeout (ms). Raise on slow network paths to api.anthropic.com. |
STAGEHAND_CONNECT_TIMEOUT_MS |
120000 |
No | TCP connect timeout for all outbound fetch calls (ms). Raised from the undici default of 10 s to match STAGEHAND_API_TIMEOUT_MS. |
STEEL_SESSION_TIMEOUT_MS |
3600000 |
No | Steel session wall-clock timeout (ms). Default is 1 hour; lower on plans that enforce shorter maximum session durations. |
FRAME_READY_TIMEOUT_MS |
20000 |
No | How long resolveFrameTarget polls for a child iframe to attach before falling back to the main frame (ms). Raise further for cross-origin OOPIFs that attach slowly under advancedStealth + proxied CDP. |
FRAME_DOCUMENT_READY_TIMEOUT_MS |
5000 |
No | How long waitForChildFrameReady polls a resolved child frame's document.readyState before proceeding anyway (ms). Independent of FRAME_READY_TIMEOUT_MS — this wait settles in well under a second once attached. |
FRAME_EVALUATE_TIMEOUT_MS |
30000 |
No | Watchdog budget for a single frame-scoped evaluate/candidate-probe call (ms), so a call against a racy frame fails the attempt instead of hanging indefinitely. |
FRAME_PRESENCE_PROBE_FLOOR_MS |
3000 |
No | Per-probe watchdog floor for probeAttachedFrameTarget's single non-polling presence check (ms) — a real budget a genuine CDP round-trip can land within, instead of the timeoutMs: 0 zero-budget pattern that always loses that race. |
SCRAPER_CAPTURE_SESSION_IP |
true |
No | Master switch for the outbound-IP echo navigation; false yields session: null / sessionIp: null everywhere without touching the rest of the submit/beacon record. |
SCRAPER_SESSION_IP_ECHO_URL |
https://api.ipify.org?format=json |
No | The IP-echo endpoint the session's own short-lived tab navigates to. Operators can point this at a self-hosted echo endpoint. |
SCRAPER_SESSION_IP_TIMEOUT_MS |
10000 |
No | Watchdog bound on the echo navigation; a page that never resolves is cut off and yields null rather than blocking the submission. |
Set USE_BEDROCK=true to route Stagehand's LLM calls through AWS Bedrock
instead of the Anthropic API. When enabled, ANTHROPIC_API_KEY is not needed.
AWS credentials resolve in standard SDK order: explicit vars → ECS task role →
EC2 instance profile → ~/.aws/credentials.
| Variable | Default | Required | Purpose |
|---|---|---|---|
USE_BEDROCK |
false |
No | Master switch — routes LLM calls through Bedrock when true. |
AWS_REGION |
us-east-1 |
No | AWS region for Bedrock calls. |
AWS_ACCESS_KEY_ID |
— | No | Explicit AWS access key (leave blank for ambient IAM). |
AWS_SECRET_ACCESS_KEY |
— | No | Explicit AWS secret key. |
AWS_SESSION_TOKEN |
— | No | Required only for temporary STS credentials. |
BEDROCK_MODEL |
us.anthropic.claude-sonnet-4-6[1m] |
No | Bedrock cross-region inference profile ID. The us. prefix enables automatic cross-region routing; the [1m] suffix selects the 1M-token context variant. |
| Variable | Default | Purpose |
|---|---|---|
CACHE_TTL_MS |
900000 (15 min) |
LRU response cache TTL. Cached responses skip the target API entirely. |
CACHE_MAX_ENTRIES |
1000 |
Maximum entries in the LRU cache. |
These limit traffic to Barnacle's own API. See per-plugin Bottleneck config
in each contract.ts for outbound rate limits to target sites.
| Variable | Default | Purpose |
|---|---|---|
RATE_LIMIT_MAX |
120 |
Max requests per window per API key (or IP for unauthenticated traffic). |
RATE_LIMIT_WINDOW_MS |
60000 (1 min) |
Rate limit window duration. |
TRUST_PROXY |
true |
Trust X-Forwarded-For when behind a reverse proxy. Set false for bare-metal deploys to prevent spoofing. |
| Variable | Default | Purpose |
|---|---|---|
READINESS_QUEUE_THRESHOLD |
20 |
/readyz returns 503 when scraper queue depth exceeds this. Lets orchestrators shed load before the pool is saturated. |
ENABLE_DOCS |
false |
Serve Swagger UI at /docs. Disable in production. |
APM tracing and DogStatsD metrics are opt-in: dd-trace and hot-shots are
optional peer dependencies, so a plain npm i @enricai/barnacle installs neither
and Barnacle runs without them. Enable either half independently — install the
package and flip its flag. If a flag is on but the package is missing, Barnacle
warns and carries on with that feature disabled; it never fails to boot.
# APM tracing
pnpm add dd-trace
DD_TRACE_ENABLED=true node --import dd-trace/initialize dist/server.js
# DogStatsD metrics
pnpm add hot-shots
DD_METRICS_ENABLED=true node dist/server.jsTracing needs --import dd-trace/initialize for full auto-instrumentation:
Datadog requires the tracer to load before any other module so it can patch
http/net/dns. Metrics have no such constraint.
| Variable | Default | Purpose |
|---|---|---|
DD_TRACE_ENABLED |
false |
Enable APM tracing. Requires the dd-trace peer dependency. |
DD_METRICS_ENABLED |
false |
Enable DogStatsD metrics. Requires the hot-shots peer dependency. Independent of DD_TRACE_ENABLED. |
DD_AGENT_HOST |
localhost |
Datadog agent hostname (the sidecar, in ECS Fargate). |
DD_DOGSTATSD_PORT |
8125 |
DogStatsD UDP port on the agent host. |
DD_SERVICE |
barnacle |
Service name tagged on spans and metrics. |
DD_ENV |
NODE_ENV |
Deployment environment tag. |
DD_VERSION |
0.1.0 |
Application version tag — git SHA or package version. |
| Variable | Default | Purpose |
|---|---|---|
TELEMETRY_ENABLED |
true |
Master switch — set false to disable all NDJSON telemetry writes. |
TELEMETRY_EVENTS_DIR |
.barnacle/events |
Directory for per-run NDJSON event stream files (<eventsDir>/<runId>.ndjson). |
CALLS_NDJSON_PATH |
.barnacle/calls.ndjson |
Append-only NDJSON sink for LLM/Stagehand call samples. One line per call; feed to the judge and self-heal skills. |
SUBMISSIONS_NDJSON_PATH |
.barnacle/submissions.ndjson |
Append-only NDJSON sink for dispatch submission envelopes and beacon-fire outcomes. kind:"submit" lines (null/"submit"-defaulted on legacy lines) capture siteId, requestId, inbound payload, status, audit payload, and duration, plus the opaque joinKeys bag a plugin's extractJoinKeys hook resolved, merged with anything attached mid-run via context.telemetry.addJoinKeys() — the durable source-of-truth for "what did we submit for jobId X and did it succeed." kind:"beacon" lines record a later (or, for beaconStatus: "skipped", immediate) independent beacon-fire outcome (beaconStatus: fired/failed/skipped, truncated trackingUrl) for the same requestId, so "submitted but the beacon did not fire" is measurable — the skipped line is always written by dispatch() itself, but a plugin managing its own tracking nav can call context.recordBeaconOutcome to append a real fired/failed line for the same requestId, which outranks skipped when the two are folded (see Reconciliation join keys). A reader folds both kinds together by requestId, so a plugin can join runs to its own attribution provider's report without re-parsing inboundPayload. |
TELEMETRY_MAX_FILE_SIZE_BYTES |
104857600 (100 MB) |
Rotate/drop the calls NDJSON once it exceeds this byte count. |
TELEMETRY_MAX_RETENTION_MS |
2592000000 (30 days) |
Drop event-stream files older than this many milliseconds. |
TELEMETRY_S3_BUCKET |
— | Optional — destination bucket for the buffered S3 telemetry mirror. Sink is entirely inert (no client, no network calls) when unset. Credentials/region resolve the same way as Bedrock (AWS_REGION, standard SDK credential order). |
TELEMETRY_S3_PREFIX |
telemetry |
Key prefix for uploaded NDJSON objects (<prefix>/<calls|submissions>/<date>/...). |
TELEMETRY_S3_FLUSH_INTERVAL_MS |
60000 |
How often buffered lines are flushed to S3. |
TELEMETRY_S3_MAX_BUFFER_LINES |
500 |
Threshold-flush trigger — flush early if either buffer exceeds this many lines, ahead of the next scheduled interval. |
TELEMETRY_S3_READ_MAX_OBJECTS |
200 |
Upper bound on the number of S3 objects a single reconciliation read-path query is allowed to scan. |
TELEMETRY_S3_READ_CONCURRENCY |
8 |
Max concurrent object fetches for a single reconciliation read-path query. |
| Variable | Default | Purpose |
|---|---|---|
JUDGE_MODEL |
us.anthropic.claude-sonnet-4-6[1m] |
Anthropic model used by the judge script. Reuses Bedrock creds via the cross-region inference profile. |
JUDGE_TEMPERATURE |
0.2 |
Sampling temperature for judge LLM calls. Keep low (≤ 0.3) for deterministic verdicts. |
JUDGE_BATCH_SIZE |
10 |
Number of call samples sent to the judge in one LLM request. |
JUDGE_TIMEOUT_MS |
120000 (2 min) |
Anthropic SDK request timeout for judge calls. |
| Variable | Default | Purpose |
|---|---|---|
SELFHEAL_MAX_ITERATIONS |
5 |
Maximum patch→replay→score iterations before BUDGET_EXHAUSTED. |
SELFHEAL_N_REPLAYS |
5 |
Number of replay runs per iteration arm. |
SELFHEAL_SUCCESS_THRESHOLD |
0.9 |
Minimum pass rate (0–1) to declare SUCCESS and stop iterating. |
SELFHEAL_PLATEAU_WINDOW |
3 |
Consecutive iterations below SELFHEAL_PLATEAU_DELTA that triggers PLATEAUED. |
SELFHEAL_PLATEAU_DELTA |
0.03 |
Minimum absolute pass-rate improvement per iteration to count as progress. |
SELFHEAL_TIMEOUT_MS |
60000 (1 min) |
Per-replay LLM request timeout. |
Set BARNACLE_SITE_<UPPERCASE_SITE_ID>_BASE_URL to override a plugin's
defaultBaseUrl without source changes. Underscores in the env key map to
hyphens in the siteId:
BARNACLE_SITE_MY_SHOP_BASE_URL="https://staging.my-shop.com" # overrides plugin `my-shop`| Variable | Default | Purpose |
|---|---|---|
BARNACLE_PLUGINS |
"" |
Comma-separated list of plugin specifiers to load at startup — relative paths (./plugins/acme) or package names (@acme/barnacle-plugin). Empty by default (built-ins only). |
BARNACLE_PLUGINS_STRICT |
false |
When true, any plugin that fails to load aborts the process instead of producing a disabled record. |
BARNACLE_PLUGINS_DIR |
process.cwd() |
Base directory used to resolve relative specifiers and locate the operator's node_modules. Defaults to wherever the binary is run — not the installed Barnacle package root. |
BARNACLE_PLUGINS_CONFIG_DIR |
(unset) | Directory scanned at startup for *.plugin.json config manifests, each loaded as a config-only plugin. Lets operators register sites by dropping a JSON file in a directory instead of editing BARNACLE_PLUGINS. An unreadable directory is logged and skipped — it never crashes boot. |
Resolution rule: a specifier starting with . or / is treated as a filesystem path resolved relative to BARNACLE_PLUGINS_DIR. Anything else is treated as an npm package name and resolved via require.resolve against the operator's own node_modules inside BARNACLE_PLUGINS_DIR.
Failure policy: by default (non-strict), a plugin that fails to load is logged at warn level and recorded as "disabled" in the load report — the server still boots with the remaining plugins. Set BARNACLE_PLUGINS_STRICT=true to abort startup on any load failure instead.
zod/v4 requirement for plugin authors: import Zod as import { z } from "zod/v4" in your plugin, not as bare "zod". Barnacle uses fastify-type-provider-zod which compiles routes against core's own zod instance; a plugin schema built against a different zod import may pass load-time validation but fail at route registration.
GET /v1/plugins (authenticated) returns the full plugin load report — one record per built-in and out-of-tree specifier — including siteId, displayName, route, specifier, resolvedPath, apiVersion, status ("loaded" or "disabled"), and an optional reason when disabled. Requires a valid Authorization: Bearer <token> header (reveals filesystem paths, so it is separate from the auth-free /healthz//readyz probes).
- Node.js 22+
- pnpm 10.4.1
- A Steel account (
STEEL_API_KEY) for managed browser sessions - An Anthropic key (
ANTHROPIC_API_KEY) for Stagehand's LLM calls, or AWS Bedrock (USE_BEDROCK=true+ AWS credentials) — see.env.examplefor details
pnpm install
cp .env.example .env # fill in STEEL_API_KEY and either ANTHROPIC_API_KEY or Bedrock credsBarnacle validates every request using bcrypt-hashed bearer tokens stored in
API_KEYS_HASHED. To create one:
# 1. Generate a random plaintext key — save this, you'll send it as Authorization: Bearer <key>
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# 2. Hash it (bcrypt cost factor 10) — paste the output into API_KEYS_HASHED
node -e "const b=require('bcryptjs');b.hash(process.argv[1],10,(e,h)=>console.log(h))" <your-key>Comma-separate multiple hashes in API_KEYS_HASHED to support key rotation.
For local development, set DEV_BYPASS_AUTH=true in .env to skip auth
entirely — never set this in production.
pnpm run devpnpm run build
pnpm startBarnacle boots with the built-in plugins registered (see BUILTIN_SITE_PLUGINS in src/plugins/discover.ts). Follow Adding a New Site above to build and register a plugin; core will register POST /v1/<your-siteId>/run automatically at startup.
With the dev server running (pnpm run dev), confirm the server is up:
curl -s http://localhost:3000/health | jq .Once a plugin is registered, every response follows the same envelope shape. The status block is always present; the plugin's responseSchema fields are spread alongside it at the root:
{
"status": {
"httpStatus": "OK",
"dateTime": "2025-05-16T12:00:00.000Z",
"details": []
},
"items": []
}The envelope is a flat merge, not nested — status lives at the root and the plugin's response fields are spread alongside it (src/api/helpers/envelope.ts:8-25). Parse as { status, ...pluginData }, not { status, data: pluginData }.
Every response — success or error — uses the same envelope shape so clients share a single parser. Error details appear in status.details[] with numeric codes:
| Code | Name | When |
|---|---|---|
| 1000 | PARTIAL_CONTENT_SUCCESS |
Partial data returned |
| 1001 | DECODING_ERROR |
Request body could not be parsed |
| 1002 | FIELD_VIOLATION |
Schema validation failure on a field |
| 1003 | EMPTY_REQUEST |
Request body was missing or empty |
| 1004 | AUTHORIZATION_ERROR |
Bearer token missing or invalid |
| 1005 | RESOURCE_NOT_FOUND |
Requested resource does not exist |
| 1006 | INDEX_NOT_FOUND |
Internal index lookup failed |
| 1007 | CLIENT_CALL_ERROR |
Downstream client call failed |
| 1008 | GENERIC_ERROR |
Unclassified server error |
| 1009 | EXTRA_DETAIL |
Supplemental detail entry (informational) |
| 1010 | THROTTLED_REQUEST |
Rate limit exceeded (hot path 429) |
| 1011 | TIME_OUT |
Request timed out |
| 2003 | SCRAPE_FAILURE |
Browser automation failed after retries |
| 2004 | CAPTCHA_ENCOUNTERED |
CAPTCHA challenge could not be resolved |
| 2005 | EMPTY_RESULTS |
Scrape succeeded but returned no data |
| 2006 | VERIFICATION_TRIGGER_FAILED |
OTP trigger to the target site failed |
| 2007 | RESUME_INVALID_OTP |
Provided OTP was rejected by the target site |
| 2008 | URL_LOCKED |
Upstream vendor locked the target URL; back off and retry later |
Full definitions: src/api/schemas/common.ts.
How scraper exceptions map to API codes (src/plugins/loader.ts:88-92):
CaptchaError→2004 CAPTCHA_ENCOUNTEREDEmptyResultsError→2005 EMPTY_RESULTSHttpRateLimitError→1010 THROTTLED_REQUEST(no browser fallback)HttpUrlLockedError→2008 URL_LOCKED(no browser fallback; distinct from rate-limit for metrics)- Any other
ScraperError→2003 SCRAPE_FAILURE - Task exceeded
TASK_TIMEOUT_MS→1011 TIME_OUT
Each registered plugin exposes a POST route following the default convention:
POST /v1/<siteId>/run. When the hot path detects that required applicant
answers are absent (e.g. Gender, Degree, EducationLevel, SignatureFullName) or
a repeat-applicant OTP challenge, /run returns HTTP 200 with
{ needsUserInfo: true, missingFields: [{ field, question }], requiresOtp }
instead of a submission result, so the caller can collect the gaps and hand back.
Plugins declare their own extra routes via meta.extraRoutes, which core registers
uniformly — the engine has no per-site knowledge. Route paths are declared as
:siteId templates, so the concrete path is whatever the plugin's siteId is. Two
conventional shapes a plugin may add:
POST /v1/<siteId>/resume— body = the full original candidate payload pluscollectedData(andotpCodewhere the site issues an OTP challenge); re-runs the hot path with the collected answers merged in; returns the same{ verified }envelope as/run, or2007 RESUME_INVALID_OTPif the OTP is rejectedPOST /v1/<siteId>/trigger-otp— body{ offerId, email }; asks the target site to email an OTP to a repeat applicant; returns{ success: true }or a2006 VERIFICATION_TRIGGER_FAILEDerror envelope
See examples/plugins/acme-jobs.plugin.json for a runnable declaration.
Operational routes:
GET /healthz— liveness probeGET /readyz— readiness probe (checks scraper credentials, queue depth)GET /docs— Swagger UI (whenENABLE_DOCS=true)GET /v1/plugins— authenticated plugin load report (see Out-of-tree plugins)GET /v1/submissions— authenticated, queryable submit+beacon reconciliation rows (filter bysiteId,requestId,status,beaconStatus,from/to; each row also carries the submit session blocksession({ id, provider, ip, ipCapturedAt }) and the beacon-firebeaconSessionIp, neither of which is filterable at this layer, same as the opaquejoinKeysbag; see Submission-envelope sink)
| Command | What it does |
|---|---|
pnpm run dev |
tsx watch --env-file=.env src/server.ts with hot reload |
pnpm run build |
compile to dist/ (tsc + path alias rewriting + copy src/sites/ fixtures and src/testing/fixtures) |
pnpm start |
node dist/server.js |
pnpm run typecheck |
strict TS noEmit |
pnpm run lint / lint:fix |
Biome |
pnpm run test |
Vitest unit + integration |
pnpm test src/scraper/fixtures.test.ts |
Run a single test file (NEVER use -- before the filter) |
pnpm run test:watch |
Vitest in watch mode (re-runs on file changes) |
pnpm run test:coverage |
Vitest with v8 coverage report |
pnpm run format |
Biome format write |
pnpm run recon:browser |
Phase 1 — drive browser + capture API calls |
pnpm run recon:http |
Phases 2–3 — replay, introspect, probe rate limits |
pnpm run recon:generate -- --site-id <id> |
Phase 4 — generate complete plugin from artifacts |
pnpm run recon:summarize -- --site-id <id> |
Phase 4 (optional) — write human-readable findings doc |
pnpm run recon:heal -- --site-id <id> --url <url> |
Self-heal a failing recon flow without modifying the source file |
pnpm run smoke -- --site <id> --payload '...' |
Phase 6 — run nightly drift-detection smoke test |
pnpm run judge:llm -- --calls-ndjson <path> --call-type <type> |
Score captured LLM calls on a three-dimensional rubric; writes a verdict JSON to judge-out/ |
pnpm run heal:llm -- --verdict-path <path> --call-type <type> |
Self-heal a failing prompt template: iterate patch→replay→score, write healing-<callType>.md with the best patch — production prompts are never modified |
src/
├── server.ts # Fastify bootstrap — calls loadAllPlugins(), registerRoutes(), site-agnostic
├── site-plugin.ts # SitePlugin<TInput,TOutput> interface (engine contract)
├── config.ts # frozen env-typed config singleton
├── plugins/
│ ├── loader.ts # dispatch(), registerRoutes(app, cfg, plugins)
│ └── discover.ts # BUILTIN_SITE_PLUGINS, loadAllPlugins(), loadPlugins()
├── sites/
│ ├── _shared/ # branch-local cross-plugin guards (coverage-expectations.test.ts)
│ └── <site-id>/ # one directory per registered plugin
├── api/
│ ├── plugins/ # auth, error-handler, request-context
│ ├── routes/ # health
│ ├── schemas/ # common envelope schemas; LLM telemetry + judge-verdict schemas
│ ├── helpers/envelope.ts # success envelope builder
│ └── errors.ts # error hierarchy + envelope builder
├── scraper/
│ ├── session.ts # Steel + Stagehand session factory
│ ├── pool.ts # p-queue over createBrowserSession
│ ├── throttle.ts # Bottleneck limiter + jitter
│ ├── retry.ts # p-retry + failure classification
│ ├── errors.ts # typed scraper error hierarchy
│ ├── http-client.ts # typed fetch wrapper (hot path)
│ ├── rate-limited-json-client.ts # factory: Bottleneck + chromiumClientHints + createHttpClient in one call — prefer this over the three-step scaffold for Chromium-hint plugins
│ ├── http-status-classifier.ts # pure status→ScraperError classifier for raw-fetch callers
│ ├── raw-fetch.ts # site-agnostic undici scaffold: network-error wrap, onResponse hook, optional classifyHttpStatus (skipClassify for callers that classify manually)
│ ├── graphql-client.ts # GraphQL POST wrapper
│ ├── metrics.ts # drift-detection counters
│ ├── fixtures.ts # static JSON fixture loader
│ ├── navigate.ts # shared awaitActivePage + goto(networkidle) helper
│ ├── behavioral-signals.ts # CDP synthetic mouse-move + scroll dispatcher for bot-detection warmup
│ ├── session-warmup.ts # generic pRetry browser-session runner: acquire → callback → close, with caller-supplied exhaustion mapping
│ ├── session-ip.ts # resolves a session's outbound IP via a throwaway tab + IP-echo navigation
│ └── require-response-field.ts # shared helpers for extracting required fields from HTTP response objects (HttpSchemaError on missing/null)
├── cache/
│ ├── response-cache.ts # lru-cache wrapper for deduplicating concurrent identical scraper requests
│ └── keyed-ttl-cache.ts # generic per-key TTL + single-flight coalescing cache factory
├── lib/ # logging, env, bedrock, db client, multipart, option-matcher, chromium-client-hints, telemetry/
├── scripts/ # recon-browser, recon-http, recon-generate, recon-summarize, recon-heal, recon-shared, smoke-test, judge-llm-batch, llm-heal
├── testing/
│ ├── integration-runner.ts # site-agnostic scaffold for integration tests (allocate inbox → dispatch → poll)
│ ├── mock-fetch-response.ts # shared undici-compatible Response stub factory for flow tests that mock fetch
│ ├── replay-integration-suite.ts # generic describe.skipIf/it.each scaffold; eliminates per-site integration boilerplate
│ ├── contract-parity-suite.ts # offline schema-parity scaffold; one-call drop-in for accept + rejection-case coverage
│ ├── coverage-guard-suite.ts # registry-driven structural guard; asserts contract.parity.test.ts exists per registered plugin
│ ├── batch-email-confirmation.ts # two-phase batch runner: submit jobs → poll inboxes (site-agnostic)
│ └── batch-report.ts # markdown table renderer for batch-test verdicts
└── types/
Library choices (battle-tested — no custom reinventions):
- API server:
fastify+ helmet + compress + rate-limit + swagger - Schema:
zodviafastify-type-provider-zod - Browser automation:
@browserbasehq/stagehand+steel-sdk - Concurrency:
p-queue,p-retry,bottleneck - Caching:
lru-cache - Logging:
pinowith CloudWatch 256KB splitting + sensitive-field redaction
Per-site base URL overrides: set BARNACLE_SITE_<UPPERCASE_SITE_ID>_BASE_URL to override a plugin's defaultBaseUrl without source changes. Underscores in the env key map to hyphens in the siteId (e.g. BARNACLE_SITE_MY_SHOP_BASE_URL → plugin my-shop).
Execution header: send x-barnacle-execution: browser on any plugin request to skip the hot path and go directly to the Stagehand browser path. Omit the header (or send any other value) to use the default hot path. Useful for debugging or when you know the hot path is broken. (Fastify lowercases incoming header keys; the dispatcher reads request.headers["x-barnacle-execution"] — supply lowercase to match.)
# .env (production)
NODE_ENV=production
ENABLE_DOCS=false # never expose Swagger in prod
TRUST_PROXY=true # set false if deploying directly to the internet (no ALB/nginx)
DEV_BYPASS_AUTH=false # this is the default — confirm it's not set to true
API_KEYS_HASHED="<bcrypt-hash>,<bcrypt-hash>" # at least one key
STEEL_API_KEY="..."
ANTHROPIC_API_KEY="..." # or USE_BEDROCK=true + AWS credsBarnacle is a plain Node.js process. Use pm2 or systemd to keep it alive and restart it on crash:
# pm2
pm2 start dist/server.js --name barnacle --env production
pm2 save && pm2 startup
# systemd (example unit)
[Service]
ExecStart=/usr/bin/node /srv/barnacle/dist/server.js
WorkingDirectory=/srv/barnacle
EnvironmentFile=/srv/barnacle/.env
Restart=on-failureRoute traffic through nginx or an Application Load Balancer (ALB). Set
TRUST_PROXY=true so Fastify uses X-Forwarded-For for the client IP
(needed for rate limiting on unauthenticated traffic).
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
}Wire /healthz as the liveness probe and /readyz as the readiness probe:
# Kubernetes example
livenessProbe:
httpGet: { path: /healthz, port: 3000 }
initialDelaySeconds: 5
readinessProbe:
httpGet: { path: /readyz, port: 3000 }
initialDelaySeconds: 10/readyz returns 503 when the scraper pool queue is saturated (depth >
READINESS_QUEUE_THRESHOLD) or when required scraper credentials are missing.
| Symptom | Cause | Fix |
|---|---|---|
Error: STEEL_API_KEY is required |
Missing env var | Add STEEL_API_KEY to .env |
useProxy rejected / 402 from Steel |
Free-tier plan doesn't support residential proxies | Set SCRAPER_PROXY_TYPE=none and SCRAPER_SOLVE_CAPTCHA=false |
401 Unauthorized on every request |
No API key configured or wrong plaintext key | Verify API_KEYS_HASHED is set; double-check the plaintext key. For dev, set DEV_BYPASS_AUTH=true |
Stagehand throws model not found |
Wrong model name format | Use the anthropic/ prefix: STAGEHAND_MODEL=anthropic/claude-sonnet-4-6 |
/readyz returns 503 on scraperCredentials |
STEEL_API_KEY or LLM key missing |
Set the missing credential |
Build succeeds but dist/sites/ is empty |
tsc ran but cp -r src/sites dist/sites was skipped |
Run pnpm run build (not tsc directly) — the build script copies site sources after compilation |
- Coding standards: CLAUDE.md
- Architecture & design rationale: docs/architecture.md
- Recon playbook (step-by-step): docs/playbook.md
- Testing guide: docs/testing.md
- Telemetry & LLM judging concept guide: docs/telemetry-and-judging.md
- Submission reconciliation runbook (join Barnacle runs to a plugin's own attribution provider's report): docs/submission-reconciliation.md
- Per-site recon findings: docs/target-recon.md (populated after first
pnpm run recon:summarize)
MIT © Enricai
Contributions welcome — see CONTRIBUTING.md.