Run Pipeline

Run Pipeline

A run is one pass over a user's job sources: read the resume, collect listings, filter them, and evaluate the remaining jobs against the resume with an LLM. The API only creates the run row and queues a BullMQ job; a single worker process (concurrency 1) does the slow work inside an Effect runtime. This page follows a run from the HTTP request to the Discord alert. It covers the state machine, the typed error model that decides whether a failure stops the run, fails one job or is only logged, recovery of interrupted runs, and how a rerun resumes without paying twice. Scraping is covered in Job Sources and Scraping, credit budgeting and caching in Firecrawl Credits and Caching, and evaluation prompts and triage in AI Matching.

Source files

File	Role
runs/create-run.ts	Creates a run row inside a transaction guarded by a per-user advisory lock
runs/runs.service.ts	Manual start, run listing, retrying a failed match save
runs/runs.controller.ts	/api/runs routes
runs/runs.queue.ts	Queue name, job payload types, scheduler id
runs/schedule.ts	enqueueRun and syncSchedule
runs/runs.processor.ts	BullMQ processor: run jobs, schedule ticks, boot-time recovery
runs/process-run.ts	The pipeline itself, as one Effect program
runs/recover-run.ts	Ends runs that lost their queue job
runs/run-settings.ts	Maps the Settings row to requirements and title filters
runs/save-match.ts	Upserts a Match, tolerating older URL spellings
runs/match-alerts.ts	Discord alerts sent from the API (retried saves)
runs/run-page-url.ts	Absolute link to a run's page
worker.ts, worker/worker.module.ts	Worker entry point (a Nest application context, no HTTP server)
worker/effect-runtime.ts	The shared ManagedRuntime
matching/errors.ts	Tagged errors and stopsRun
matching/dedupe.ts	Canonical URLs and listing de-duplication
matching/requirements.ts	Location and experience filters
matching/title-filters.ts	Include/exclude title keywords
matching/discord.ts	Discord webhook notifier

Overview

flowchart TD
  A["POST /api/runs"] --> B["createRun: transaction + pg_advisory_xact_lock"]
  B -->|"NoResume / SourceNotFound / AlreadyActive"| X["HTTP 400 / 409"]
  B -->|Created| C["enqueueRun: queue.add('run', jobId = runId)"]
  S["Scheduler tick (every N minutes)"] --> S1["recoverOrphanedRuns for the user"]
  S1 --> S2["createScheduledRun (all sources)"]
  S2 --> C
  C -->|"add failed"| F1["Run FAILED: Could not queue the run"]
  C --> D["Worker (concurrency 1): claim QUEUED to RUNNING"]
  D -->|"not QUEUED"| F2["failActiveRun (interrupted)"]
  D --> E["Load settings, sources"]
  E --> P1["OpenRouter preflight"]
  P1 --> P2["Local Firecrawl health check"]
  P2 --> P3["Read Firecrawl credits, compute allowance"]
  P3 --> P4["Cloud mode: estimate source credits"]
  P4 --> R["Resolve resume"]
  R --> L["Read each source's listings"]
  L --> DD["dedupeListings"]
  DD --> FL["rulesOutJob + acceptsTitle"]
  FL --> K["Replace context key, write found / eligible"]
  K --> SK["Drop jobs already settled in this context"]
  SK --> T["Title triage (if enabled)"]
  T --> M["Take maxJobsPerRun; scheduled: fitJobs to today's credits"]
  M --> J["For each job: read page, expiry, acceptsJob, evaluate"]
  J --> SV["Save match, record JobResult, increment counters"]
  SV --> N["Discord alert (failure only logged)"]
  N --> J
  J --> Z["Run COMPLETED"]
  P1 & P2 & P3 & P4 & R & L -->|"error"| ST["Run STOPPED with the error message"]
  J -->|"stopsRun error"| ST

Loading

Creating a run

createRun (in create-run.ts) runs in one Prisma transaction:

1. SELECT pg_advisory_xact_lock(hashtext(userId)) serialises starts per user, so two near-simultaneous requests cannot both pass the active-run check. The lock is released when the transaction ends.
2. Source ids are de-duplicated. The user must have a resumeUrl in Settings, and every source id must belong to the user (an empty list also counts as not found).
3. If the user has a run in QUEUED or RUNNING, the result is AlreadyActive.
4. The Run row is created with a provisional contextKey that uses the resume URL in place of the resume content hash. The worker replaces it once the resume has been read (see Firecrawl Credits and Caching).
5. The first run also sets Settings.onboardingCompletedAt if it is still null.

createRun returns a tagged result, which RunsService.start maps to HTTP responses:

Result	Response
Created	The run row, after the job is queued
NoResume	400 "Add a resume URL in settings before starting a run"
SourceNotFound	400 "Source not found"
AlreadyActive	409 "A run is already in progress"
Queue add failed	503 "Could not queue the run. Try again shortly."

When the first attempt returns AlreadyActive, start first tries to end runs that lost their queue job (recoverOrphanedRuns, bounded by recoveryTimeoutMs = 2 s so a Redis outage cannot hold the request) and, if any were ended, tries createRun once more.

Routes

All routes are under the global /api prefix and need a session (see Authentication).

Method and path	Purpose
GET /api/runs	The user's 20 most recent runs
GET /api/runs/:id	One run with its JobResult rows, oldest first
POST /api/runs	Start a manual run; body { sourceIds: string[] } (validated by the shared StartRun schema)
POST /api/runs/:id/results/:resultId/save	Save a match whose first save failed (see Saving matches)

Queueing

The queue is named runs (runs.queue.ts) and carries two job types:

Job name	Payload	Added by	Options
run	{ runId }	enqueueRun	jobId: runId, attempts: 1
scheduled	{ userId }	A BullMQ job scheduler with id user:<userId>	every: minutes * 60000, attempts: 1, removeOnComplete: true, removeOnFail: 100

Run id as job id. enqueueRun passes the run id as the BullMQ job id. BullMQ ignores an add whose job id already exists, so the same run cannot be queued twice. Overlapping runs for one user are prevented separately, by the advisory lock and the active-run check.

A run that cannot be queued is failed. If queue.add throws, enqueueRun calls failActiveRun(…, "Could not queue the run"). Otherwise the QUEUED row would block every later start.

Scheduled re-checks

syncSchedule(queue, userId, minutes) upserts the user's job scheduler, or removes it when minutes is null. It is called:

• by SettingsService.update whenever settings are saved (scheduleMinutes must be null or at least 15, per the shared UpdateSettings schema);
• by RunsProcessor.onApplicationBootstrap for every user with a schedule, which restores schedulers if a settings save reached Postgres while Redis was down;
• by a tick whose user no longer has a schedule, which removes the scheduler.

On each scheduled tick the processor:

1. Re-reads Settings; with no scheduleMinutes it removes the scheduler and returns.
2. Calls recoverOrphanedRuns for that user, so a run that lost its job does not hold up the schedule.
3. Calls createScheduledRun, which creates a SCHEDULED run over all of the user's sources. If createRun returns anything other than Created (no resume, no sources, a run already active), the tick does nothing.
4. Queues an ordinary run job with enqueueRun. The tick never processes the run itself, so a run is never left QUEUED without a job.

Processing

The worker and the Effect runtime

worker.ts boots WorkerModule as a Nest application context (Prisma, Redis, the queue, EffectRuntime and RunsProcessor). The processor is declared with @Processor("runs", { concurrency: 1 }), so jobs are handled one at a time.

EffectRuntime builds a single ManagedRuntime at module init from either fakeProvidersLayer (PROVIDERS=fake) or liveProvidersLayer(OPENROUTER_MODEL), merged with the Postgres-backed PageCacheStore and ScrapeRouteStore layers. Building it once at boot means stateful services, in particular the Firecrawl rate limiters and the ATS host limiter, are shared by every run. The runtime is disposed on application shutdown. processRun requires these services (RunServices): Firecrawl, PageCacheStore, HttpFetch, HostLimiter, ListingExtractor, PdfParser, ResumeSource, ScrapeHealth, FirecrawlPlan, MatchEvaluator, TitleTriage and Notifier.

Stages of processRun

processRun(prisma, runId, model) creates the run's credit tracker (makeRunCredits) and provides its budget as RunCloudBudget, so every cloud Firecrawl request in the run is checked and counted (see Firecrawl Credits and Caching). The stages are:

1. Claim. UPDATE … SET status = RUNNING WHERE id = runId AND status = QUEUED. If no row changes, the job is a redelivery: BullMQ delivers a job again only after the worker holding it lost its lock, so a run still RUNNING was interrupted. It is ended with failActiveRun rather than resumed (resuming could analyse the same jobs twice); a run already COMPLETED, STOPPED or FAILED is left alone.
2. Load. Run, settings and sources. A missing resume URL ends the run STOPPED with "Add a resume URL in settings before starting a run".
3. Prepare (any error here stops the run, see below):
   1. evaluator.preflight: checks the OpenRouter key and that the model has an endpoint under the price cap (AI Matching).
   2. scrapeHealth.check: in local and hybrid mode, GET {FIRECRAWL_LOCAL_URL}/v0/health/readiness with a 3 s timeout; failure raises LocalFirecrawlDown ("Local Firecrawl is not running at …").
   3. credits.start: reads the cloud plan's remaining credits once and computes the allowance. For scheduled runs it first sums what the user's other scheduled runs spent since UTC midnight. The plan's credits are stored in Run.cloudCreditsAtStart.
   4. In cloud mode only, the source read costs are estimated up front; if they exceed the allowance the run stops with the credit refusal message.
   5. resumeSource.resolve: downloads or scrapes the resume and returns its text and content hash.
   6. For each source in run.sourceIds order: check the run is still RUNNING, read its listings through the page cache, remember which source kind first listed each canonical URL (this decides how the job ad is read later), and update the source's lastCheckedAt, listingHash and any embedded-ATS suggestion.
   7. dedupeListings over all sources.
4. Filter. Listings are kept when !rulesOutJob(requirements, listing) && acceptsTitle(titleFilters, listing.title).
5. Context key. The real context key is computed from the resume hash and written to the run with found (listings after de-duplication) and eligible (after filtering).
6. Skip settled jobs. Jobs that already have a JobResult in the same context with error = null are dropped. With triage off, results skipped by triage (skipped = TRIAGE) do not count as settled, so those jobs are checked.
7. Triage (when triageEnabled and jobs remain). The kept jobs come back ranked; rejected jobs are upserted as JobResult rows with skipped = TRIAGE, isMatch = null and a reason starting "Skipped by title triage". A triage failure that does not stop the run is logged as a warning and the run continues with the untriaged list.
8. Select. The first maxJobsPerRun jobs (1–20, default 5). A scheduled run then keeps only the jobs whose estimated cloud cost fits what is left of today's allowance (fitJobs; free jobs always fit) and logs how many it dropped.
9. Per job, in order:
   1. jobPageFor reads the ad: from the listing itself, from the ATS API, from the cache, or by scraping (records scrapedVia).
   2. If the page's schema.org validThrough is in the past, the verdict is skipped = EXPIRED.
   3. The listing is enriched with the page's structured data (withPosting) and checked with acceptsJob. A job now ruled out gets isMatch = false with "Doesn't meet your requirements (from the job page's structured data)"; a job whose structured data omits location or experience while includeUnknown is off gets isMatch = false with a reason saying so.
   4. Otherwise evaluator.evaluate returns isMatch and a reason.
   5. A match is saved with upsertMatch (a failed save raises SaveFailed).
   6. In one transaction, only if the run is still RUNNING: increment checked, increment failed when the job errored or its save failed, update cloudCredits, and upsert the JobResult. If the run is no longer RUNNING, the worker stops.
   7. If the match was saved and the user has a Discord webhook, send the alert. NotifyFailed is caught and logged as a warning.
10. Finish. COMPLETED, with cloudCredits and finishedAt.

finish(status, reason) only updates a run whose status is still RUNNING, so a run already ended by recovery keeps its own reason. The whole program is wrapped in Effect.onError, which ends the run FAILED with "The run failed unexpectedly" on any unhandled failure or defect (for example a database error inside Effect.promise).

Stop checks

Before each source read and before triage, stillRunning re-reads the run's status. A worker whose run was ended by recovery (while the worker itself was still alive) stops there, so it spends no more credits and does not run alongside a newer run. The per-job transaction applies the same rule before writing each result.

Run and job-result states

Run status

RunStatus in prisma/schema.prisma has five values: QUEUED, RUNNING, COMPLETED, STOPPED, FAILED. RunTrigger is MANUAL or SCHEDULED.

stateDiagram-v2
  [*] --> QUEUED: createRun
  QUEUED --> RUNNING: worker claims the run
  QUEUED --> FAILED: queue add failed
  QUEUED --> FAILED: job lost (older than 60 s, recovery)
  QUEUED --> FAILED: BullMQ job failed event
  RUNNING --> COMPLETED: all selected jobs processed
  RUNNING --> STOPPED: prepare-phase error or stopsRun error
  RUNNING --> FAILED: unexpected error
  RUNNING --> FAILED: interrupted (redelivery, recovery, stalled job)
  COMPLETED --> [*]
  STOPPED --> [*]
  FAILED --> [*]

Loading

Status	stopReason examples
COMPLETED	null
STOPPED	The error message, e.g. an OpenRouter auth failure, "Local Firecrawl is not running at …", "Firecrawl credit reserve reached (N left until YYYY-MM-DD)", or "Add a resume URL in settings before starting a run"
FAILED	"The run failed unexpectedly", "Could not queue the run", "The run was never picked up. Start a new run.", or "The run was interrupted when its worker stopped. Results saved so far are kept; start a new run to check the rest."

Job results

A JobResult has no status column. Its outcome is encoded in isMatch, skipped (SkipReason: EXPIRED, TRIAGE), error and matchSaved, and scrapedVia (ScrapeVia: ATS, CACHE, LOCAL, CLOUD, or null when the page was not read or a PDF was downloaded). Rows are unique on (userId, contextKey, jobUrl), so a rerun in the same context updates the row rather than adding one.

stateDiagram-v2
  [*] --> TriageSkipped: triage rejects title
  [*] --> Reading: selected for this run
  Reading --> Errored: page read or evaluation failed
  Reading --> Expired: validThrough in the past
  Reading --> NotMatch: ruled out by structured data or model says no
  Reading --> Match: model says yes
  Match --> MatchSaved: upsertMatch succeeds
  Match --> SaveFailed: upsertMatch fails
  SaveFailed --> MatchSaved: POST retry save
  Errored --> Reading: next run in the same context
  TriageSkipped --> Reading: next run with triage off

Loading

Outcome	isMatch	skipped	error	matchSaved	Counted in failed	Settled for the context
Triage skipped	null	TRIAGE	null	false	no (not counted in checked either)	yes while triage is on
Expired	null	EXPIRED	null	false	no	yes
Not a match	false	null	null	false	no	yes
Match saved	true	null	null	true	no	yes
Match, save failed	true	null	null	false	yes	yes (saved later from the run page)
Errored	null	null	message	false	yes	no, retried by the next run

Match status

A saved Match starts as PENDING; marking it applied sets APPLIED. Finding the same job again keeps its status (see Saving matches).

Typed error model

Every expected failure in the matching pipeline is a tagged error (Data.TaggedError) declared in errors.ts. MatchingError is their union, and stopsRun(error) is an exhaustive switch that says whether an error ends the run. Because the switch has no default branch, adding a new tag fails type-checking until its behaviour is decided.

What processRun does with an error

Where the error occurs matters as much as its tag:

Where	Behaviour
Prepare phase (preflight, health check, credit read, resume, source listings)	Any error stops the run (STOPPED, message as stopReason), because nothing can be analysed without the resume and listings. Exception: a SEARCH source whose error does not satisfy stopsRun is skipped with a warning and the run reads its other sources.
Title triage	stopsRun errors stop the run; other errors are logged and the run continues untriaged.
Per job (page read, evaluation)	stopsRun errors stop the run; other errors are recorded on the job's JobResult.error, counted in failed, and the run moves on.
Match save	SaveFailed never stops the run; the verdict is kept with matchSaved = false.
Discord alert	NotifyFailed is caught and logged.
Anything unhandled (defects)	Effect.onError ends the run FAILED with "The run failed unexpectedly".

So a ScrapeFailed on a job page fails only that job, but a ScrapeFailed while reading a page or ATS source's listings stops the run.

Error reference

Error	Fields	Raised where	stopsRun	Effect on the run
OpenRouterAuthError	status, message	Any OpenRouter 401: the preflight's /key and /models/…/endpoints calls (preflight.ts) and completions (completion.ts), via openRouterErrorFromStatus	true	Stop run
OpenRouterUnavailable	status, message	OpenRouter 402, 403, 404 or 429 (openRouterErrorFromStatus); a completion 404 or 503 whose message says no endpoint meets the routing or price cap ("No flex endpoint under the price cap …")	true	Stop run. Exception: a completion 429 is first retried as a capacity error (below) and, once retries run out, becomes OpenRouterRequestFailed
OpenRouterRequestFailed	status (nullable), message	Other OpenRouter statuses (e.g. 5xx); no HTTP response; an unexpected preflight response body; flex capacity still unavailable after 5 retries	false	Fail job (or skip triage)
ManagementKeyError	message	Preflight, when the key is a management or provisioning key	true	Stop run
ModelUnsupported	model, message	Preflight, when the model has no endpoint that supports JSON Schema structured output, is from the allowed providers, offers flex (when flex is configured and the endpoint data shows it), and is under the price cap	true	Stop run
FirecrawlBlocked	status, message	Cloud Firecrawl 401, 402 or 429 (firecrawlErrorFromStatus in scrape-router.ts); a cloud-mode request refused by the run's credit budget (status 402, credit message); the up-front source estimate in process-run.ts	true	Stop run. In hybrid mode a 402 (plan or budget) is converted to ScrapeFailed "needs cloud; credit reserve reached"
ScrapeFailed	status (nullable), message	Other Firecrawl errors and pages classified as blocked, gone or failed; local Firecrawl errors (never stop a run); ATS API non-2xx, 404, invalid JSON or unexpected data (sources/http.ts); missing board key; empty search query; non-ATS URL given to a per-job endpoint; resume download failures; PDF parse failures; listing extraction failures wrapped by the page adapter; downloads over 20 MB	false	Fail job; stops the run if raised while reading a non-search source or the resume
MatchOutputInvalid	message	Model refusal, empty content, schema mismatch, no choices, or finish_reason other than stop (match-output.ts, completion.ts, triage.ts, listing-extractor.ts)	false	Fail job (or skip triage)
SaveFailed	message	saveMatch in process-run.ts when upsertMatch throws	false	Job recorded with matchSaved = false, counted in failed; can be retried from the run page
NotifyFailed	message	discord.ts: URL is not a Discord webhook, network error, non-2xx, or a 429 without a short enough Retry-After	false	Log only

Two tagged errors live outside errors.ts and are not part of MatchingError:

Error	Where	Behaviour
LocalFirecrawlDown	scrape-router.ts, from the health check	Raised in the prepare phase, so it stops the run
InvalidRequirements	requirements.ts, from makeRequirements validation	Not raised by processRun

OpenRouter capacity retries

In completion.ts, a completion that fails with status 429 or 503 (and is not the price-cap case) is retried on an exponential schedule starting at 1 minute, up to 5 times (1, 2, 4, 8, 16 minutes). If it still fails, it becomes OpenRouterRequestFailed ("OpenRouter flex capacity unavailable after 5 retries: …"), which fails only the current job. Because the worker has concurrency 1, these waits delay the whole queue.

Progress reporting

Progress lives on the Run row; there is no push channel.

Field	Written when
status	Claim, finish, recovery
cloudCreditsAtStart	After the credit read (null when there is no cloud plan, i.e. local mode)
found, eligible, contextKey	Once listings are read, de-duplicated and filtered
checked, failed, cloudCredits	After each job, in the same transaction as its JobResult
stopReason, finishedAt, cloudCredits	At finish

The web app's useRun hook (apps/web/lib/runs.ts) polls GET /api/runs/:id every 2 seconds while the run is QUEUED or RUNNING and stops polling once it ends. Triage-skipped rows appear among the results without incrementing checked.

Recovery

recover-run.ts handles runs whose queue job is gone, which would otherwise block every later run for that user.

failActiveRun(prisma, runId, reason) sets FAILED, the reason and finishedAt, but only if the run is still QUEUED or RUNNING. Saved results are kept, and the next run in the same context skips them.

recoverOrphanedRuns(queue, prisma, { userId?, now? }) looks at every RUNNING run and every QUEUED run older than queueGraceMs (60 s; a younger queued run may still be getting its job). For each, it fetches the BullMQ job with the run id: if the job is missing (unknown), completed or failed, the run is failed with lostJobReason (was QUEUED) or interruptedReason (was RUNNING). Jobs that are still active or waiting are left to BullMQ's stall handling.

Trigger	Scope
Worker boot (onApplicationBootstrap)	All users; errors are logged and do not stop the worker
Schedule tick	The tick's user
Manual start that hit AlreadyActive	The requesting user, bounded to 2 s
Redelivered run job (claim found the run not QUEUED)	That run
BullMQ failed worker event	That run: interruptedReason if the job stalled (stalledCounter > 0), else "The run failed unexpectedly"

Run settings

run-settings.ts maps Settings columns to the pipeline's inputs:

Pipeline field	Settings column	Default
requirements.experienceYears	experienceYears	0
requirements.experienceMonths	experienceMonths (0–11)	0
requirements.jobMinYears	targetMinYears	null
requirements.jobMaxYears	targetMaxYears	null
requirements.locations	locations	[]
requirements.includeUnknown	includeUnknown	true
titleFilters.include	titleInclude	[]
titleFilters.exclude	titleExclude	[]

processRun also reads maxJobsPerRun (1–20, default 5), triageEnabled (default true), resumeUrl and discordWebhookUrl directly. Requirements and title filters are part of the context key; maxJobsPerRun, triageEnabled and the webhook are not.

Filters

De-duplication

dedupe.ts:

• canonicalUrl lowercases the host (via URL), removes the fragment, removes tracking parameters (utm_*, ref, source, gh_src, case-insensitive), and strips trailing slashes except on the root. Text that is not a URL is returned unchanged.
• dedupeListings keeps the first listing (in source order) for each canonical URL and for each key of normalised company, normalised title and sorted lowercase locations. normaliseText lowercases and collapses everything but letters and digits to single spaces, so "Engineer, Backend" and "engineer - backend" agree. Kept listings carry their canonical URL.

Because the first source to list a URL wins, it also decides how that job's ad is read (kindByUrl in process-run.ts; for a search source, the ATS detected from the result URL).

Location and experience requirements

requirements.ts defines acceptsJob(requirements, job):

Check	Rule
Location	When the user lists locations: if the job has locations, at least one user location must appear as a whole word (Unicode-aware, case-insensitive) in at least one job location. If the job has none, it passes only when includeUnknown is true.
User experience	The user's years are experienceYears + experienceMonths / 12. A job whose minimum exceeds that is rejected, regardless of includeUnknown.
Target range	When jobMinYears or jobMaxYears is set: a job with no stated experience passes only if includeUnknown; a job whose minimum exceeds jobMaxYears, or whose maximum is below jobMinYears, is rejected.

rulesOutJob(requirements, job) is !acceptsJob({ ...requirements, includeUnknown: true }, job): missing details never rule a job out. The pipeline uses rulesOutJob on listings (which often omit details the job page states) and the strict acceptsJob only after the page's structured data has been merged in.

Title filters

title-filters.ts: a title passes when it contains at least one include keyword (or there are none) and no exclude keyword. Keywords match as whole words, case-insensitively, with any whitespace between the words of a phrase; blank keywords are ignored. canonicalTitleFilters (lowercased, trimmed, unique, sorted) is used only for the context key.

Saving matches

upsertMatch in save-match.ts first looks for an existing match of the user with the exact URL, then one whose stored URL has the same canonical form (matches saved before URLs were canonicalised). If found, only reason and matchedAt are updated, so the application status (PENDING/APPLIED) is kept. Otherwise it upserts on (userId, jobUrl) with the listing's title, company, locations and experience bounds.

If the save fails, the JobResult keeps isMatch = true with no error and matchSaved = false, and the run's failed counter is incremented. The job is not analysed again. POST /api/runs/:id/results/:resultId/save (RunsService.retrySave) saves it from the stored result without scraping or evaluating:

• 404 if the result does not exist for the user and run; 409 if it is not a match; no-op if already saved.
• The flag flip matchSaved: false → true and the failed decrement happen in one transaction guarded by matchSaved = false, so two quick retries decrement once.
• Only the request that flipped the flag sends the Discord alert that the run would have sent.

Alerts

discord.ts posts one embed per saved match:

Aspect	Behaviour
Validation	The URL must pass the shared isDiscordWebhookUrl, otherwise NotifyFailed
Payload	Embed titled "New job match", linked to the job URL, colour 0x5865f2; description is "title at company", the reason, then a link "Open the run in Job Matcher". The text is cut to 4,000 characters with the run link always kept last. allowed_mentions: { parse: [] } so scraped text never pings anyone.
Request	POST, JSON, 10 s timeout, redirect: "error"
Rate limit	A 429 is retried after the Retry-After header (seconds), at most 3 attempts in total and only when the wait is 30 s or less
Errors	Only the error's name is kept in the message, because a message could include the webhook URL, whose path holds the token

The worker sends alerts through the Effect runtime's Notifier. The API's MatchAlerts service (match-alerts.ts) sends the alerts for retried saves, using the fake notifier when PROVIDERS=fake. Both only log NotifyFailed.

Run page URLs

runPageUrl(runId) returns new URL("/runs/<encoded id>", BETTER_AUTH_URL).href. BETTER_AUTH_URL is the origin users open the app on, so links in Discord point at the web app's run page (apps/web/app/(app)/runs/[id]/page.tsx). See Configuration Reference.

Concurrency and rate limiting

Limit	Value	Where
Worker concurrency	1 job at a time	@Processor(runsQueue, { concurrency: 1 })
Active runs per user	1 (QUEUED or RUNNING)	createRun, under pg_advisory_xact_lock
Cloud Firecrawl	One request per 6.1 s across all runs (about 9.8 a minute, under the free plan's 10)	cloudLimiter in firecrawl.ts
Local Firecrawl	One request per 2 s per host; searches share a "search" key	localLimiter
Firecrawl 429	Back off by the "retry after Ns" hint in the error message (plus 1 s), or 61 s without one; up to 2 retries	makeBackend, retry-after.ts
ATS public APIs	One request per 1.5 s per host	HostLimiter in sources/http.ts
OpenRouter capacity	429/503 retried after 1, 2, 4, 8, 16 minutes	completion.ts
Discord 429	Up to 2 retries, waits of at most 30 s	discord.ts

The limiters are in-memory Refs inside the single ManagedRuntime, so they are correct only because there is one worker process. Running more workers would need a shared (Redis-backed) limiter.

Related pages

• Job Sources and Scraping: adapters, hybrid routing, page classification
• Firecrawl Credits and Caching: credit guard, page cache, context key
• AI Matching: preflight, triage, evaluation
• Data Model: Run, JobResult, Match, PageCache, ScrapeRoute
• Architecture