Architecture

Job Matcher runs as three application processes (a Next.js web app, a NestJS HTTP API, and a BullMQ run worker) on top of PostgreSQL and Redis, and calls up to three external providers: Firecrawl for scraping, OpenRouter for LLM evaluation, and Discord for match alerts. The browser only ever talks to the web app, which proxies /api/* to the API. The API answers quickly and does no scraping or LLM work; it records a run and queues a job. The worker picks the job up, runs the matching pipeline (written with Effect), and writes progress to Postgres, which the run page polls. This page describes those pieces, how they are wired, and why.

System overview

flowchart LR
  browser(["Browser"])

  subgraph host["Docker Compose host"]
    web["Web<br/>Next.js (apps/web)"]
    api["API<br/>NestJS main.ts (apps/api)"]
    worker["Run worker<br/>NestJS worker.ts (apps/api)"]
    migrate["migrate<br/>prisma migrate deploy (one-off)"]
    pg[("PostgreSQL 17")]
    redis[("Redis 8<br/>BullMQ queue 'runs'")]
  end

  subgraph ext["External providers (PROVIDERS=live)"]
    firecrawl["Firecrawl<br/>cloud and/or self-hosted"]
    openrouter["OpenRouter"]
    discord["Discord webhook"]
    ats["ATS public APIs<br/>Greenhouse, Lever, Ashby,<br/>SmartRecruiters, Recruitee"]
  end

  browser -- "HTTP, one origin" --> web
  web -- "rewrite /api/* to API_URL" --> api
  api -- "Prisma" --> pg
  api -- "enqueue run, job schedulers" --> redis
  redis -- "deliver job" --> worker
  worker -- "Prisma" --> pg
  worker --> firecrawl
  worker --> openrouter
  worker --> discord
  worker --> ats
  api -. "retry-save alert only" .-> discord
  migrate --> pg

Loading

Component	Source	Runs as	Responsibility
Web	apps/web	next start (port 3000 in the container)	Pages, client-side data fetching, the /api/* rewrite, optimistic auth redirects in proxy.ts. The only service the production Compose file publishes.
API	apps/api/src/main.ts	bun src/main.ts (port API_PORT, default 4000)	Auth (better-auth), sources, settings, runs, matches, cache, overview, onboarding, health. Creates runs and queues jobs.
Run worker	apps/api/src/worker.ts	bun src/worker.ts (no HTTP port)	Processes runs queue jobs one at a time; handles schedule ticks; recovers interrupted runs at boot.
migrate	deploy/compose.yml	bunx --bun prisma migrate deploy, then exits	Applies pending migrations before the API and worker start.
PostgreSQL	postgres:17 image		All durable state: users and sessions, settings, sources, runs, results, matches, the page cache and scrape routes. See Data Model.
Redis	redis:8 image (append-only file enabled in production)		BullMQ queue runs and per-user job schedulers. No application data is cached in Redis.
Firecrawl	cloud API, or the self-hosted stack in the root docker-compose.yml (profile firecrawl)		Scrapes careers pages, job pages and the resume; runs web searches; reports cloud credits. See Job Sources and Scraping.
OpenRouter	cloud API		Model preflight, title triage, listing extraction and job evaluation. See AI Matching.
Discord	webhook URL from the user's settings		Receives an alert per saved match.

The API and worker share one Docker image (the api stage of the root Dockerfile); the worker service only overrides the command. The web image is the web stage. See Deployment.

Request path: browser to API

The web app rewrites every /api/:path* request to ${API_URL}/api/:path* (next.config.ts). API_URL defaults to http://localhost:4000 in development; the production image bakes in http://api:4000 as a build argument because Next.js resolves rewrites at build time.

Consequences of this design:

• The browser sees one origin, so the better-auth session cookie is first-party and the API needs no CORS configuration (none is enabled in app.ts).
• The client helper apps/web/lib/api.ts always calls relative /api${path} URLs.
• BETTER_AUTH_URL must be the origin people open in the browser (the web app), not the API. See Authentication.

The API sets the global prefix api and disables Nest's built-in body parser (bodyParser: false); the better-auth Nest integration installs its own JSON and URL-encoded parsers that skip /api/auth/*, so better-auth can read raw auth requests. See Backend API.

Run lifecycle: Analyze to results

sequenceDiagram
  autonumber
  actor User
  participant Web as Web (Next.js)
  participant API as API (RunsService)
  participant PG as PostgreSQL
  participant Q as Redis (BullMQ)
  participant W as Worker (RunsProcessor)
  participant P as Providers (Firecrawl, OpenRouter, Discord)

  User->>Web: Press Analyze with selected sources
  Web->>API: POST /api/runs {sourceIds}
  API->>PG: BEGIN, pg_advisory_xact_lock(hashtext(userId))
  API->>PG: check resume URL, source ownership, no QUEUED or RUNNING run
  API->>PG: INSERT Run (status QUEUED, provisional contextKey), COMMIT
  API->>Q: queue.add("run", {runId}, jobId = runId, attempts 1)
  API-->>Web: 201 Run (QUEUED)
  loop every 2 s while QUEUED or RUNNING
    Web->>API: GET /api/runs/:id
    API->>PG: Run with JobResults
    API-->>Web: Run detail
  end
  Q->>W: deliver job (concurrency 1)
  W->>PG: UPDATE Run SET RUNNING WHERE status = QUEUED (claim)
  W->>P: model preflight, local Firecrawl health check, credit allowance
  W->>P: resolve resume, scrape each source's listings (page cache first)
  W->>PG: Run.contextKey, found, eligible
  W->>P: title triage (optional)
  loop each selected job
    W->>P: read job page (ATS, cache, local or cloud Firecrawl)
    W->>P: evaluate against resume
    W->>PG: upsert JobResult, increment checked and failed, upsert Match
    W->>P: Discord alert for a saved match
  end
  W->>PG: UPDATE Run SET COMPLETED, STOPPED or FAILED
  Web->>API: GET /api/runs/:id
  API-->>Web: final status, polling stops

Loading

Notes on the flow, all from apps/api/src/runs:

• Idempotent queueing. The run id is the BullMQ job id (schedule.ts), so adding the same run twice does not create two jobs. If queue.add throws, the run is marked FAILED ("Could not queue the run") and the API answers 503, so a stuck QUEUED row never blocks later starts.
• One active run per user. create-run.ts takes a transaction-scoped Postgres advisory lock keyed by hashtext(userId) before checking for an active run, so two concurrent starts serialise. A conflicting start returns 409.
• Claiming. The worker only proceeds if it flips the run from QUEUED to RUNNING. A redelivered job for a run that is still RUNNING means the previous worker lost its lock; the run is ended as FAILED with the "interrupted" reason instead of being resumed (process-run.ts).
• Cooperative cancellation. Before each source and before triage the worker re-reads the run's status, and every job result is written in a transaction that only succeeds while the run is still RUNNING. A run ended by recovery therefore stops spending credits.
• Polling. apps/web/lib/runs.ts polls GET /api/runs/:id every 2 seconds until the status is no longer QUEUED or RUNNING. The home page's overview query refetches every 60 seconds (apps/web/lib/overview.ts).

The full pipeline, including filters, triage, credit checks and error handling, is on Run Pipeline.

Scheduled runs

Saving settings with a non-null scheduleMinutes (minimum 15) upserts a BullMQ job scheduler named user:<userId> that repeats every scheduleMinutes * 60 000 ms and enqueues a job named scheduled with { userId } (schedule.ts). Setting it to null removes the scheduler. Each tick, handled by the same worker:

1. Removes the scheduler if the user's settings no longer have a schedule.
2. Ends any of the user's runs that lost their job (see below).
3. Creates a SCHEDULED run over all of the user's sources, through the same createRun checks. A tick is skipped while a run is active or before the user has a resume and a source.
4. Queues an ordinary run job, so a run is never left QUEUED without a job.

At boot the worker re-applies every user's scheduler from the Settings table, in case a settings save stored a schedule while Redis was down (runs.processor.ts).

Recovering orphaned runs

A run whose BullMQ job is gone, completed or failed can never be delivered again and would block every later run. recover-run.ts ends such runs as FAILED and keeps their saved results:

Trigger	Scope
Worker boot (onApplicationBootstrap)	All users
Each schedule tick	That user
POST /api/runs when a run is already active	That user, bounded to 2 s; if anything was recovered, the start is retried once
BullMQ failed event for a run job	That run ("interrupted" if the job had stalled, otherwise "The run failed unexpectedly")

QUEUED runs younger than 60 seconds are left alone, because the API saves the run before it adds the job.

Process boundaries

main.ts and worker.ts are two entry points into the same codebase, each with its own Nest module graph.

	API	Worker
Entry point	main.ts calls createApp() from app.ts, then listen(API_PORT)	worker.ts calls NestFactory.createApplicationContext(WorkerModule): no HTTP server
Root module	app.module.ts	worker/worker.module.ts
Imports	PrismaModule, RedisModule, AuthModule, QueueModule	PrismaModule, RedisModule, QueueModule
Providers	Controllers and services for every HTTP resource, plus MatchAlerts	EffectRuntime, RunsProcessor
Uses Effect runtime	No (only MatchAlerts runs a single Effect with Effect.runPromise)	Yes, a ManagedRuntime built at boot
Talks to providers	Discord only, when a failed match save is retried	Firecrawl, OpenRouter, ATS APIs, Discord
Shutdown	enableShutdownHooks()	enableShutdownHooks()

Both processes load config.ts at import time, so both require the full base environment (DATABASE_URL, API_PORT, BETTER_AUTH_SECRET, BETTER_AUTH_URL, REDIS_URL) even though the worker serves no HTTP and does no authentication. The worker does use BETTER_AUTH_URL: it builds the run-page links in Discord alerts (run-page-url.ts). Only the worker with live providers additionally requires OPENROUTER_API_KEY and the Firecrawl settings for the selected FIRECRAWL_MODE (providers/live.ts, scrape-router.ts); a missing key stops the worker at startup, not the API.

Shared infrastructure modules:

• PrismaModule is global and exports PrismaService, a PrismaClient built with the @prisma/adapter-pg driver adapter.
• RedisModule is global and exports RedisService, an ioredis client with maxRetriesPerRequest: null so commands wait for a reconnect rather than fail (BullMQ workers need this).
• QueueModule configures BullMQ on that Redis connection and registers the runs queue.

The Effect and Nest boundary

Nest controllers and services are plain async TypeScript. Effect is used where the failures are: scraping, rate limiting, retries, evaluation and notifications. The two meet in one class, worker/effect-runtime.ts:

• EffectRuntime is a Nest @Injectable() that builds a ManagedRuntime from the provider layer (fake or live) merged with two Prisma-backed layers: the page cache store and the scrape route store.
• onModuleInit awaits runtime.context(), so all layers are built once when the worker boots. As a result, the Firecrawl rate limiters live for the life of the process and are shared by every run.
• RunsProcessor.process calls effect.runPromise(processRun(prisma, runId, model)); the run's Effect program requires the RunServices union declared in process-run.ts.
• onApplicationShutdown disposes the runtime.

Inside the pipeline, every provider failure is a tagged error from matching/errors.ts, and stopsRun decides what each does:

Error	Effect on the run
OpenRouterAuthError, OpenRouterUnavailable, ManagementKeyError, ModelUnsupported, FirecrawlBlocked	Stops the whole run (STOPPED, with the error message as stopReason)
OpenRouterRequestFailed, ScrapeFailed, MatchOutputInvalid, SaveFailed	Fails only the current job (recorded on its JobResult)
NotifyFailed	Logged only; the match is already saved

Any defect that escapes the program ends the run as FAILED with "The run failed unexpectedly".

Provider swapping

PROVIDERS (live or fake, default live) selects which layer EffectRuntime builds:

	PROVIDERS=live (providers/live.ts)	PROVIDERS=fake (providers/fake.ts)
Firecrawl	Routed per FIRECRAWL_MODE (cloud, local or hybrid)	Fake backends per FIRECRAWL_MODE, serving Markdown fixtures from apps/api/test/fixtures; pages with ?fake-blocked block the fake local instance
Plain HTTP (ATS APIs, PDFs)	Real fetch	Always fails, so job pages fall back to Firecrawl fixtures
Resume	Scraped or downloaded, cached	Fixture resume.md for every URL
Listing extraction	OpenRouter	listings.json fixture
Title triage	OpenRouter	Every title scores 3 (kept, in order)
Evaluation	OpenRouter, after a model preflight	A Verdict: line in the fixture page scripts the outcome (match, auth, invalid, otherwise no match)
Discord	Real webhook POST	Logs the alert; rejects non-Discord URLs
Required keys	OPENROUTER_API_KEY, plus Firecrawl settings	None

The fake layer also scripts errors: a URL with ?fake-error=<status> fails the scrape, and 401, 402 or 429 behave as a blocked Firecrawl. MatchAlerts in the API makes the same choice for the one alert the API sends. Demo mode (the installer's default) and the Playwright suite both run on PROVIDERS=fake; see Testing.

Design decisions and trade-offs

Decision	Rationale	Cost
A queue, not the request	A run scrapes several pages and makes one LLM call per job, which can take minutes. The API answers immediately with a QUEUED run.	Needs Redis, a second process, polling, and the orphaned-run recovery described above.
Postgres for queue-adjacent invariants	Run state, the single-active-run rule (advisory lock) and claiming live in Postgres, so Redis losing a job cannot corrupt state; it can only orphan a run, which recovery ends.	Recovery logic in three places (boot, tick, start).
Effect only where failures are	Typed errors make each provider failure's consequence explicit; Nest code stays conventional.	Two programming models in one codebase, joined at EffectRuntime.
Cache in Postgres (PageCache)	Survives restarts, needs no extra service, can be cleared per user from Settings.	Expired rows are overwritten on the next fetch; nothing sweeps them.
Work keyed by contextKey	Results are unique per user, context and job URL, so a rerun with the same resume, sources, model and requirements skips jobs already settled.	Changing any input starts a fresh context and re-evaluates.
One web origin	First-party cookies, no CORS, only one published port.	BETTER_AUTH_URL must exactly match the address users open.
Fake providers as a layer	Demo mode and end-to-end tests exercise the real pipeline code with no keys or cost.	Fixtures must be kept in step with provider contracts.

Scaling limits

• One worker, concurrency 1. @Processor(runsQueue, { concurrency: 1 }) processes one job at a time, so runs never interleave and share one in-process Firecrawl rate limiter. Running a second worker process would give it its own, independent limiters; more throughput would first need a shared, Redis-backed limiter.
• Rate limits are in memory. Requests to the Firecrawl cloud are spaced 6.1 s apart, requests to a self-hosted Firecrawl 2 s apart per host, and ATS API requests 1.5 s apart per host (firecrawl.ts, sources/http.ts).
• Bounded run size. Each run evaluates at most maxJobsPerRun jobs (1 to 20), and scheduled runs also keep to a daily share of cloud credits. See Firecrawl Credits and Caching.
• Multi-user, single tenant. Every row is scoped by userId, but the app is designed as a personal tool: anyone who can reach the instance can sign up, and all users share one worker queue.
• The API is stateless apart from Postgres and Redis, so more API replicas would work behind the web proxy, but the Compose stack runs one.

Related pages

• Backend API: modules and every endpoint.
• Authentication: sessions, cookies and the web proxy.
• Data Model: tables, relations and migrations.
• Run Pipeline: what the worker does with a run, step by step.
• Deployment: the production Compose stack.