Job Sources and Scraping

Job Sources and Scraping

A job source is a JobSource row of one of seven kinds: five ATS boards (Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee), a generic careers PAGE, or a web SEARCH. Each kind has a source adapter that turns the source into listings and, where the ATS allows it, reads a single job's ad from its public API. ATS boards are read through their public JSON APIs at no Firecrawl cost; pages, searches, resumes that are not plain downloads, and job ads with no API are read through Firecrawl, which runs in cloud, local or hybrid mode behind a scrape router. This page describes the adapters, URL detection, resume ingestion, job-page parsing, the Firecrawl client and rate limiting, and the hybrid routing rules. Credit budgeting and caching are on Firecrawl Credits and Caching; the run that drives all of this is on Run Pipeline.

Source kinds

SourceKind in prisma/schema.prisma: GREENHOUSE, LEVER, ASHBY, SMARTRECRUITERS, RECRUITEE, SEARCH, PAGE.

JobSource column	Used by	Meaning
url	all	The board or page URL; for searches a synthetic search:<lowercased query> so (userId, url) stays unique
kind	all	Picks the adapter (default PAGE)
adapterKey	ATS kinds	Board token or company slug
query	SEARCH	The search query
selector	PAGE	Optional CSS selector for the part of the page that lists jobs (not exposed in the UI)
maxPages	PAGE	Listing pages to follow, counting the first (default 3; not exposed in the UI)
useMap	PAGE	Read job links from Firecrawl's site map first
listingHash	PAGE	Hash of the page markdown (or mapped links) when listings were last read
suggestedKind, suggestedAdapterKey	PAGE	An ATS board found embedded in the page, offered as a switch
lastCheckedAt	all	Set after each successful listing read; null means never checked

When a source is added, SourcesService.create stores the kind and key returned by detectSource(url). Searches are added with createSearch, which collapses whitespace in the query. Changing a source's kind (SourcesService.update) clears the suggestion and listingHash and deletes the source's cached listings; a search source cannot be switched to another kind.

Adapter architecture

All adapters live in apps/api/src/matching/sources/.

The SourceAdapter interface

adapter.ts:

Member	Purpose
fetchListings(source, previous?)	Returns { listings, embedded?, listingHash? }. previous holds the source's last listings when the cache still has an expired copy; only the page adapter uses it. Fails with ScrapeFailed or FirecrawlBlocked.
fetchDescription?(listing)	Reads the job ad for a listing that came without one, instead of scraping its page
describes?(url)	Whether fetchDescription can read this URL; when absent, every URL is accepted

Adapters may use four services (AdapterServices): Firecrawl, HttpFetch, HostLimiter and ListingExtractor. Helpers in the same file:

• boardKey(source, label) returns adapterKey or fails with "Set the {label} board name for {url}."
• isEuSource(source) is true when the source host contains .eu.; Lever uses it to pick its EU API host.
• listingPage(listing) formats a listing that carries its ad as the text handed to the evaluator: # title, then company and locations joined by " · ", "Remote" when remote === true, then the description.

A Listing (firecrawl.ts) has title, url, company, locations, experienceMinYears, experienceMaxYears, remote, postedAt, and an optional description. remote and postedAt decode to null when absent, so listings cached before those fields existed stay readable.

Registry

registry.ts maps every SourceKind to its adapter (adapterFor(kind)). descriptionReader(kind, url) returns the adapter's fetchDescription when it exists and describes(url) (if defined) is true, otherwise null, in which case the job page is scraped.

How a job's ad is read

jobPageFor in page-cache.ts picks the cheapest route and records it as JobResult.scrapedVia:

Order	Route	scrapedVia	Cached
1	The listing already carries description	ATS	No (it is part of the cached listings)
2	descriptionReader exists for the listing's kind and URL: per-job ATS API	ATS (or CACHE on a hit)	Yes, as JOB_PAGE
3	Scrape: a PDF is downloaded and parsed locally, otherwise Firecrawl scrapeJob	LOCAL or CLOUD (null for a downloaded PDF; CACHE on a hit)	Yes, as JOB_PAGE

The kind used is the kind of the first source that listed the job. For a search source it is the ATS detected from the result URL (listingKind), so search results on Greenhouse, Lever and SmartRecruiters job URLs are read through those APIs.

URL detection

detect.ts: detectSource(url) returns { kind, adapterKey }. A URL that does not parse, is on no supported host, or yields no valid key becomes PAGE.

• A key must match ^[A-Za-z0-9][A-Za-z0-9_.-]*$ and must not be a reserved path segment (api, b, embed, external-referrals, meeting).
• A leading www. is ignored.
• Recruitee subdomains api, app, blog, docs, help, support and www are not boards.
• isAtsHost(hostname) is true for greenhouse.io, lever.co, ashbyhq.com, smartrecruiters.com, recruitee.com and their subdomains. The page adapter treats links to these hosts as same-site job links.

Supported ATS

ATS	URL patterns detected (key in braces)	Listings endpoint	Ad included with listings	Per-job endpoint
Greenhouse	boards.greenhouse.io/{token}, job-boards.greenhouse.io/{token} and their .eu. variants (any boards. or job-boards. host under greenhouse.io); boards.greenhouse.io/embed/job_board?for={token}; boards-api.greenhouse.io/v1/boards/{token} and boards-api.eu.greenhouse.io/v1/boards/{token}	GET https://boards-api.greenhouse.io/v1/boards/{token}/jobs?content=true (always the US host; there is no EU Job Board API host)	Yes (content, HTML-escaped HTML)	Yes: GET …/v1/boards/{token}/jobs/{id}?content=true, for URLs on (job-)boards(.eu).greenhouse.io/{token}/jobs/{id}. A job hosted on the company's own site (?gh_jid=) is scraped.
Lever	jobs.lever.co/{company}, jobs.eu.lever.co/{company}; api.lever.co/v0/postings/{company}, api.eu.lever.co/v0/postings/{company}	GET https://api.lever.co/v0/postings/{company}?mode=json (api.eu.lever.co when the source host contains .eu.)	Yes (descriptionPlain, lists, additionalPlain)	Yes: GET https://api(.eu).lever.co/v0/postings/{company}/{uuid}, for URLs jobs(.eu).lever.co/{company}/{uuid}
Ashby	jobs.ashbyhq.com/{board}; api.ashbyhq.com/posting-api/job-board/{board}	GET https://api.ashbyhq.com/posting-api/job-board/{board}?includeCompensation=true; jobs with isListed: false are dropped	Yes (descriptionPlain, plus "Compensation: …" when present)	No
SmartRecruiters	jobs.smartrecruiters.com/{company}, careers.smartrecruiters.com/{company}; api.smartrecruiters.com/v1/companies/{company}	GET https://api.smartrecruiters.com/v1/companies/{company}/postings?limit=100&offset={n}, up to 5 pages (the newest 500 postings)	No	Yes: GET https://api.smartrecruiters.com/v1/companies/{company}/postings/{id}, for URLs with path /{company}/{digits}
Recruitee	{company}.recruitee.com (excluding reserved subdomains)	GET https://{company}.recruitee.com/api/offers/	Yes (description and requirements, HTML)	No

Per-adapter details:

ATS	Company name	Locations	Remote	Posted	Experience
Greenhouse	company_name, else the token	location.name	true when "remote" appears in the location or metadata, else null	first_published, else updated_at	From the ad text
Lever	The board slug (the API has no company name)	categories.allLocations, else categories.location	From workplaceType (remote true; onsite, hybrid false)	createdAt	From the ad text
Ashby	The board name (the API has no company name)	location plus secondaryLocations, de-duplicated	isRemote	publishedAt	From the ad text
SmartRecruiters	company.name, else the slug	location.fullLocation, with empty comma parts removed	location.remote	releasedDate	Not extracted (null)
Recruitee	company_name, else the slug	location, else "city, country"	remote	published_at (format YYYY-MM-DD HH:MM:SS UTC)	From the ad text

SmartRecruiters' list endpoint returns no ad, so each selected job's ad is read from the posting endpoint later, rather than fetching every posting up front. Its listing URLs are built as https://jobs.smartrecruiters.com/{company}/{id}. The ad text joins the job description, qualifications, additional information and company description sections, each with its title.

Shared ATS helpers

http.ts: fetchJson(url, schema, { label, notFound }) waits for the host's turn on the HostLimiter (one request per 1.5 s per host, since public ATS APIs publish no limits), sends accept: application/json, and fails with ScrapeFailed:

Condition	Message
HTTP 404	The adapter's notFound, e.g. "Greenhouse board 'acme' not found."
Other non-2xx	"{label} returned HTTP {status}."
Body is not JSON	"{label} returned invalid JSON."
JSON does not match the Effect Schema	"{label} returned unexpected data: …"

text.ts:

• decodeEntities decodes numeric and a fixed set of named HTML entities.
• htmlToText drops script/style, turns block elements and <br> into line breaks and <li> into "- " items, strips the remaining tags and normalises whitespace.
• experienceFromText takes the first stated requirement: a range such as "3-5 years" or "3 to 5 yrs" (wins over an open bound), else "at least / minimum / min. N years", else "N+ years". Numbers have one or two digits, and anything over 40 years is ignored (a company's age or a typo).
• isoDate returns an ISO timestamp or null.

Page sources

page.ts handles any careers page.

1. Site map (optional). With useMap, it calls Firecrawl map(url, { search: "job", limit: 200 }), keeps links that pass isJobLink, titles them from their URL slug (slugTitle) and keeps usable titles. If any remain, they are the listings and listingHash is the hash of the sorted URLs. Otherwise it logs "Firecrawl's map has no job links for …" and falls back to the page.
2. Pages. It scrapes the source URL with scrapeListingPage (markdown and links; also raw HTML on the first check, when lastCheckedAt is null), then follows "next page" links until maxPages pages, a repeated URL, or a page whose markdown hash was already seen.
3. Unchanged pages. listingHash is the SHA-256 of the pages' markdown with whitespace collapsed. If the cache handed over expired listings (previous) and the hash equals the stored listingHash, the previous listings are reused without reading links or calling the model.
4. Listings. Job links with usable anchor titles are read from each page's markdown (findJobLinks). If none have titles, the LLM reads the pages (see Listing extraction). Company is the page's site name (og:site_name, or the last part of the title after |, –, — or -), else the host without www..
5. Embedded boards. On the first check only, detectEmbeddedBoard (embedded.ts) looks through the page's links and then every supported-ATS URL in its HTML (including protocol-relative script and iframe sources) and returns the first one detectSource recognises. The run saves it as suggestedKind/suggestedAdapterKey, and the sources page offers the switch.

Link rules

page-links.ts:

Rule	Detail
normaliseUrl	Absolute http(s) URL resolved against the page, without fragment or trailing slash
Same site	Same registrable domain (with co, com, org, net, ac, gov second-level handling for two-letter country TLDs), or an ATS host
Looks like a job	Path contains /job(s)/…, /position(s)/…, /opening(s)/…, /vacancy/…, /vacancies/… or /careers/<x>/<y>; or /careers/<slug> where the slug has a hyphen or digit and is not a section name (benefits, teams, life-at-… and others); or the last segment is a 4+ digit number or a UUID
Not navigation	Not /page/N, not a category, department, location, team, office, search or filter path, and no page, p, offset, department, location, category, team, office, q, query or search parameter; not the listing page itself
Usable title	3–120 characters, contains a letter, and not a generic label ("Apply now", "Learn more", "View job", "See all jobs" and similar)
Anchor title	The first line of the link text containing a letter, without images or markdown emphasis; a URL repeated as "Apply" keeps the earlier title
Next page	A same-host link labelled "Next", "Next page", "›", "»", ">" or "→", else the nearest later page of the source's own path (page/p/offset parameter or /page/N)
slugTitle	Title from the last non-ID path segment: extension, UUID suffix and 4+ digit words removed, words capitalised

Listing extraction

When a page's links carry no titles, listing-extractor.ts asks the model to extract listings from the markdown of all scraped pages (truncated to the first 60,000 characters, with an info log) using a strict JSON schema (job_listings, maxTokens 16,000). The page adapter keeps only returned URLs that actually appear among the pages' links, so invented or mangled links are dropped, and fills an empty company with the site name. A failure is wrapped as ScrapeFailed "Could not read the listings on {url}: …". The prompt and routing are described on AI Matching.

Search sources

search.ts runs one Firecrawl /search per read:

Option	Value
includeDomains	job-boards.greenhouse.io, job-boards.eu.greenhouse.io, boards.greenhouse.io, jobs.lever.co, jobs.eu.lever.co, jobs.ashbyhq.com, jobs.smartrecruiters.com, recruitee.com
tbs	qdr:w (past week)
limit	10
scrapeOptions	Not sent, so results are not scraped and cost only the search (2 credits on the cloud)

An empty query fails with "The search has no query." Results become listings when they are not ATS board roots (an ATS URL needs at least two path segments). Titles come from the result title, with Greenhouse's "Job Application for {title} at {company}" reduced to the title, else from the URL slug. Company is the detected ATS key, else the host. Locations and experience are unknown until the job page is read.

Searches run on the cloud in cloud and hybrid mode. In local mode they need FIRECRAWL_LOCAL_SEARCH=true (SearXNG on the self-hosted instance); otherwise the search fails with a ScrapeFailed naming that setting, and the run skips the source with a warning. Search listings are cached for 24 hours, so a search runs at most once a day per user and query (see Firecrawl Credits and Caching).

Resume ingestion

resume-source.ts resolves Settings.resumeUrl to { text, hash }, where hash is the SHA-256 of the extracted text. The resume is cached as a RESUME page-cache row holding hash, bytesHash (SHA-256 of the downloaded bytes), etag, lastModified and the text.

flowchart TD
  A["resolve(userId, url)"] --> B{"Cached row with future expiresAt?"}
  B -->|yes| R1["Return cached text"]
  B -->|no| C["GET directDownloadUrl(url) with If-None-Match / If-Modified-Since"]
  C -->|"network error or non-2xx"| FB{"URL ends in .pdf?"}
  FB -->|yes| ERR["ScrapeFailed: Could not download the resume"]
  FB -->|no| FC["Firecrawl scrapeMarkdown; cache 24 h"]
  C -->|"304 and cached"| R2["Reuse cached text"]
  C -->|"2xx"| D{"Same bytesHash as cached?"}
  D -->|yes| R3["Reuse cached text"]
  D -->|no| E{"Body starts with %PDF?"}
  E -->|yes| P["Parse PDF locally; cache with no expiry"]
  E -->|no| W["Firecrawl scrapeMarkdown; cache 24 h"]

Loading

Detail	Behaviour
Share links	download.ts directDownloadUrl rewrites Google Drive /file/d/{id} or ?id= links to https://drive.google.com/uc?export=download&id={id}, and Dropbox dl=0 to dl=1
PDF rows	Stored with no expiry, but revalidated with a conditional GET on every run; unchanged bytes are never parsed again
Web page rows	Expire after 24 hours, because pages carry nonces that change the bytes on every fetch
Firecrawl fallback	A non-PDF URL that cannot be downloaded directly (login wall, bot check) is scraped through Firecrawl; if that fails too, the message combines both errors
Downloads	30 s timeout for GET, 10 s for HEAD, redirects followed, 20 MB limit (by content-length and by body size)

PDF parsing

pdf.ts uses LiteParse with text output (which keeps two-column resumes readable). If the text is shorter than 200 characters, the PDF is parsed again with OCR and the longer result wins; an OCR failure keeps the plain text. A PDF with no text fails with "Could not read the PDF: the PDF has no text." isPdf checks the %PDF magic bytes. pdfOrMarkdown detects a PDF that Firecrawl returned as base64 (markdown starting with JVBERi) and parses it locally, because the SDK client sends parsers: [] so a PDF costs one credit rather than one per page.

Job-page parsing

job-posting.ts reads the schema.org JobPosting from a scraped page's raw HTML (scrapeJob requests markdown and rawHtml, which costs no extra credits). It scans every application/ld+json block, flattens arrays and @graph, skips invalid JSON, and takes the first node whose @type is JobPosting.

Field	Source
description	description, HTML converted to text (entity-decoded first if it contains no tags)
validThrough	ISO timestamp; a date-only value is valid to the end of that day, a time without offset is read as UTC
locations	jobLocation addresses (locality, region, country) or names, plus applicantLocationRequirements
remote	true when jobLocationType is TELECOMMUTE, else null
Experience	experienceRequirements.monthsOfExperience / 12 as minimum, else experienceFromText on its description or name

The pipeline uses it in three ways:

• isExpired(posting, now): a past validThrough settles the job as EXPIRED.
• withPosting(listing, posting): fills locations, remote and experience only where the listing has none.
• evaluationText(listing, page): the posting's own description (formatted with listingPage) when there is one, otherwise the page markdown cut to 24,000 characters.

Job pages are cached together with the parsed posting. Rows cached as plain markdown, from before structured data was read, are still accepted.

Firecrawl client

firecrawl.ts defines the Firecrawl service, the per-instance ScrapeBackend, and the SDK client (@mendable/firecrawl-js).

Service method	Request	Used for
scrapeMarkdown(url)	formats: ["markdown"]	Resumes rendered by Firecrawl
scrapeJob(url)	formats: ["markdown", "rawHtml"], chrome tags excluded	Job pages
scrapeListingPage(url, { selector, withHtml })	formats: ["markdown", "links"] (plus rawHtml when withHtml), chrome tags excluded, includeTags: [selector] when set	Page sources
map(url, { search, limit })	/map	Page sources with useMap
search(query, options)	/search without scraping	Search sources

Every scrape sends onlyMainContent: true, blockAds: true, removeBase64Images: true and parsers: []. Chrome tags are nav, header, footer, aside, form, script and style. A scrape that returns no markdown (for scrapeMarkdown and scrapeJob) fails with "Firecrawl returned no document markdown." The SDK's own retries are off (maxRetries: 1 counts attempts). The local backend uses a 60 s scrape timeout (65 s request timeout) and disables the SDK's auto-resume, so a hung local scrape falls back to the cloud in about a minute instead of five. The local client sends an empty API key so FIRECRAWL_API_KEY is never sent to the self-hosted instance.

Rate limiting and Retry-After

Each backend wraps its calls in a Limiter that reserves the next slot for a key before sleeping, so concurrent callers stay spaced.

Backend	Spacing	Key
Cloud	6,100 ms (ten a minute with a margin)	One key, "cloud", shared by scrapes, maps and searches
Local	2,000 ms	Per host; searches use the key "search"
Credit usage	Not rate limited	Not a scrape

On a 429 the backend pushes the key's next slot back by retryAfterMs(hint) and retries, up to 2 times. The SDK does not expose response headers, so the hint is the number in a "retry after Ns" phrase in the error message. retry-after.ts accepts seconds or an HTTP date, adds a 1 s margin, and falls back to 61 s when the value is missing or unparseable. (Discord alerts read the real Retry-After header with their own logic; see Run Pipeline.)

Scrape modes

FIRECRAWL_MODE is parsed by firecrawlConfigFrom in scrape-router.ts. Empty values count as unset. Invalid values throw at startup.

Mode	Requires	Scrapes and maps	Searches	Credit reserve	Health check
cloud (default)	FIRECRAWL_API_KEY	Cloud, with FIRECRAWL_CLOUD_PROXY (basic default, or auto)	Cloud	FIRECRAWL_CREDIT_RESERVE (default 50)	None
local	FIRECRAWL_LOCAL_URL (http/https)	Local only, no proxy, 0 credits	Local via SearXNG only with FIRECRAWL_LOCAL_SEARCH=true	None (no plan)	GET {url}/v0/health/readiness, 3 s
hybrid	Both	Local first, cloud fallback with the auto proxy	Cloud	FIRECRAWL_CREDIT_RESERVE	As local

A failed health check raises LocalFirecrawlDown before anything is scraped, so a stopped local instance stops the run once instead of failing every job.

Page classification

classify-scrape.ts judges a local scrape, and a hybrid run's cloud fallback, as one of five outcomes. Cloud-mode scrapes are not classified: only Firecrawl's own errors fail a page.

Outcome	When
ok	Page usable
blocked	The page shows the site kept the scraper out: target status 401, 403, 407, 429 or 503; a challenge marker ("just a moment", "cf-chl", "verify you are human", "access denied", "px-captcha" and others) in the title, the first 2,000 characters of markdown, or HTML under 30,000 characters; a vendor marker (captcha, recaptcha, hcaptcha, datadome, incapsula) on a page that otherwise looks like a challenge (challenge-like title or under 1,500 characters); or a job page with under 400 characters of markdown
unreachable	The instance gave no answer about the page: no response, timeout, API status 408, any 5xx, or the instance's own 401/403/407/429/503
gone	Target status 404 or 410
failed	Any other 4xx, or another API error

A map is judged only on errors, since it has no page content.

Hybrid routing

flowchart TD
  A["Scrape or map request for URL"] --> B{"ScrapeRoute.preferCloudUntil in the future for this domain?"}
  B -->|yes| C
  B -->|no| L["Send to local Firecrawl"]
  L --> O{"classifyScrape outcome"}
  O -->|ok| OK1["localWorked: reset localBlocks; return LOCAL, 0 credits"]
  O -->|"gone / failed"| F1["ScrapeFailed for this page"]
  O -->|blocked| NB["noteLocalBlock: localBlocks + 1; at 2, preferCloudUntil = now + 7 days"]
  O -->|unreachable| C
  NB --> C{"Run budget allows the minimum cost?"}
  C -->|no| F2["ScrapeFailed: needs cloud; credit reserve reached"]
  C -->|yes| S["Send to cloud with proxy auto"]
  S -->|"402"| F2
  S -->|"401 / 429"| FB["FirecrawlBlocked: stops the run"]
  S -->|"other error"| F3["ScrapeFailed for this page"]
  S -->|response| K{"classifyScrape on cloud page"}
  K -->|ok| OK2["Return CLOUD, record credits"]
  K -->|"not ok"| F4["ScrapeFailed for this page"]

Loading

Rules, from scrape-router.ts and scrape-routes.ts:

• Only a blocked outcome counts against the domain. An unreachable outcome goes to the cloud without counting, so a local instance that is down or failing does not send every domain to the cloud for a week.
• After localBlocksBeforeCloud (2) consecutive local blocks, the domain's preferCloudUntil is set to now plus preferCloudFor (7 days). The count is kept until a local success, so after the week one more block sends the domain back to the cloud.
• A local success resets the count and clears preferCloudUntil, writing only when there is a count to reset.
• The cloud is tried once per page. Failure messages say which scraper was blocked, e.g. "The site blocked the local scraper (HTTP 403)." or "The site served the cloud scraper a bot check or a page without its content."
• Out of credits (plan 402 or the run's budget), hybrid fails only the pages local cannot scrape, and the run goes on. A cloud 401 or 429 still raises FirecrawlBlocked, which stops the run.
• Searches in hybrid mode go straight to the cloud; a 402 or budget refusal becomes ScrapeFailed "needs cloud; credit reserve reached", and the run skips that search source.

The ScrapeRoute table is keyed by domain and shared by all users, since sites block the instance, not the user. All route reads and writes are best effort: a database error is logged and the page is routed as if the domain had no history (a failed block count returns 0, so it never prefers the cloud).

Local mode errors

In local mode every failure is a ScrapeFailed prefixed "Local Firecrawl:" or describing the block. A local instance has no credits or plan limits, so none of its errors stop a run; they fail the page (or, for a source, stop the run as any source-read failure does).

Note on routing.ts

routing.ts is not scrape routing. It reads the OpenRouter routing settings (OPENROUTER_SERVICE_TIER, OPENROUTER_PROVIDERS, price caps, reasoning effort); see AI Matching and Configuration Reference.

Fake providers

With PROVIDERS=fake, providers/fake.ts replaces Firecrawl, HTTP, PDF parsing, the model and Discord with fixtures. Missing fixtures fail with ScrapeFailed or MatchOutputInvalid, and some fixtures script specific errors (for example a 401 from the evaluator) for the end-to-end tests. See Testing.

Related pages

• Run Pipeline
• Firecrawl Credits and Caching
• AI Matching
• Configuration Reference
• Troubleshooting