Scanner Providers

Scanner Providers

As of v1.136.0 the scanner ships 79 adapters — 74 EN-region + 5 RU. All scanning is in-process and zero-token: adapters hit public job-board APIs/feeds directly over HTTPS; no LLM is involved and no external script is spawned.

The two registries (read this first)

Adding an EN board touches two registries:

1. Source meta registry — server/lib/sources/<slug>.mjs with an export const meta = { value, label, region, configKey? } block. server/lib/sources/registry.mjs readdirSync-scans the folder at boot and dynamically import()s every *.mjs (P-14, v1.69.0), so dropping a valid file auto-registers it (malformed meta → skipped with one console.warn). This drives GET /api/scan/sources and the #/scan dropdown.
2. Fetch-walk registry — server/lib/portals/adapters/<slug>.mjs plus its entry in server/lib/portals/registry.mjs::ALL_ADAPTERS (asserted === 74 with the exact sorted id list by tests/adapter-registry.test.mjs). Adapters implement matches()/buildEndpoint() (string-or-null contract) and the fetch/parse walk over tracked_companies: in portals.yml.

RU sources additionally need a RU_DISPATCH row in server/lib/ru-scanner.mjs, keyed by the meta's configKey.

EN adapters (74)

Kind: board-wide = one global feed searched by keyword · per-tenant = per-company slug/board (driven by tracked_companies:) · single-company = one fixed employer's careers backend.

id (value)	Label	Kind	Notes
greenhouse	Greenhouse	per-tenant	boards-api.greenhouse.io/v1/boards/<slug>/jobs
ashby	Ashby	per-tenant	api.ashbyhq.com/posting-api/job-board/<slug>
lever	Lever	per-tenant	api.lever.co/v0/postings/<slug>; v1.118.0: detects EU boards jobs.eu.lever.co → api.eu.lever.co
workable	Workable	per-tenant
smartrecruiters	SmartRecruiters	per-tenant
workday	Workday	per-tenant	Reports workdayFallback in scan results; v1.119.0: browser-like headers (UA/accept-language/origin/referer derived from the CXS URL) clear Cloudflare-gated tenants
recruitee	Recruitee	per-tenant
teamtailor	Teamtailor	per-tenant	Feed-based
personio	Personio	per-tenant
breezy	Breezy HR	per-tenant
bamboohr	BambooHR	per-tenant
rippling	Rippling	per-tenant	ats.rippling.com/<slug>
pinpoint	Pinpoint	per-tenant
comeet	Comeet	per-tenant	careers-api URL
avature	Avature	per-tenant
successfactors	SAP SuccessFactors (multi-brand RMK brand paths preserved — v1.125.1)	per-tenant
jibeapply	JibeApply (iCIMS)	per-tenant	Paginated (max 50 pages) — v1.117.0
softgarden	softgarden	per-tenant	Cap 1000 jobs — v1.117.0
csod	Cornerstone	per-tenant	25/page pagination — v1.118.0
phenom	Phenom	per-tenant	100/page widget API — v1.118.0
radancy	Radancy	per-tenant	Branded hosts carry no vendor token → provider:-selected only, endpoint-shape checked — v1.118.0; legacy TalentBrew markup + JSON results-fragment transport — v1.130.0
beesite	beesite (GJB)	single-company	jobs.mercedes-benz.com backend, 100/page — v1.117.0
deutschebahn	Deutsche Bahn	single-company	db.jobs (anchored host regex) — v1.118.0
tkms	TKMS	single-company	jobs.tkmsgroup.com — v1.118.0
hecklerkoch	Heckler & Koch	single-company	v1.118.0
meituan	Meituan	single-company	zhaopin.meituan.com JSON API; per-entry keywords/max_pages, empty-page retry — v1.119.0
tencent	Tencent	single-company	careers.tencent.com Query API; per-entry keywords/max_pages — v1.119.0
oraclecloud	Oracle Cloud (ORC)	per-tenant	recruitingCEJobRequisitions REST API of *.fa[.<region>][.ocs].oraclecloud.com tenants (JPMC, Oracle, BNY Mellon, Amex, …); siteNumber from careers_url, WAF-aware UA — v1.123.0
wttj	Welcome to the Jungle	board-wide	public Algolia index (wttj_jobs_production_en), rotating creds bootstrapped from /api/env; explicit wttj: { queries: [...] } config required — v1.124.0
agenticjobs	Agentic Jobs	board-wide	agentic-engineering-jobs.com server-rendered listing, parsed in-process — v1.124.0
jobvite	Jobvite	per-tenant	public XML feed app.jobvite.com/CompanyJobs/Xml.aspx?c=<companyEId>; companyEId from company_eid: config → c= of an explicit api: → board-page discovery. Two hosts SSRF-pinned (jobs.jobvite.com discovery + app.jobvite.com feed), no redirect followed. Migrated off the retired JSON API — v1.154.0
gem	Gem	per-tenant	public GraphQL batch endpoint jobs.gem.com/api/public/graphql/batch, ?board= per company — v1.124.0
alibaba	Alibaba	single-company	talent.alibaba.com/position/search JSON API (double-submit CSRF, no login); per-entry keywords/max_pages — v1.124.0
flowxtra	Flowxtra	board-wide	no-auth cross-tenant aggregator app.flowxtra.com/api/central/jobs (paginated) — v1.129.1
vdab	VDAB	board-wide	Flanders public employment service vindeenjob JSON API (www.vdab.be), keyword-driven, recall-first — v1.129.1
icims	iCIMS	per-tenant	hosted-portal search pages careers-<tenant>.icims.com/jobs/search, auto-detected from any *.icims.com careers_url (distinct from jibeapply) — v1.129.1
a16z-speedrun-talent	a16z Speedrun	board-wide	a16z Speedrun talent-network board feed; host-pinned speedrun-talent-network.com JSON API, HTTPS-only, 0-indexed pagination + page cap, per-company q/config threading — v1.130.0; adapter override host-pinned (exact host, HTTPS) with fallback to the canonical feed — v1.131.1
cryptocurrencyjobs	Cryptocurrency Jobs	board-wide	Web3 board cryptocurrencyjobs.co via public RSS 2.0 (zero-auth); two-pass XML-entity decode, remote-only, employer from the "… at <Company>" title tail; parser cleanUrl uses the exact-match host guard (no subdomains) — v1.131.1
manfred	getManfred	board-wide	Spanish/EU tech board with published salaries; host-pinned www.getmanfred.com/api/v2/public/offers JSON (zero-auth), single-request full catalogue, salary + remote-percentage parsing — v1.134.0
join	JOIN	per-company	A company's JOIN board read from the Next.js __NEXT_DATA__ in join.com/companies/<slug> (host-pinned, page-capped, redirect:'error'); detected from a join.com careers URL — v1.135.0
getro	Getro	board-wide	VC talent-network portfolio boards via api.getro.com/api/v2/collections/{id}/search/jobs POST (zero-auth); explicit numeric getro_collection:, newest-first pagination with age-bounded early stop, portfolio-employer attribution — v1.135.0
consider	Consider	board-wide	getconsider.com VC portfolio boards via same-origin /api-boards/search-jobs POST; config-driven host pinned by a structural SSRF guard (public HTTPS only — rejects IP-literals/loopback/*.internal) — v1.135.0
joinup	JOINUP	board-wide	Swiss startup board joinup.ch, SSR'd newest page of /browse/jobs (__NEXT_DATA__), fail-closed on a scraper break — v1.135.0
remotli	Remotli	board-wide	remotli.ch curated Swiss remote board (CHF salaries) via ?remote=all JSON API; emits the employer's own ATS applyUrl so cross-listings dedup — v1.135.0
eightfold	Eightfold	per-tenant	Eightfold AI talent-acquisition boards via https://<tenant>.eightfold.ai/api/apply/v2/jobs (zero-auth); host-pinned *.eightfold.ai (branded careers.<company>.com CNAME rejected), paginated with a safety cap, dead-board-throw, url-dedup — v1.136.0
rheinmetall	Rheinmetall	single-company	v1.118.0
dassault	Dassault Systèmes	single-company	Exalead feed
amazon	Amazon	single-company
ibm	IBM	single-company
arbeitsagentur	Arbeitsagentur	board-wide	German federal job board
arbeitnow	Arbeitnow	board-wide
remoteok	RemoteOK	board-wide
remotive	Remotive	board-wide
weworkremotely	We Work Remotely	board-wide
workingnomads	Working Nomads	board-wide
nodesk	NoDesk	board-wide
himalayas	Himalayas	board-wide
jobicy	Jobicy	board-wide
jobspresso	Jobspresso	board-wide
4dayweek	4 Day Week	board-wide
justjoin	JustJoin.it	board-wide	Polish IT board
nofluffjobs	NoFluffJobs	board-wide	Polish IT board
solidjobs	SolidJobs	board-wide	Polish IT board
thehub	The Hub	board-wide	Nordic startups
landingjobs	Landing.jobs	board-wide
getonbrd	Get on Board	board-wide	LatAm
glints	Glints	board-wide	SE Asia
jobstreet	Jobstreet / SEEK	board-wide	SE Asia / ANZ
themuse	The Muse	board-wide
echojobs	EchoJobs	board-wide	v1.118.0; hybrid roles kept distinct from remote ("City · Hybrid", never collapsed to Remote) — v1.130.0
larajobs	LaraJobs	board-wide	RSS — v1.118.0
higheredjobs	HigherEdJobs	board-wide	RSS category feeds — v1.117.0
hackernews	Hacker News (Who is hiring)	board-wide	Monthly thread
rss	RSS	generic	Point it at any job RSS/Atom feed — the escape hatch when a board has no adapter

RU sources (5)

Dispatched by ru-scanner.mjs::RU_DISPATCH and configured under russian_portals: in portals.yml (sources: ["hh", "habr", "trudvsem", "getmatch", "geekjob"] is the default).

id	Label	configKey	Notes
hh.ru	hh.ru	hh	HTML scrape of hh.ru/search/vacancy (the JSON API 403s programmatic clients)
habr-career	Habr Career	habr	career.habr.com/api/...
trudvsem	Trudvsem	trudvsem	Russian federal job board
getmatch	GetMatch	getmatch
geekjob	GeekJob	geekjob

How to add a new provider

The full walkthrough lives in the in-app help guide §17 ("How to add a new job-portal source"). Summary:

1. Create server/lib/sources/<slug>.mjs exporting meta = { value, label, region: 'en'|'ru', configKey? } and a search<Name>(query, opts) returning normalized job objects: { id, title, company, url, salary, location, isRemote, workplaceType, relocates, date, snippet, source }. It auto-registers at boot — no registry edit.
2. EN boards: add server/lib/portals/adapters/<slug>.mjs implementing the adapter contract — matches(url) and buildEndpoint(...) (must return a string or null, never an object) — and register it in server/lib/portals/registry.mjs::ALL_ADAPTERS. Update the count assertions in tests/adapter-registry.test.mjs and tests/scan-sources-endpoint.test.mjs.
3. RU sources: add a RU_DISPATCH row in ru-scanner.mjs.
4. SSRF host-pinning (mandatory): every adapter pins its hostname with an anchored regex (e.g. /(^|\.)justjoin\.it$/i — so justjoin.it.evil.com can't spoof) and rejects http:. Adapters with no stable vendor host (e.g. Radancy) validate the endpoint path shape and are selected only via an explicit provider: key in portals.yml. Parity tests must cover: evil host rejected, http: rejected, suffix-spoof rejected, parse happy path, pagination stop condition.
5. Config-driven providers: POST/GraphQL/paginated boards thread opts.company through buildEndpoint; keep buildEndpoint returning a string.
6. Add a parity test suite (tests/sources-parity-*.test.mjs, stubbed transport — no network), bump the registry counts, and fan out docs ×17 per the Release Process.

Scan behavior

• Entry point: GET /api/stream/scan?source=ats|regional|both (SSE, one done per phase, final:true on the last).
• dryRun=1 skips writes to data/scan-history.tsv / data/last-scan.json; company=<slug> narrows the ATS walk.
• Results are deduped against data/scan-history.tsv, sanitized (scan-sanitize.mjs), quarantine-filtered (scan-quarantine.mjs), location-filtered, and matched against your target roles / two-pager (◎ badge).
• Client disconnect aborts in-flight upstream fetches (AbortSignal).

Since v1.119.1 the #/scan Source dropdown's offline fallback (FALLBACK_SOURCES in public/js/views/scan.js) is gated by tests/scan-fallback-sources.test.mjs — exact value+label parity with the source registry, so a new adapter can't ship without the client list.