job-crawler

A modular, polite crawler for job boards, written in Go with no external dependencies. Providers are pluggable; a settings file decides which ones run and how. The first provider crawls job boards hosted on Ashby (jobs.ashbyhq.com), e.g. https://jobs.ashbyhq.com/duck-duck-go.

Quick start

go build ./cmd/jobcrawler
./jobcrawler -settings settings.json          # updates crawler-db.json, writes jobs.jsonl
./jobcrawler -settings settings.json -output - # stream the JSONL export to stdout instead

Each crawled posting is exported as one JSON object per line (JSON Lines):

{"provider":"ashby","company":"duck-duck-go","company_name":"DuckDuckGo","id":"…","title":"Senior Software Engineer","location":"Remote","employment_type":"FullTime","compensation":"$150K – $180K","description_html":"<p>…</p>","url":"https://jobs.ashbyhq.com/duck-duck-go/…"}

The JSON database

All results and crawl state live in a JSON file database (database_path, default crawler-db.json), written atomically after every provider:

{
  "version": 1,
  "providers": {
    "ashby": {
      "known_companies": ["duck-duck-go", "…"],
      "last_discovery_at": "2026-06-10T02:00:00Z",
      "companies": {
        "duck-duck-go": { "crawled_at": "2026-06-10T02:00:00Z", "jobs": [ … ] }
      }
    }
  }
}

• known_companies — every company board found so far. The next session starts from this list (plus the configured seeds), so discovery work is never lost.
• last_discovery_at — when the last full company mapping ran. The expensive mapping (following links to other boards, probing the sitemap) only runs again once discovery_interval (default 48h) has passed; sessions in between just re-crawl the already-known companies.
• companies — the jobs of each company, replaced whenever that company's board is crawled to completion (companies that fail mid-crawl keep their previous jobs). Boards that return 404 are dropped from the known list.

The JSON Lines export (output_path) is optional on top of the database; set it to "" to disable it, or "-" to stream to stdout.

Politeness

All requests go through one shared HTTP client (internal/httpclient) that provides:

• Timeouts — every request is bounded by request_timeout.
• Exponential retries — transient failures (network errors, 429, 5xx) are retried up to max_retries times, doubling from retry_initial_backoff up to retry_max_backoff, with jitter, and honoring Retry-After headers.
• Request spacing — at least request_spacing between consecutive requests, including retries, to stay clear of rate limits.
• Custom User-Agent — user_agent identifies the service and a contact address on every request.
• Same-host redirects only — a crawl is never redirected onto another domain, and non-429/5xx errors are never retried.

Settings

settings.json (override the path with -settings):

{
  "user_agent": "JobCrawlerBot/1.0 (+https://github.com/curiousfurbytes/job-crawler; contact: you@example.com)",
  "request_timeout": "15s",
  "request_spacing": "1500ms",
  "max_retries": 3,
  "retry_initial_backoff": "500ms",
  "retry_max_backoff": "30s",
  "database_path": "crawler-db.json",
  "output_path": "jobs.jsonl",
  "providers": {
    "ashby": {
      "enabled": true,
      "options": { "companies": ["duck-duck-go"] }
    }
  }
}

Every key under providers names a registered provider; enabled toggles it and options is parsed by the provider itself, so each provider can define its own schema.

Ashby provider options

Option	Default	Meaning
base_url	https://jobs.ashbyhq.com	The only host the provider crawls.
companies	—	Seed list of board slugs, e.g. "duck-duck-go".
discover_from_links	true	Follow links to other boards on the same host.
discover_from_sitemap	false	Probe <base_url>/sitemap.xml for boards.
discovery_interval	"48h"	How often the full company mapping runs; in between, only known companies are crawled.
fetch_descriptions	true	Visit each posting's page for the full description.
max_companies	0 (no cap)	Cap on boards visited per crawl.

How company discovery works

Every company on Ashby lives under its own path on a single subdomain (jobs.ashbyhq.com/<company>), and the root path does not enumerate companies — so a crawler can't simply start at /. Instead of brute forcing paths, the Ashby provider:

1. starts from the configured seed list plus the known companies persisted in the database from earlier sessions,
2. extracts links to other boards on the same host from every page it visits (deduplicated, UUID posting IDs and reserved application paths filtered out, optionally capped by max_companies), and
3. optionally probes the host's sitemap.

Steps 2 and 3 are the full company mapping and run at most once every discovery_interval (48h by default); everything they find is persisted, so the sessions in between start with those companies already. To grow the seed list, add slugs to companies — they are easy to harvest from sources like a web search for site:jobs.ashbyhq.com — and the crawler takes it from there. Nothing is ever requested outside the configured host.

Rather than scraping markup, the provider parses the window.__appData JSON blob that Ashby's server-rendered pages embed, which carries the board's postings (title, team, location, employment type, compensation summary) and each posting page's full description. The parser is deliberately tolerant: every field is optional, so structure changes degrade the output instead of breaking the crawl. If Ashby changes the embedding entirely, internal/provider/ashby/appdata.go is the one file to update.

Architecture

cmd/jobcrawler          CLI entry point; wires settings → client → providers → output
internal/config         settings file loading, defaults, validation
internal/httpclient     polite HTTP client (timeouts, retries, spacing, UA)
internal/provider       Provider interface + registry (the plug-in point)
internal/provider/ashby the jobs.ashbyhq.com provider
internal/crawler        runs enabled providers, persists results and state
internal/store          the JSON file database (companies, jobs, crawl state)
internal/output         optional JSON Lines export
internal/model          the normalized Job type all providers emit

Adding a provider

1. Create internal/provider/<name> with a type implementing provider.Provider (Name() + Crawl(ctx, emit)).
2. Register it in an init function: provider.Register("<name>", factory).
3. Blank-import the package in cmd/jobcrawler/main.go.
4. Enable it in settings.json under providers.<name>.

Providers receive a provider.Fetcher (the shared polite HTTP client), their raw JSON options, and the CrawlState persisted from their last session; they emit model.Job values through the sink and return the state the next session should start from. The orchestrator handles persistence — providers never touch the database directly. One failing company or provider never stops the rest of the crawl; errors are joined and reported at the end.

Tests

go test ./...

Everything is covered by unit tests — config parsing, retry/spacing/UA behavior of the HTTP client (with a fake clock, so no slow tests), the __appData parser, company discovery, the Ashby provider end-to-end against a local httptest server, the orchestrator, and the JSONL writer.