Integrating goodjobs (self-hosted job aggregator) with career-ops

[vnk8071]

What goodjobs is

goodjobs is a self-hosted, open-source job-search aggregator (a sibling project of mine) that scrapes 18 boards behind one FastAPI backend — LinkedIn, ITViec, TopCV, VietnamWorks, CareerViet, TopDev, JobsGo, CareerLink, Glints, ViecOi, Indeed VN, plus a global US/UK/SG set. One search box, real-time SSE streaming, a Redis cache warmed every 2 hours, AI intent detection, skill extraction, and a vector-embedding "related jobs" supplement.

It's a genuinely different shape of source from anything currently in providers/: every existing provider either scrapes one board directly (itviec.mjs, careerviet.mjs) or hits one ATS's public API (greenhouse.mjs, ashby.mjs). goodjobs is neither — it's a pre-aggregated multi-board feed that a user (or the community) self-hosts.

What's already shipped

#3520 adds providers/goodjobs.mjs as a thin client: it POSTs to goodjobs' own public POST /scrape endpoint and normalizes the JSON array into career-ops' Job shape. Explicit opt-in only (provider: goodjobs in portals.yml) — no auto-detection, since there's no fixed host or URL pattern to key off for a self-hosted service.

Two things worth calling out from actually running it end-to-end against the live instance rather than just reading the API docs:

• The API host isn't the frontend host. goodjobs.io.vn is a static GitHub Pages frontend; the actual backend lives on api.goodjobs.io.vn. Caught this before merging by reading the frontend's built JS bundle for its baked-in VITE_API_URL.
• Found and fixed a real bug in goodjobs itself along the way. Live testing hit a reproducible 500 on every /scrape call — root cause was a Pydantic model rejecting a None sentinel that 8 of the scrapers legitimately write (summary_description: null = "not yet summarized"). Fixed upstream (vnk8071/goodjobs@0748103) the same day, verified live, and it's now a resolved historical note in the provider's own header comment rather than an open caveat.

Ran it for real afterward: node scan.mjs --company goodjobs against "AI Engineer" / Ho Chi Minh City pulled 57 postings, 45 of them new after title-filter and dedup.

Where I'd like input

1. Depth of integration. Right now this only touches the plain /scrape endpoint. goodjobs also exposes /recent-jobs (a cached, pre-filtered VN-facing view) and /search-semantic (vector search over the cache). Is a second, cheaper provider variant worth it for users who want a fast/cached read instead of a live multi-board sweep, or is that scope creep for a scan provider whose whole job is "return jobs for this keyword/location"?
2. Self-hosting story. The provider defaults to my own public instance (https://api.goodjobs.io.vn), but the more interesting long-term case is someone running their own goodjobs instance and pointing portals.yml's api: field at it. Is there appetite for a short "run your own aggregator" doc pairing career-ops with goodjobs (or a similar self-hosted aggregator), given career-ops' SSRF guard means only a publicly routable instance works — a local docker compose up won't do?
3. Class of provider, generally. goodjobs is the first "aggregator I self-host" source in providers/. If there's interest in more of this shape (as opposed to single-board scrapers or single-ATS clients), worth thinking about whether the contract needs anything generalized, or whether goodjobs.mjs's thin-client pattern is already the right template for the next one.

Happy to hear if this is useful to anyone outside VN-market job searches too — the global US/UK/SG side of goodjobs is there and untouched by this PR.

[santifer]

Answered on the PR, @vnk8071, and the same answer holds here: under rule 1 of the Source Indexing Policy an aggregator that republishes other boards' postings isn't a source career-ops indexes, so #3520 is closed on that rule and not on the code. Boards that publish their own employer-attributed postings behind a public API go straight in, as your ITviec and CareerViet providers did.