v0.1.4

Behaviour changes for existing users. --route ssr now refuses, with
exit 2, any filter other than category, city, remote and keyword (it used
to accept them and ignore them). diff_runs.py now refuses a run with no
sidecar unless --force is passed. --retries 0 is refused.

CSV output now prefixes a cell that starts with
=, +, -, @, a tab, CR or LF with an apostrophe (JSON is
unchanged), and the sidecar gains csv_cells_escaped. .env.example
no longer sets JUSTJOIN_URL; a JUSTJOIN_URL from the environment that
names a different mode than --mode is now ignored with a log line
instead of exiting 2.

Fixed

• An offer queue no longer ends at the first delisted offer (all three
  engines). A 404 on one offer was read as the end of the run.

• --mode offer works in Selenium and pyppeteer, and pyppeteer no
  longer crashes without a proxy pool.

• --route ssr no longer accepts filters it ignores, and no longer
  records a sort, order or total the page does not carry.

• --mode facets is partial when a group is missing, and the parity
  engines now request the categories.

• diff_runs.py refuses runs without a sidecar, and tells an absent
  filters key from an empty one. Engines always record filters.

• The Scraper API client writes a sidecar and picks the row class by
  mode (its facets CSV had the wrong header).

• Output writes are atomic. A failed write no longer replaces the
  previous good JSON, CSV or sidecar with a truncated one.

• CSV cells are formula-escaped. Every string in a row is written by an
  employer, and a spreadsheet runs a cell starting with =, +, - or
  @ as a formula. Escaped in the CSV only, after list cells are joined;
  numbers stay numbers and JSON keeps the site's bytes. The count goes in
  the sidecar as csv_cells_escaped, under any caller-provided field of
  the same name.

• JUSTJOIN_URL no longer breaks --mode offer and --mode facets.
  The listings URL that .env.example shipped was read as if --url had
  been typed, so a copied example made both modes exit 2. The engines now
  record what was typed before reading the environment; only a typed URL is
  checked against --mode. The example now shows the variable commented
  out.

• diff_runs.py sees changes in a facets run. It compared the job
  columns on facet rows, which have none of them, so an offer_count
  going from 100 to 250 reported "0 changed". Facets runs are now compared
  on title, offer_count, is_cumulative and overlaps_other_keys.

• The credential scan works from any parent directory. Its skip list
  was matched against the absolute path, so a clone under a directory
  named build, env or venv scanned 0 files and reported clean. It now
  matches the path inside the repo, and the suite runs the scan from such a
  copy with a planted key.

• --dump-html help pointed at a TROUBLESHOOTING.md this repo does not
  have; it now points at the README section on traps that look like bugs.

• Engine comments said the paid captcha path runs only for a challenge
  state that does not exist here; they now describe what
  page_flow.STATE_POLICY does.

• .gitignore covers --dump-html page files of any extension
  (*.page[0-9]*), a live/ scratch directory and HTML dumps at the root.

• CI tests Python 3.13, which the README badge already claimed.

• Comments, docstrings and changelog entries no longer cite internal notes
  that are not public; each rule is stated in place. A suite check keeps it
  that way.

• Scraper API: waitFor is now sent as a JSON object. Measured
  2026-09-23 against scraper.2captcha.com/tasks/sync: the JSON-encoded
  string this client sent (the form older docs described) is refused with
  HTTP 422 "params.waitFor must be an object" -- and the task is still
  billed ($0.0005) -- so every run with --wait-text or its sibling wait flags
  failed with exit 5. The object form answers HTTP 200.

• Scraper API: the target page's status is read from http_code. The
  response's status field is the API's own verdict string ("success"),
  not the target site's HTTP code, so a target 403/503 was never seen by
  this client. http_code (an int) is read now, with status kept as a
  fallback only when it is an int.

Leftovers from the repos this one was bootstrapped from, each describing
another site as if it were this one:

• CONTRIBUTING.md described Mercor's sources, markers, captcha, live-run
  notes and scope (and Wellfound's pagination and pay strings). Rewritten
  for justjoin.it from this repo's own README measurements.

• SECURITY.md asked for "mercor-scraper security" in the subject line and
  named Mercor in the out-of-scope list; it also said the project has no
  releases, and pointed at a README section that does not exist.

• The issue templates were Etsy's: DataDome advice, etsy.com example
  URLs, a JSON-LD evidence recipe for a site whose parser reads JSON, and
  links to a TROUBLESHOOTING.md this repo does not have.

• requirements.txt was headed # mercor-scraper.

• .gitignore and .dockerignore ignored wellfound_jobs.* instead of
  this repo's own default output prefix justjoin_jobs.

• smoke_test.py: the sidecar check used Mercor run metadata
  (mercor.com, /explore, 390 records); it now uses a real
  --route ssr sidecar from this repo. The suite's --help description
  said "mercor-scraper offline suite", and three comments described
  Mercor's routes as this site's.

• captcha_solver.py's docstring pointed at a "No DataDome solver" section
  that does not exist in this repo (it came with the copied core). Removed.

v0.1.3 — audit fixes

An audit against this family's own checklist, run after publishing
rather than before it. Everything here was found by a check or a live run,
not by reading.

Fixed

scraper_api_client.py reported an upstream failure as an empty board.
It did not consult page_flow.should_parse() the way the three browser
engines do, so a body the site never served came back as Parsed 0 job(s)
and exit 4 — "the query matched nothing". It now names the state, saves
the body and exits 5. Blocked, empty and failed are three different
answers.

A latent NameError in the same client — EXIT_NO_PRODUCTS used on
the new not_found branch and never imported. Caught by the suite's
undefined-name walk, and the control was verified: planting the fault makes
the check name that exact symbol.

--url was validated against HOST while HOSTS sat unread. The
tuple is the single source of truth now rather than decoration.

Corrected

The README had the Scraper API backwards. It said the service suits the
rendered listing page better than the endpoint — reasoned, not measured.
Measured 2026-09-18:

--url	result
/api/candidate-api/offers?… (47 KB)	HTTP 200 in 12s, 20 rows, $0.0005
/job-offers/all-locations (1.8 MB)	HTTP 408 at the default 60s
the same page at --timeout 120	API 200, upstream HTTP 500, 0 bytes

Point it at the endpoint.

An inherited figure replaced with a measured one. A comment repeated a
"17 of 18 repos" count for the fingerprint flags. Re-counted across the 26
siblings: 25 of 26 — with the command beside it, so the next reader
re-measures instead of inheriting.

Verified live, with a real 2Captcha key

• --fingerprint applies what it fetches. Four defects have shipped on
  this path across this family; none is present here. A PL fingerprint
  reached the browser intact — user agent Windows NT 10.0 … Chrome/151,
  locale pl-PL (not en-PL), timezone Europe/Warsaw — and
  --fp-tags Windows is accepted by the API.
• Credentials never reach a log, including an exception message. A dead
  proxy carrying secretuser:secretpass123 logged
  http://***:***@127.0.0.1:9 — host and port kept, both credentials
  masked, zero occurrences of either anywhere in the run — and was named a
  proxy failure rather than a timeout. Identical on all three engines.

597 offline checks, up from 591.

v0.1.2 — walk the sitemap newest-first

Fixes --mode offer coming back empty on a site that is serving
perfectly. If a small offer run gave you zero rows, this is that.

The canary found it — the second thing it has found in two runs, which is
what a canary that is not gated on a secret is for.

justjoin.it serves sitemaps/active-jobs/part0.xml oldest first (its
first lastmod was 2026-09-14 and its last 2026-09-18) and generates it
ahead of any fetch, so its head is where offers that have since been taken
down collect. Measured by asking the endpoint for a sample of each end:

sample	alive	gone
first 12 entries	10	2
last 12 entries	12	0
12 at random	12	0

So --mode offer --pages 3 was being pointed at the three likeliest-dead
addresses in the file.

The enumeration is now sorted by lastmod descending — which is also what
--pages N should have meant all along: the N most recently updated
offers, not the N most stale. Entries with no lastmod are kept and sort
last rather than being dropped, and the document order is still available
so a check can assert the sort loses nothing.

The offer canary asks for five offers rather than three, so one delisted
offer is an ordinary event rather than a red check — while zero rows out of
five still means something real broke.

591 offline checks, up from 586.

v0.1.1 — a delisted offer is a 404, not a parse error

Fixes a wasted fetch and a spurious debug dump in --mode offer. If
you saw Page N came back as parse_error on an offer run, this is that.

The canary found it on its first run — which is the point of a canary that
is not gated on a secret.

justjoin.it's sitemap is generated ahead of the fetch, so an offer can be
taken down in between. The first slug in active-jobs/part0.xml had
been. All three engines were discarding what the navigation returned, so
the site's 404 reached the classifier as status=None, fell past the
not_found branch and landed on parse_error — a retry, a debug dump and
a wasted fetch for an address that will never exist again.

Two fixes, because one of them cannot cover every engine:

• Playwright and pyppeteer now bind response.status and pass it
  through.
• Selenium cannot. driver.get() returns None and WebDriver exposes no
  HTTP status at all. So product_parser.problem_status reads the status
  out of the site's own RFC 7231 problem document, which justjoin.it states
  in the body of every 404. That reader is what makes the three engines
  agree rather than two of them being better informed than the third.

Verified live against the real delisted offer: all three now report
not_found, exit 4, no retry and no dump. Pinned by a check that asserts
both the body reader and that each engine actually threads a status, so
the fix cannot go missing from a twin.

586 offline checks, up from 573.

v0.1.0 — justjoin.it scraper

First release. Reads justjoin.it, the Polish IT job
board, through Playwright, Selenium, Puppeteer or the 2Captcha Scraping
Browser API over CDP.

pip install -r requirements.txt -r requirements-playwright.txt
python3 -m playwright install chromium
python3 playwright_scraper.py --category python --pages 5 --format csv

That command needs no account, no key and no proxy.

Three modes

• --mode listings (default) — the board through justjoin.it's own
  offers endpoint, with offset pagination and nine filters measured to
  change what the site sends.
• --mode offer — one offer in full: the description body, per-skill
  levels, the company website and size, the country. Enumerated from the
  site's own sitemap (10,646 offer URLs), from --url, or from a previous
  run's output via --slugs-file.
• --mode facets — the board's own taxonomy with its own counts, in a
  second row schema. These counts are not capped, which is what makes
  them the tool for planning runs that reach past the per-query ceiling.

What running it found

All measured 2026-09-18 from a netcup VPS in Nuremberg (AS197540).

Nothing on the read path is gated. HTTP 200 to curl with no headers at
all, to python-requests, to wget and to a headless Chromium. Sixteen
candidate bot-challenge markers counted at zero across seven captures. One
exception, and it is a denylist entry rather than a gate: a
Python-urllib/* User-Agent gets HTTP 403.

The captcha is reCAPTCHA v2 — on the application form, and on no route
this scraper reads. The site publishes its own key
(googleRecaptchaV2Key), and the only string referencing it lives in the
job-application form's i18n namespace. In a live browser window.grecaptcha
was undefined on a rendered listing, an offer page and the endpoint alike:
the script is never even loaded on the read path.

The endpoint the front end calls is not the host the page config names.
api.justjoin.it — the site's own baseApiUrl — answers HTTP 503 from
nginx to everything. The route that works is the same-origin proxy at
justjoin.it/api/candidate-api/.

The salary is not employmentTypes[0]. The employer's own figure
carries currencySource: "original" and sits elsewhere on 321 of 1,000
listing rows and 29 of 40 detail records. On the detail route 16 of those 40
would get a wrong number, not just a wrong currency — 39.15 CHF printed
for a job offering 180 PLN.

from is the rate normalised to a month while unit names the
employer's unit. Reading the two adjacent keys together — the obvious
thing to do — says "18,480 PLN per hour" for a job paying 110 an hour, on
307 of 582 quoted entries. The scalar columns carry the quoted rate; the
monthly figure lives in columns that say salary_monthly_*.

?page=2 on a rendered listing returns page 1 again under HTTP 200, so
page_url() refuses to invent one and --route ssr plans a single page.

Every query is capped at 10,000 while the board held 19,381, and
from=10000 is an HTTP 500 rather than an empty page. The sidecar records
the whole arithmetic, because "complete" and "exhaustive" are different
words.

The default ordering is not a paid placement — checked rather than
assumed. isSuperOffer was spread through the result (9–34 per hundred at
five offsets) rather than banked at the top, so the scraper keeps the site's
own default sort. The sort is recorded, because on a capped result set it
decides which offers are in the file: two orderings of one query overlapped
on 5 of their first 100 ids.

Two defects fixed in the family core

Both inherited, both found by running the code rather than reading it.

• The two parity engines called mask(pool.current) unguarded in the
  block-retry path while the Playwright engine guarded it, so any retryable
  state on a run without --proxy died with AttributeError instead of
  retrying.
• scraper_api_client.py could not import at all — it called a parser entry
  point that had been renamed, so --help itself was an ImportError.
  Caught by the suite's signature-binding check.

Verified

All three engines produce byte-identical rows and identical sidecars for
the same command. 573 offline checks, and a daily canary that runs a real
three-page scrape from a bare GitHub runner with no secrets — expected to be
green, because that is what keeps the README's central claim honest.