Skip to content

v0.1.4

Latest

Choose a tag to compare

@jehrr jehrr released this 15 Oct 22:50
8839455

Behaviour changes for existing users. --route ssr now refuses, with
exit 2, any filter other than category, city, remote and keyword (it used
to accept them and ignore them). diff_runs.py now refuses a run with no
sidecar unless --force is passed. --retries 0 is refused.

CSV output now prefixes a cell that starts with
=, +, -, @, a tab, CR or LF with an apostrophe (JSON is
unchanged), and the sidecar gains csv_cells_escaped. .env.example
no longer sets JUSTJOIN_URL; a JUSTJOIN_URL from the environment that
names a different mode than --mode is now ignored with a log line
instead of exiting 2.

Fixed

  • An offer queue no longer ends at the first delisted offer (all three
    engines). A 404 on one offer was read as the end of the run.

  • --mode offer works in Selenium and pyppeteer, and pyppeteer no
    longer crashes without a proxy pool.

  • --route ssr no longer accepts filters it ignores, and no longer
    records a sort, order or total the page does not carry.

  • --mode facets is partial when a group is missing, and the parity
    engines now request the categories.

  • diff_runs.py refuses runs without a sidecar, and tells an absent
    filters key from an empty one. Engines always record filters.

  • The Scraper API client writes a sidecar and picks the row class by
    mode (its facets CSV had the wrong header).

  • Output writes are atomic. A failed write no longer replaces the
    previous good JSON, CSV or sidecar with a truncated one.

  • CSV cells are formula-escaped. Every string in a row is written by an
    employer, and a spreadsheet runs a cell starting with =, +, - or
    @ as a formula. Escaped in the CSV only, after list cells are joined;
    numbers stay numbers and JSON keeps the site's bytes. The count goes in
    the sidecar as csv_cells_escaped, under any caller-provided field of
    the same name.

  • JUSTJOIN_URL no longer breaks --mode offer and --mode facets.
    The listings URL that .env.example shipped was read as if --url had
    been typed, so a copied example made both modes exit 2. The engines now
    record what was typed before reading the environment; only a typed URL is
    checked against --mode. The example now shows the variable commented
    out.

  • diff_runs.py sees changes in a facets run. It compared the job
    columns on facet rows, which have none of them, so an offer_count
    going from 100 to 250 reported "0 changed". Facets runs are now compared
    on title, offer_count, is_cumulative and overlaps_other_keys.

  • The credential scan works from any parent directory. Its skip list
    was matched against the absolute path, so a clone under a directory
    named build, env or venv scanned 0 files and reported clean. It now
    matches the path inside the repo, and the suite runs the scan from such a
    copy with a planted key.

  • --dump-html help pointed at a TROUBLESHOOTING.md this repo does not
    have; it now points at the README section on traps that look like bugs.

  • Engine comments said the paid captcha path runs only for a challenge
    state that does not exist here; they now describe what
    page_flow.STATE_POLICY does.

  • .gitignore covers --dump-html page files of any extension
    (*.page[0-9]*), a live/ scratch directory and HTML dumps at the root.

  • CI tests Python 3.13, which the README badge already claimed.

  • Comments, docstrings and changelog entries no longer cite internal notes
    that are not public; each rule is stated in place. A suite check keeps it
    that way.

  • Scraper API: waitFor is now sent as a JSON object. Measured
    2026-09-23 against scraper.2captcha.com/tasks/sync: the JSON-encoded
    string this client sent (the form older docs described) is refused with
    HTTP 422 "params.waitFor must be an object" -- and the task is still
    billed ($0.0005) -- so every run with --wait-text or its sibling wait flags
    failed with exit 5. The object form answers HTTP 200.

  • Scraper API: the target page's status is read from http_code. The
    response's status field is the API's own verdict string ("success"),
    not the target site's HTTP code, so a target 403/503 was never seen by
    this client. http_code (an int) is read now, with status kept as a
    fallback only when it is an int.

Leftovers from the repos this one was bootstrapped from, each describing
another site as if it were this one:

  • CONTRIBUTING.md described Mercor's sources, markers, captcha, live-run
    notes and scope (and Wellfound's pagination and pay strings). Rewritten
    for justjoin.it from this repo's own README measurements.

  • SECURITY.md asked for "mercor-scraper security" in the subject line and
    named Mercor in the out-of-scope list; it also said the project has no
    releases, and pointed at a README section that does not exist.

  • The issue templates were Etsy's: DataDome advice, etsy.com example
    URLs, a JSON-LD evidence recipe for a site whose parser reads JSON, and
    links to a TROUBLESHOOTING.md this repo does not have.

  • requirements.txt was headed # mercor-scraper.

  • .gitignore and .dockerignore ignored wellfound_jobs.* instead of
    this repo's own default output prefix justjoin_jobs.

  • smoke_test.py: the sidecar check used Mercor run metadata
    (mercor.com, /explore, 390 records); it now uses a real
    --route ssr sidecar from this repo. The suite's --help description
    said "mercor-scraper offline suite", and three comments described
    Mercor's routes as this site's.

  • captcha_solver.py's docstring pointed at a "No DataDome solver" section
    that does not exist in this repo (it came with the copied core). Removed.