Skip to content

v0.1.1 — a delisted offer is a 404, not a parse error

Choose a tag to compare

@jehrr jehrr released this 18 Sep 12:13
· 12 commits to main since this release

Fixes a wasted fetch and a spurious debug dump in --mode offer. If
you saw Page N came back as parse_error on an offer run, this is that.

The canary found it on its first run — which is the point of a canary that
is not gated on a secret.

justjoin.it's sitemap is generated ahead of the fetch, so an offer can be
taken down in between. The first slug in active-jobs/part0.xml had
been. All three engines were discarding what the navigation returned, so
the site's 404 reached the classifier as status=None, fell past the
not_found branch and landed on parse_error — a retry, a debug dump and
a wasted fetch for an address that will never exist again.

Two fixes, because one of them cannot cover every engine:

  • Playwright and pyppeteer now bind response.status and pass it
    through.
  • Selenium cannot. driver.get() returns None and WebDriver exposes no
    HTTP status at all. So product_parser.problem_status reads the status
    out of the site's own RFC 7231 problem document, which justjoin.it states
    in the body of every 404. That reader is what makes the three engines
    agree rather than two of them being better informed than the third.

Verified live against the real delisted offer: all three now report
not_found, exit 4, no retry and no dump. Pinned by a check that asserts
both the body reader and that each engine actually threads a status, so
the fix cannot go missing from a twin.

586 offline checks, up from 573.