Repository navigation
v0.1.1 — a delisted offer is a 404, not a parse error
Fixes a wasted fetch and a spurious debug dump in
--mode offer. If
you sawPage N came back as parse_erroron an offer run, this is that.
The canary found it on its first run — which is the point of a canary that
is not gated on a secret.
justjoin.it's sitemap is generated ahead of the fetch, so an offer can be
taken down in between. The first slug in active-jobs/part0.xml had
been. All three engines were discarding what the navigation returned, so
the site's 404 reached the classifier as status=None, fell past the
not_found branch and landed on parse_error — a retry, a debug dump and
a wasted fetch for an address that will never exist again.
Two fixes, because one of them cannot cover every engine:
- Playwright and pyppeteer now bind
response.statusand pass it
through. - Selenium cannot.
driver.get()returns None and WebDriver exposes no
HTTP status at all. Soproduct_parser.problem_statusreads the status
out of the site's own RFC 7231 problem document, which justjoin.it states
in the body of every 404. That reader is what makes the three engines
agree rather than two of them being better informed than the third.
Verified live against the real delisted offer: all three now report
not_found, exit 4, no retry and no dump. Pinned by a check that asserts
both the body reader and that each engine actually threads a status, so
the fix cannot go missing from a twin.
586 offline checks, up from 573.