Skip to content

v0.2.1 — output is replaced atomically

Latest

Choose a tag to compare

@jehrr jehrr released this 30 Sep 07:57

The output files are now replaced atomically. Before this release an interrupted run could leave a shorter file where a complete one had been — measured, a 2,084-byte good out.json came back 0 bytes and invalid JSON after a crash mid-write. If you have ever seen a truncated output or sidecar, that was this.

CSV cells beginning =, +, -, @, tab, CR or LF now carry a leading apostrophe, so a spreadsheet reads them as text rather than as a formula. The JSON is unchanged and still carries the site's own bytes; csv_cells_escaped in <out>.meta.json says how many cells the two files differ in. On this site that number is 0 today, measured across 2,662 stored rows, so no existing CSV changes.

Fixed

  • Atomic writes. write_json, write_csv and write_run_meta used open(path, "w"), which truncates before a byte is written. Each now writes to a temporary file in the same directory (os.replace is only atomic within one filesystem), fsyncs it and renames over the target. The sidecar matters most: it is the file a consumer branches on, so a truncated one beside good rows reads as a broken run over data that is fine.
  • CSV formula neutralisation, strings only — escaping a number would turn -5 into text and break every sum written over the column.
  • Scraper API: waitFor is sent as a JSON object. The string form is refused with HTTP 422 and still billed, so every --wait-* run was a paid exit 5.
  • Scraper API: the target's HTTP status is read from http_code, not from the API's own verdict string, so the page classifier now sees a target 403/503.
  • The end-of-listing check compared against the wrong page number, so --url '...?p=2' --pages 1 threw away 45 good rows as "the end of the listing".
  • The credential scan was blind to JSON-escaped fixtures — the key-shaped-field rule used bare quotes and matched zero times in the largest file in the repository.
  • Donor-repo leftovers, none of which changes behaviour.

Changed

  • CI tests Python 3.13. The badge has claimed 3.9 | 3.13 while the offline matrix ran 3.9 and 3.12 — the ceiling a reader sees first was the one version nobody ran.

Full notes: CHANGELOG.md