Skip to content

v0.2.0

Choose a tag to compare

@jehrr jehrr released this 07 Sep 13:41
· 17 commits to main since this release
fdd027a

First release verified against the live site. A real run of playwright_scraper.py on the README's own known-good category URL, from a residential IP with no proxy, no API key and no paid product, returned 96 products and cleared Akamai silently — confirming the README's central claim, which every earlier release had only fixture-level evidence for. The documented geo-redirect was confirmed at the same time (.com/shopping.com/de/shopping, EUR, localised titles), as were the compounded-discount figures the tests were built against.

Changed

  • Output is now sixteen columns, not fifteen: Product gained price_source. Anything parsing the CSV header or asserting a column count needs updating.
  • A JSON-LD offer with no priceCurrency now yields currency: null instead of a guessed "USD", and the DOM discount-overlay no longer overwrites a currency the structured data stated explicitly.

Added

  • price_source — says how much to trust price: jsonld+dom (the rendered tile was found and reconciled — trustworthy), jsonld (structured data only; the tile was missing or disagreed, so on a discounted item this may be the pre-promo price), or dom (CSS fallback, no cross-check). The same column previously held all three with no way to tell them apart.
  • Run metadata sidecar — every run that writes output also writes <out>.meta.json with status (complete/partial/failed), stop_reason, pages requested vs. completed, and the start/final URL. A failed run writes no sidecar, so it can't contradict the previous run's still-intact output.
  • Exit code 6 for a partial run — a timeout or challenge partway through pagination still saves what it gathered but no longer looks identical to a complete run. The site's own pagination running out still exits 0: there was nothing more to fetch.
  • 3-letter ISO currency codes in the fallback parser (AED 100, 100 CHF), matched against an allowlist of real ISO 4217 codes so a size chart (XXL 100) can't become a phantom price. Such tiles previously matched nothing and were dropped as "not a product", losing every product on those locales.
  • diff_runs.py refuses an assortment diff when either side's sidecar says the run was partial (products on never-fetched pages would otherwise read as delisted); --force opts out. It also reports a source_changed bucket for a price that differs alongside price_source — two snapshots that rendered differently, not a site-side change — which --fail-on-change ignores.
  • The parser logs DOM price-confirmation coverage per run, warning below 90%.

Fixed

  • canary.yml never recorded the scraper's exit code. GitHub Actions runs run: steps under bash -e, so a non-zero exit aborted the script before the line that captured it — leaving the entire exit-3-vs-4-vs-124 interpretation dead in exactly the cases it existed for. The canary now also sanity-checks the data it fetched (≥10 products, ≥90% with a non-null price) rather than treating "exit 0" as success.

See CHANGELOG.md for the full list and README.md for usage.