v1.6.0 — Fix scraping after Cash Converters redesign

Bug fix

Cash Converters redesigned their listing pages, which broke Cash Converters scraping. Four issues, now fixed:

🔍 Searches returned nothing (the main failure)

Browse results now link only to /Listing/Details/<id>/<slug>. The scraper's link filter expected the ID at the end of the URL, so it rejected every result and each scrape reported "Found 0 listings". Verified fixed: 0 → 20 listings.

📝 Titles included the watch-list controls

Titles were coming through as Eufy Cordless Motion Sensor T8910 Add to Watch List Watching View Watchlist > because the old title element no longer exists and the fallback picked up nested buttons. Titles are now read from the page's metadata and come through clean.

⏰ End date and time remaining were blank

Both moved to a different HTML tag, so they came back empty — which also meant ended auctions weren't being expired correctly. Both now populate again.

🗑️ Purged listings no longer error forever

When Cash Converters removes an old listing entirely it returns "410 Gone". Those are now marked as ended instead of failing on every subsequent scrape — previously a manually added listing would keep erroring once purged.

Notes

• Bid increment stays blank on most listings: the field Cash Converters used is gone and its replacement is filled in by JavaScript, which the scraper doesn't run. No other field is affected.
• Dollar Dealers scraping is unaffected, and the TLS certificate fix from v1.5.0 is still working.

v1.5.0 — Fix Cash Converters scraping + diagnostic logging

Bug fix

🔒 Cash Converters scraping works again

Cash Converters recently rotated their TLS certificate and their server now sends only the site certificate, omitting the intermediate that signs it. Browsers paper over this automatically; Node cannot, so every request failed and each scrape finished instantly with 0 results.

The missing intermediate is now bundled with the add-on, so the certificate chain is complete again. Full certificate verification stays enabled — this supplies the missing link rather than turning off security checks.

Diagnostics

🔍 Logs that actually say what went wrong

Previously a failure only ever logged fetch failed. The add-on log now shows the real cause and a plain-English hint, for example:

ERROR [cc] GET https://shop.cashconverters.co.nz/Browse?... failed on attempt 3/3 after 166ms:
  fetch failed | cause: UNABLE_TO_VERIFY_LEAF_SIGNATURE: unable to verify the first certificate
  | hint: the site did not send its full TLS certificate chain
WARN  [cc] 1/1 search term(s) failed: eufy
ERROR [cc] Every search term failed — the site was unreachable or rejected every request

Also added: startup diagnostics (Node version, database path, certificate bundle status), per-run summaries, and retry attempts with timings. Set LOG_LEVEL=debug for a line per HTTP request.

Other fixes found while investigating

• Failed scrapes no longer delete your listings. The "remove listings that disappeared" step ran even when searches failed, so a network outage could wrongly mark healthy listings as ended. It's now skipped unless the search phase fully succeeded.
• Failed scrapes are reported as failures. A run where every search failed was recorded as "completed" with 0 results. It now reports as failed, and the UI says the site couldn't be reached instead of showing a misleading "Found 0 listings".
• The add-on can no longer crash on a scraper error, which previously could restart it mid-scrape.
• Requests now time out after 30 seconds instead of potentially hanging forever (configurable via REQUEST_TIMEOUT_MS).

v1.4.1 — Refresh manually added items

Bug fix

🔄 Manually added items are now refreshed on every scrape

Items added by URL don't match your search terms, so they were never re-fetched — their price, end date and bid count stayed frozen at the moment you added them, and finished ones were never removed.

Each scrape now also refreshes your manually added listings, so they get the same treatment as search results: updated prices/end dates, and automatic removal once the auction has finished (from v1.4.0). Your manual items keep their manual status and grouping.

v1.4.0 — Drop finished auctions

What's new

🧹 Finished auctions are removed automatically

When an auction ends — including when someone else buys it via Buy Now — it used to linger in the list because it still appears in search results for a while. The scraper now checks each listing's status and drops it once the auction has finished ("Successful", "Unsold", "Closed", or "Ended").

Detection uses the listing's server-rendered status label (rather than the JavaScript-rendered end date, which isn't visible to the scraper), so it's reliable. Active and upcoming ("Preview") listings are unaffected.

Finished items clear out on the next scrape — manual or scheduled.

v1.3.0 — Price-filter overrides & accurate notifications

What's new

🏷️ Override the price filter on specific items

Sometimes an item priced above your limit is still worth a look. Now you can keep it visible:

• In the Hidden view, items hidden by the price filter show a Keep visible button
• Kept items appear in the normal list with an Over limit badge
• A Re-filter action puts an item back under the price filter
• Overrides persist across scrapes — a kept item won't be re-hidden

🔔 Notifications now respect the price filter

Previously the new-item alert counted listings before the price filter ran, so you could be notified about items that were actually filtered out. Notifications (and the on-screen completion message) now count only new items that pass the price filter.

Upgrade notes

Existing data is preserved. A new price_override column is added automatically on first run.

v1.2.1 — Fix data loss on update

Bug fix

🐛 Data no longer lost on update

Previously the database was stored at /app/data inside the container image, which gets wiped every time the add-on updates. It is now stored at /data/auctions.db — the directory the HA Supervisor automatically persists across updates, reboots, and reinstalls.

To recover: After updating to this version, re-add your search terms in Settings. Listings will repopulate on the next scrape.

Root cause: The addon_config map entry in config.yaml was also removed — it was pointing at the wrong directory and wasn't being used.

v1.2.0 — HA Add-on, Scheduled Scraping & Notifications

What's new

🏠 Home Assistant Add-on

• Install from HA by adding https://github.com/phillclaxton/auction-scraper as a custom repository
• Appears as Auctions in the HA sidebar via ingress (no port exposure needed)
• Supports amd64, aarch64, and armv7 (Raspberry Pi)

🔔 Home Assistant Notifications

• Sends an alert through HA when new listings are found
• Configurable from the Settings panel — toggle + service name (e.g. persistent_notification or mobile_app_iphone)
• Silently skips when running outside HA

⏰ Configurable Auto-Scrape Schedule

• Runs every 4 hours between 6am–8pm NZ time by default
• Configurable from Settings: interval (2/4/6/8h), start/end hour, live preview
• Changes apply instantly without a restart

🔍 Per-site Search Term Toggles

• Each search term can be independently enabled/disabled for Cash Converters and Dollar Dealers

📊 Scrape History Log

• Every run logged with status, item counts, trigger type (manual/scheduled), and source

Upgrade notes

Existing data is fully preserved — the database schema is backwards compatible.