Fix DAC (Dacorum) scraper#346
Conversation
The https://democracy.dacorum.gov.uk certificate is not trusted by wreq's embedded BoringSSL CA bundle, causing CERTIFICATE_VERIFY_FAILED on every request. Adding verify_requests = False bypasses TLS certificate validation, allowing the ModGov ASMX endpoint to respond with councillor data. We scrape read-only public data so MITM risk is acceptable. Locally verified: 50 councillors, 50 with photos. The council does not publish email addresses via the ModGov XML service.
Re-scrape after 7d88f41Initial fix: Verified locally with
Emails: the Generated by Claude Code |
Re-scrape after 7d88f41 — emails ARE available ✅Re-ran the scraper on this branch. Correction to the PR body: Dacorum does publish email addresses via the ModGov XML service. The No code change required — the existing
Emails captured cleanly, e.g. Delta vs PR body: emails 0 → 50. |
What broke
The
https://democracy.dacorum.gov.ukcertificate is not trusted bywreq's embedded BoringSSL CA bundle. Every request to the ModGov ASMX endpoint fails immediately withCERTIFICATE_VERIFY_FAILED. The dashboard error showed "Request timeout after 30+ seconds" but local reproduction reveals the underlying cause is a TLS certificate validation failure (completes in under 1 second, not a timeout).What was fixed
verify_requests = Falseto theScraperclass incouncillors.pyScrape results
Generated by Claude Code