Skip to content

v0.3.0 Operational Hardening & Scraper Reliability

Choose a tag to compare

View all tags
@JustPlausible JustPlausible released this 21 Jul 16:44
v0.3.0
aeeb217
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Summary:
Version 0.3.0 focuses on strengthening the operational foundations of the AFL Supplemental API.

This release improves security, testing, deployment, scraper maintainability, and operational confidence without changing the core data model or scraping functionality. It introduces stronger automated validation, Docker health monitoring, improved parser protection against AFL website changes, and a significantly more maintainable scraper architecture through centralized selector management.

The project now has a much stronger platform for future operational improvements, with database migrations, durable scheduler recovery, unified scrape auditing, and enhanced administration planned for the v0.4.x series.

Highlights:
Security

  • API keys are now stored as secure hashes rather than plaintext.
  • Admin authentication and session security have been strengthened.
  • CSRF protection has been added to administrative actions.
  • Sensitive headers are redacted from diagnostic output.

Deployment & Operations

  • Added Docker healthchecks and application health/readiness endpoints.
  • Improved Docker deployment validation.
  • Added GitHub Actions Continuous Integration to automatically validate every push and pull request.
  • Repository now includes automated Docker build verification.

Testing & Quality

  • Introduced fixture-based parser tests using captured AFL HTML.
  • Parser behaviour is now protected against unintended regressions.
  • Added selector regression tests ensuring scraper implementations consume the shared selector definitions.
  • Expanded automated test coverage across security, administration, parser behaviour and infrastructure.

Scraper Maintainability

  • Centralised AFL scraper selectors into a shared selector layer.
  • Removed duplicated selector definitions across active scraper modules.
  • Preserved existing parser behaviour while making future AFL website updates significantly easier to manage.
  • Historical scraper implementations remain archived and intentionally isolated from active development.

Developer Experience

  • Pytest execution now works consistently in development environments.
  • CI provides immediate feedback for contributors.
  • Development workflow is more predictable through improved testing and validation.

Current Project Status
Version 0.3.0 represents a significant hardening milestone for the AFL Supplemental API.

Current capabilities include:

  • Secure API and administrative interfaces.
  • Automated Continuous Integration.
  • Docker-based deployment with health monitoring.
  • Centralised scraper selector management.
  • Deterministic fixture-based parser testing.
  • Improved protection against AFL website structural changes.
  • Expanded automated regression testing.

Known limitations intentionally deferred to v0.4.x include:

  • Database schema migration framework.
  • Persistent scheduler recovery after restart.
  • Unified scrape execution audit history.
  • Manual scraper controls within the admin interface.
  • Shared rate-limited HTTP client for non-Playwright scraping.

These remaining items are focused on operational resilience rather than core functionality and form the roadmap for the next development milestone.

Assets 2
Loading