AFL-api v0.5.0

AFL-api 0.5.0

v0.4.0 - Operational Reliability Foundation

v0.4.0 strengthens the operational foundations of AFL-api and marks the project’s transition from a collection of scraping scripts into a small, maintainable data service.

This release focuses on database safety, scheduler reliability, scrape observability, controlled administrative operations and safer network access.

Highlights
Added an explicit SQLite migration system for reliable upgrades of existing databases.
Added persistent scheduler job metadata and startup reconciliation.
Added a unified scrape-run audit model across major scraper workflows.
Added authenticated admin controls for manually triggering selected scheduler jobs.
Hardened and documented the intended production network exposure of the API, admin and scheduler services.
Improved scheduler startup and shutdown behaviour when invoked incorrectly or when historical jobs are present.
Added a shared synchronous scraper HTTP client with:
bounded retries for transient failures;
separate connect and read timeouts;
exponential backoff and jitter;
bounded Retry-After support;
per-host, per-process rate limiting;
repository-specific request identification;
sanitised errors and sensitive-value redaction.
Expanded automated coverage for migrations, scheduling, scraper auditing, management controls and HTTP behaviour.
Reliability and compatibility

Existing scraper CLI behaviour remains compatible. Playwright-based scraping has been retained where JavaScript-rendered content may still be required; v0.4.0 does not perform speculative conversions to plain HTTP.

The full automated test suite and Docker build validation pass through GitHub Actions.

Looking ahead

The next development phase will focus on scraper reliability and source intelligence. Planned work includes documenting AFL page structures, collecting representative page fixtures, determining HTTP-versus-Playwright requirements for each source, and progressively separating fetching, parsing and persistence responsibilities.

The architectural review recommends the theme:

v0.5.0 — Scraper Reliability Foundation

That aligns directly with the report’s recommended sequence: source inventory first, fixtures alongside it, HTTP/API discovery next, and scraper restructuring only once the source contracts are understood.

v0.3.0 Operational Hardening & Scraper Reliability

Summary:
Version 0.3.0 focuses on strengthening the operational foundations of the AFL Supplemental API.

This release improves security, testing, deployment, scraper maintainability, and operational confidence without changing the core data model or scraping functionality. It introduces stronger automated validation, Docker health monitoring, improved parser protection against AFL website changes, and a significantly more maintainable scraper architecture through centralized selector management.

The project now has a much stronger platform for future operational improvements, with database migrations, durable scheduler recovery, unified scrape auditing, and enhanced administration planned for the v0.4.x series.

Highlights:
Security

• API keys are now stored as secure hashes rather than plaintext.
• Admin authentication and session security have been strengthened.
• CSRF protection has been added to administrative actions.
• Sensitive headers are redacted from diagnostic output.

Deployment & Operations

• Added Docker healthchecks and application health/readiness endpoints.
• Improved Docker deployment validation.
• Added GitHub Actions Continuous Integration to automatically validate every push and pull request.
• Repository now includes automated Docker build verification.

Testing & Quality

• Introduced fixture-based parser tests using captured AFL HTML.
• Parser behaviour is now protected against unintended regressions.
• Added selector regression tests ensuring scraper implementations consume the shared selector definitions.
• Expanded automated test coverage across security, administration, parser behaviour and infrastructure.

Scraper Maintainability

• Centralised AFL scraper selectors into a shared selector layer.
• Removed duplicated selector definitions across active scraper modules.
• Preserved existing parser behaviour while making future AFL website updates significantly easier to manage.
• Historical scraper implementations remain archived and intentionally isolated from active development.

Developer Experience

• Pytest execution now works consistently in development environments.
• CI provides immediate feedback for contributors.
• Development workflow is more predictable through improved testing and validation.

Current Project Status
Version 0.3.0 represents a significant hardening milestone for the AFL Supplemental API.

Current capabilities include:

• Secure API and administrative interfaces.
• Automated Continuous Integration.
• Docker-based deployment with health monitoring.
• Centralised scraper selector management.
• Deterministic fixture-based parser testing.
• Improved protection against AFL website structural changes.
• Expanded automated regression testing.

Known limitations intentionally deferred to v0.4.x include:

• Database schema migration framework.
• Persistent scheduler recovery after restart.
• Unified scrape execution audit history.
• Manual scraper controls within the admin interface.
• Shared rate-limited HTTP client for non-Playwright scraping.

These remaining items are focused on operational resilience rather than core functionality and form the roadmap for the next development milestone.

v0.2.0 – Stable development baseline

Summary:
This release establishes the first formally versioned development
baseline for AFL Stats. It consolidates the initial API, database,
deployment and security foundations and resolves all identified
Priority 0 review findings.

Highlights:

• Established the core AFL statistics API and supporting services
• Added automated testing and CI validation
• Improved API key storage and authentication security
• Added application health and readiness checks
• Improved Docker build and deployment reliability
• Resolved all Priority 0 findings from the project review

Status:
This remains a pre-1.0 development release. Interfaces, schemas and
deployment requirements may still change before v1.0.0.