v0.4.0 - Operational Reliability Foundation

v0.4.0 strengthens the operational foundations of AFL-api and marks the project’s transition from a collection of scraping scripts into a small, maintainable data service.

This release focuses on database safety, scheduler reliability, scrape observability, controlled administrative operations and safer network access.

Highlights
Added an explicit SQLite migration system for reliable upgrades of existing databases.
Added persistent scheduler job metadata and startup reconciliation.
Added a unified scrape-run audit model across major scraper workflows.
Added authenticated admin controls for manually triggering selected scheduler jobs.
Hardened and documented the intended production network exposure of the API, admin and scheduler services.
Improved scheduler startup and shutdown behaviour when invoked incorrectly or when historical jobs are present.
Added a shared synchronous scraper HTTP client with:
bounded retries for transient failures;
separate connect and read timeouts;
exponential backoff and jitter;
bounded Retry-After support;
per-host, per-process rate limiting;
repository-specific request identification;
sanitised errors and sensitive-value redaction.
Expanded automated coverage for migrations, scheduling, scraper auditing, management controls and HTTP behaviour.
Reliability and compatibility

Existing scraper CLI behaviour remains compatible. Playwright-based scraping has been retained where JavaScript-rendered content may still be required; v0.4.0 does not perform speculative conversions to plain HTTP.

The full automated test suite and Docker build validation pass through GitHub Actions.

Looking ahead

The next development phase will focus on scraper reliability and source intelligence. Planned work includes documenting AFL page structures, collecting representative page fixtures, determining HTTP-versus-Playwright requirements for each source, and progressively separating fetching, parsing and persistence responsibilities.

The architectural review recommends the theme:

v0.5.0 — Scraper Reliability Foundation

That aligns directly with the report’s recommended sequence: source inventory first, fixtures alongside it, HTTP/API discovery next, and scraper restructuring only once the source contracts are understood.