Skip to content

v0.3.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@grregis grregis released this 14 Apr 06:31
v0.3.0-alpha
a7cb132

MuckScraper v0.3.0

Release Date: April 14, 2026

This is the largest update since the initial release. v0.3.0 adds a newspaper-style
front page, multi-source deep reports, a two-stage headline ranking pipeline,
authentication, automatic bad-scrape detection, and a complete architectural refactor.

What's New

Newspaper Front Page
MuckScraper now has a proper front page at /headlines. A ranked top 20 stories
displayed in a broadsheet layout — masthead, lead story, 3-column grid, article
images filtered to grayscale newsprint style, weather widget, dark mode. No login
required, safe to expose publicly.

Deep Reports
Multi-source stories now get full analytical reports, not just summaries. Six
topic-aware formats: political stories get a left/center/right framing breakdown,
science stories get findings and expert commentary, sports stories get recaps and
standings context, business stories get market impact and analyst commentary.
Generated automatically for top headline stories after each fetch.

Headline Ranking Pipeline
Stories are now ranked by a two-stage scoring system. Stage one polls 16 RSS feeds
across the political spectrum and uses an LLM to match external headlines to
MuckScraper stories, producing a coverage-weighted base score. Stage two asks the
LLM to re-rank the top 50 candidates by real-world importance. Final score is a
60/40 blend. Every run is logged to the database for auditability.

Authentication
Admin routes are now protected by login. The public headlines, story, and article
pages remain open. Run create_admin.py to set up credentials.

Scrape Blocklist
Bad scrapes are now detected and blocked automatically. The system recognises login
walls, captcha pages, bot-detection responses, and outlet-level duplicate content
(the same HTML returned for every blocked request). Offending domains are added to
a blocklist and skipped on future scrapes. Hard paywalls (NYT, WSJ, FT, Bloomberg,
and others) are pre-blocked. An audit function scans all stored content retroactively.
Manage the list at /scrape-blocklist.

Performance
Story re-grouping now uses pgvector's native nearest-neighbour SQL operator instead
of a Python cosine loop, making it significantly faster as the database grows.
Flask development server replaced with Gunicorn.

Architecture

Routes are now split into three blueprints:

  • personal — public read-only routes (headlines, story, article)
  • admin — authenticated write and trigger routes
  • auth — login/logout

Template filters and shared constants extracted into their own modules.
create_app() is now wiring only.

Upgrading from 0.2.x

Run migrations after pulling:

docker compose down
docker compose up --build
docker compose exec app flask db upgrade
docker compose exec app python create_admin.py

New environment variables (all optional):
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=

Full Changelog

See CHANGELOG.md for the complete list of changes, fixes, and technical details.

Assets 2
Loading