v0.5.0

MuckScraper v0.5.0 Beta

Release Date: June 3, 2026

MuckScraper v0.5.0 moves the project into beta. The core fetch, grouping, admin, and edition workflows are now usable end-to-end on a fresh install. The remaining rough edges are mostly around source balance, scrape quality, headline tuning, and runtime optimization rather than basic project viability.

What's New

Fresh Install Bootstrap

Fresh installs are now much easier. bootstrap_admin.py initializes the database, stamps Alembic current for a fresh schema, and creates or updates the admin user from .env.

New .env values:

ADMIN_USERNAME=admin
ADMIN_EMAIL=admin@example.com
ADMIN_PASSWORD=replace_with_a_real_admin_password

Recommended first-run flow:

docker compose up -d --build postgres meilisearch app
docker compose exec app python bootstrap_admin.py
docker compose up -d scheduler

Scrape Telemetry and Quality Checks

Scrape telemetry is now more complete across ingestion, retries, and admin tooling. MuckScraper stores scrape status, method, failure reason, HTTP status, and scrape outcome history over time.

Low-value article detection now catches more weak inputs, including roundups, live updates, video pages, and repetitive bad scrape output.

Admin and Operations Improvements

This release adds or improves:

• centralized Admin Tools page
• Meilisearch-backed admin search support
• rebuild/status endpoints for search
• expanded troubleshooting documentation
• clearer run metrics for fetch and scrape checks

Documentation Cleanup

The README and project docs were cleaned up for public use. Personal integrations like Langfuse, n8n, and Matrix notifications are no longer part of the default Docker Compose setup.

They can still be added with custom environment variables, compose overrides, or notification code.

Fixes

• fixed fresh installs failing when create_admin.py ran before the users table existed
• fixed Compose warnings for removed personal Langfuse/n8n variables
• fixed editions sometimes exceeding the 20-story target
• fixed unchanged previous-edition stories repeating too aggressively
• fixed title-only stories producing hallucinated summaries/deep reports
• reduced duplicate same-event clusters in published editions
• reduced excessive retry churn on chronically blocked scrape domains
• cleaned up misleading public docs and deployment-specific references

Upgrade Notes

For an existing install:

git pull
docker compose up -d --build
docker compose exec app flask db upgrade

For a fresh install:

git clone https://github.com/grregis/MuckScraper.git
cd MuckScraper
cp .env.sample .env
# Edit .env
docker compose up -d --build postgres meilisearch app
docker compose exec app python bootstrap_admin.py
docker compose up -d scheduler

create_admin.py still exists, but bootstrap_admin.py is now the recommended first-run setup path.

v0.3.0

MuckScraper v0.3.0

Release Date: April 14, 2026

This is the largest update since the initial release. v0.3.0 adds a newspaper-style
front page, multi-source deep reports, a two-stage headline ranking pipeline,
authentication, automatic bad-scrape detection, and a complete architectural refactor.

---

What's New

Newspaper Front Page
MuckScraper now has a proper front page at /headlines. A ranked top 20 stories
displayed in a broadsheet layout — masthead, lead story, 3-column grid, article
images filtered to grayscale newsprint style, weather widget, dark mode. No login
required, safe to expose publicly.

Deep Reports
Multi-source stories now get full analytical reports, not just summaries. Six
topic-aware formats: political stories get a left/center/right framing breakdown,
science stories get findings and expert commentary, sports stories get recaps and
standings context, business stories get market impact and analyst commentary.
Generated automatically for top headline stories after each fetch.

Headline Ranking Pipeline
Stories are now ranked by a two-stage scoring system. Stage one polls 16 RSS feeds
across the political spectrum and uses an LLM to match external headlines to
MuckScraper stories, producing a coverage-weighted base score. Stage two asks the
LLM to re-rank the top 50 candidates by real-world importance. Final score is a
60/40 blend. Every run is logged to the database for auditability.

Authentication
Admin routes are now protected by login. The public headlines, story, and article
pages remain open. Run create_admin.py to set up credentials.

Scrape Blocklist
Bad scrapes are now detected and blocked automatically. The system recognises login
walls, captcha pages, bot-detection responses, and outlet-level duplicate content
(the same HTML returned for every blocked request). Offending domains are added to
a blocklist and skipped on future scrapes. Hard paywalls (NYT, WSJ, FT, Bloomberg,
and others) are pre-blocked. An audit function scans all stored content retroactively.
Manage the list at /scrape-blocklist.

Performance
Story re-grouping now uses pgvector's native nearest-neighbour SQL operator instead
of a Python cosine loop, making it significantly faster as the database grows.
Flask development server replaced with Gunicorn.

---

Architecture

Routes are now split into three blueprints:

• personal — public read-only routes (headlines, story, article)
• admin — authenticated write and trigger routes
• auth — login/logout

Template filters and shared constants extracted into their own modules.
create_app() is now wiring only.

---

Upgrading from 0.2.x

Run migrations after pulling:

docker compose down
docker compose up --build
docker compose exec app flask db upgrade
docker compose exec app python create_admin.py

New environment variables (all optional):
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_HOST=

---

Full Changelog

See CHANGELOG.md for the complete list of changes, fixes, and technical details.

v0.2.2

MuckScraper v0.2.2

What's New

Cleaner UI

The sidebar has been reorganized — maintenance actions (Force Re-group, Reclassify Topics, etc.) have moved into a hamburger menu (☰) in the top right corner, keeping the sidebar focused on navigation. The sidebar is now collapsible to give more screen space to articles. State is remembered between sessions.

Grouped Stories View

A dedicated Grouped Stories view now shows only stories covered by multiple outlets — the most newsworthy content — paginated at 50 per page. This is the fastest way to see what's actually being widely covered.

Smarter Story Ordering and Matching

Stories are now ordered by the date of their most recent article rather than when the story was first created, so the freshest news always appears first. Story matching now uses single linkage — a new article is compared against every article already in a story rather than just the first one, which improves grouping accuracy.

Aggregator Deduplication

When a story has articles from both original outlets and aggregators (Yahoo News, Google News, MSN, AOL), the aggregator versions are now hidden automatically. You see the original reporting instead.

Timestamps in Local Time

Article published and fetched dates are now shown in your local timezone rather than UTC.

Database Migration Required

If upgrading from v0.2.1, run this in pgAdmin:

ALTER TABLE articles ADD COLUMN IF NOT EXISTS fetched_at TIMESTAMP DEFAULT NOW();

Known Limitations

• No authentication — do not expose to the internet
• Development server only
• cleanup_duplicates.py is a work in progress

v0.2.1

MuckScraper v0.2.1

What's New

AI-Generated Story Headlines

Multi-article stories now get a wire service style headline generated by Ollama — who/what/where in one line, 15 words max. Headlines are generated automatically when a second article is added to a story, and during Ollama Catchup for existing grouped stories. Single-article stories continue to use the auto-generated title from the first article.

Single-Article Story Filter

A toggle on the main page lets you hide single-article stories and focus only on stories covered by multiple outlets — the most newsworthy content. Default is off so nothing changes unless you enable it.

Logging Cleanup

All print() statements across the news_fetcher modules have been replaced with Python's logging module, making log output cleaner and properly leveled (INFO, WARNING, ERROR).

Known Limitations

• Story grouping quality still being tuned — some unrelated articles may be grouped together
• No authentication — do not expose to the internet

v0.2.0

MuckScraper v0.2.0

This is a significant release — the core story grouping and topic classification systems have been completely rebuilt.

Major Changes

Vector Embedding Story Clustering

Story grouping now uses pgvector and nomic-embed-text embeddings instead of asking the LLM to compare headlines. Articles are matched to existing stories using cosine similarity — faster, more accurate, and works even when Ollama's chat model is busy. Requires pulling nomic-embed-text on your Ollama instance and adding pgvector/pgvector:pg14 as your PostgreSQL image.

LLM Topic Classification

Articles are now classified into topics by Ollama based on their actual content, replacing the old system where topic tags came from which API call fetched the article. This means a gaming article that appeared in a US headlines fetch will now correctly appear under Gaming, not US Headlines. New topics: US Headlines, US Politics, International Headlines, Science/Technology, Gaming, Sports, Business/Finance, Other.

Pagination

The main articles page now shows 25 stories per page with prev/next navigation.

New Maintenance Buttons

• ⚡ Force Re-group — rebuild all story groupings from scratch using vector similarity
• ↻ Reclassify Topics — reclassify all existing articles into the new topic system
• ⚡ Wake Ollama — send a Wake on LAN magic packet to your Ollama machine
• ↻ Scrape Missing — bulk re-scrape articles missing full text
• Per-article [scrape] link for individual articles missing content

Scraping Improvements

• python-readability now used as primary content extractor — significantly better results
• Googlebot user agent fallback for soft-paywalled sites (Axios, Politico, The Atlantic)
• archive.ph fallback as last resort

Breaking Changes

• PostgreSQL image must be changed to pgvector/pgvector:pg14 in docker-compose.yml
• nomic-embed-text must be pulled on your Ollama instance: ollama pull nomic-embed-text
• Run CREATE EXTENSION IF NOT EXISTS vector; in pgAdmin after upgrading
• Run ALTER TABLE articles ADD COLUMN IF NOT EXISTS embedding vector(768); in pgAdmin

Known Limitations

• No authentication — do not expose to the internet
• Development server only
• Topic classification accuracy varies by LLM model
• Hard-paywalled sites still difficult to scrape

0.1.3

MuckScraper v0.1.3

What's New

Article Scraping Improvements

• Added python-readability for smarter article content extraction — significantly better results across all sites
• Added Googlebot user agent fallback for soft-paywalled sites (Axios, Politico, The Atlantic, and others)
• Added archive.ph fallback when all other scraping strategies fail
• Added per-article [scrape] button for articles missing full text
• Added global ↻ Scrape Missing sidebar button to bulk re-scrape up to 20 articles at a time

Story Grouping & Summarization

• Fixed Ollama catchup button breaking article links and summarization — was caused by session corruption and missing db.session.flush() before story deletion
• Fixed re-grouping creating new stories instead of only matching existing ones
• Capped auto-summarization to 10 stories per batch to prevent Flask request timeouts
• Fixed HTML tags being sent to Ollama in summaries — content is now stripped to plain text before summarization
• Increased content snippet size from 500 to 1500 chars per article for richer summaries

UI

• Added sticky sidebar with purple drop shadow
• Added dark/light mode toggle — preference saved in localStorage
• Added Smart Brevity summary format with labeled sections and bullet points
• Added full article reader page at /article/<id> showing scraped HTML content
• Improved story title cleanup — no longer aggressively truncates at : and -
• Purple accent color (#a43cb5) throughout the UI

Infrastructure

• Added DB indexes on articles.url, articles.date, articles.outlet_id, articles.story_id, articles.bias_score, and stories.created_at
• Added raw API payload storage with 30-day auto-cleanup
• Added AppSetting model for persisting state across restarts
• Added smart restart timer — skips fetch on startup if last fetch was less than 3 hours ago
• Added automatic Ollama catchup when scheduler detects Ollama came back online
• Added manual ↻ Ollama Catchup sidebar button
• Added restart.sh script for soft rebuilds that preserve the database
• Removed standalone news_fetcher container — scheduler handles all fetching

Bug Fixes

• Fixed race condition causing duplicate topic creation when scheduler and news_fetcher ran simultaneously
• Fixed scheduler running stale cached code after restarts
• Fixed POSTGRES_USER typo in docker-compose.yml

Known Limitations

• No authentication — do not expose to the internet
• Development server only
• NYT, Washington Post and hard-paywalled sites still difficult to scrape fully
• Story clustering quality depends on Ollama being online during fetches

0.1.2

MuckScraper v0.1.2-alpha — Initial Release

MuckScraper is a self-hosted news aggregator with LLM-powered bias scoring and summarization. This is an early alpha release — expect rough edges.

What it does

• Fetches articles from NewsAPI and GNews across configurable topics (US Headlines, World Headlines, US Politics, Technology, Gaming)
• Scrapes full article text using BeautifulSoup with Playwright fallback for JavaScript-heavy sites
• Scores outlet bias on a 1–5 left-to-right scale using a local Ollama LLM
• Groups articles into stories using LLM-assisted clustering across all topics
• Generates on-demand AI summaries of stories via Ollama
• Runs on a 3-hour fetch schedule with smart restart detection to avoid redundant API calls
• Clean UI with topic sidebar, bias tags, full article reader, and Ollama status indicator
• Fully self-hosted via Docker Compose with PostgreSQL

Requirements

• Docker and Docker Compose
• NewsAPI and GNews API keys (both have free tiers)
• Ollama running on your network with a compatible model

Known Limitations

• Development server only — not production hardened
• NYT, Washington Post, and some paywalled sites cannot be fully scraped
• Story clustering quality depends on your Ollama model
• No user authentication

See README.md and .env.example to get started.