0.1.3

MuckScraper v0.1.3

What's New

Article Scraping Improvements

• Added python-readability for smarter article content extraction — significantly better results across all sites
• Added Googlebot user agent fallback for soft-paywalled sites (Axios, Politico, The Atlantic, and others)
• Added archive.ph fallback when all other scraping strategies fail
• Added per-article [scrape] button for articles missing full text
• Added global ↻ Scrape Missing sidebar button to bulk re-scrape up to 20 articles at a time

Story Grouping & Summarization

• Fixed Ollama catchup button breaking article links and summarization — was caused by session corruption and missing db.session.flush() before story deletion
• Fixed re-grouping creating new stories instead of only matching existing ones
• Capped auto-summarization to 10 stories per batch to prevent Flask request timeouts
• Fixed HTML tags being sent to Ollama in summaries — content is now stripped to plain text before summarization
• Increased content snippet size from 500 to 1500 chars per article for richer summaries

UI

• Added sticky sidebar with purple drop shadow
• Added dark/light mode toggle — preference saved in localStorage
• Added Smart Brevity summary format with labeled sections and bullet points
• Added full article reader page at /article/<id> showing scraped HTML content
• Improved story title cleanup — no longer aggressively truncates at : and -
• Purple accent color (#a43cb5) throughout the UI

Infrastructure

• Added DB indexes on articles.url, articles.date, articles.outlet_id, articles.story_id, articles.bias_score, and stories.created_at
• Added raw API payload storage with 30-day auto-cleanup
• Added AppSetting model for persisting state across restarts
• Added smart restart timer — skips fetch on startup if last fetch was less than 3 hours ago
• Added automatic Ollama catchup when scheduler detects Ollama came back online
• Added manual ↻ Ollama Catchup sidebar button
• Added restart.sh script for soft rebuilds that preserve the database
• Removed standalone news_fetcher container — scheduler handles all fetching

Bug Fixes

• Fixed race condition causing duplicate topic creation when scheduler and news_fetcher ran simultaneously
• Fixed scheduler running stale cached code after restarts
• Fixed POSTGRES_USER typo in docker-compose.yml

Known Limitations

• No authentication — do not expose to the internet
• Development server only
• NYT, Washington Post and hard-paywalled sites still difficult to scrape fully
• Story clustering quality depends on Ollama being online during fetches