Skip to content

v0.2.0

Pre-release
Pre-release

Choose a tag to compare

View all tags
@grregis grregis released this 20 Mar 02:28
v0.2.0-alpha
0b729be

MuckScraper v0.2.0

This is a significant release — the core story grouping and topic classification systems have been completely rebuilt.

Major Changes

Vector Embedding Story Clustering

Story grouping now uses pgvector and nomic-embed-text embeddings instead of asking the LLM to compare headlines. Articles are matched to existing stories using cosine similarity — faster, more accurate, and works even when Ollama's chat model is busy. Requires pulling nomic-embed-text on your Ollama instance and adding pgvector/pgvector:pg14 as your PostgreSQL image.

LLM Topic Classification

Articles are now classified into topics by Ollama based on their actual content, replacing the old system where topic tags came from which API call fetched the article. This means a gaming article that appeared in a US headlines fetch will now correctly appear under Gaming, not US Headlines. New topics: US Headlines, US Politics, International Headlines, Science/Technology, Gaming, Sports, Business/Finance, Other.

Pagination

The main articles page now shows 25 stories per page with prev/next navigation.

New Maintenance Buttons

  • ⚡ Force Re-group — rebuild all story groupings from scratch using vector similarity
  • ↻ Reclassify Topics — reclassify all existing articles into the new topic system
  • ⚡ Wake Ollama — send a Wake on LAN magic packet to your Ollama machine
  • ↻ Scrape Missing — bulk re-scrape articles missing full text
  • Per-article [scrape] link for individual articles missing content

Scraping Improvements

  • python-readability now used as primary content extractor — significantly better results
  • Googlebot user agent fallback for soft-paywalled sites (Axios, Politico, The Atlantic)
  • archive.ph fallback as last resort

Breaking Changes

  • PostgreSQL image must be changed to pgvector/pgvector:pg14 in docker-compose.yml
  • nomic-embed-text must be pulled on your Ollama instance: ollama pull nomic-embed-text
  • Run CREATE EXTENSION IF NOT EXISTS vector; in pgAdmin after upgrading
  • Run ALTER TABLE articles ADD COLUMN IF NOT EXISTS embedding vector(768); in pgAdmin

Known Limitations

  • No authentication — do not expose to the internet
  • Development server only
  • Topic classification accuracy varies by LLM model
  • Hard-paywalled sites still difficult to scrape
Assets 2
Loading