v1.2.0 — Charset fixes, UA rotation, extraction improvements

This release focuses on extraction quality and self-hoster experience. Six community-reported issues fixed, several with direct contributions from @WinFuture23 and @andrewthetechie.

Bug fixes

• Volume mount fix — Docker compose files now mount to /data matching the actual code path. New CACHE_DB env var for explicit control. (#15, PR #16 by @andrewthetechie)
• MCP response visibility — Dropped structuredContent from MCP responses; metadata now embedded as YAML frontmatter in content[0].text. Claude Code and other MCP clients see the full markdown instead of just metadata. Added share_url field so LLMs stop hallucinating pullmd.com. (#1, #9, #11)
• Charset detection — Pages without explicit charset in the HTTP header (e.g. winfuture.de) are now correctly decoded via BOM / <meta charset> / <meta http-equiv> detection. Fixes mojibake on ISO-8859-1 and Windows-1252 pages. (#8)

New features

• User-Agent rotation — Outbound requests now rotate through a pool of current desktop browser UAs, live-refreshed every 48h from @WinFuture23's real-world-user-agents feed. Static seed pool as fallback. Configurable via PULLMD_USER_AGENT (pin single UA) and PULLMD_UA_FEED_URL (custom feed or disable). (#14)
• HTML preprocessing — New preprocessing step strips aria-hidden from <p> elements and removes widget class tokens from article containers before extraction. Recovers silently dropped paragraphs on Future PLC sites (WindowsCentral, GamesRadar, TechRadar, etc.). (#17)
• Extractor override — New ?extractor=readability|trafilatura|playwright query param and MCP read_url parameter to skip pickBest and force a specific extractor. (#17)

New env vars

Variable	Purpose
CACHE_DB	Explicit path to the SQLite cache database
PULLMD_USER_AGENT	Pin a single User-Agent for all outbound requests
PULLMD_UA_FEED_URL	Custom UA feed URL, or empty string to disable live refresh

Architecture

• Filed #18 — Site recipe engine as the long-term home for per-site extraction rules. The Future PLC preprocessing in this release is generic enough to ship as code, but follow-on patterns will move into a declarative recipe file.

Contributors

Thanks to @WinFuture23 (Sebastian Kuhbach) for five detailed issue reports with reproduction steps, root-cause analysis, and suggested fixes — including the UA feed repo that's now integrated as a default data source. And to @andrewthetechie for PR #16 that caught the volume mount mismatch.