Skip to content

Release v1.1.0

Choose a tag to compare

@ivbeg ivbeg released this 03 Jul 12:42

News feed server, reusable parsing specs, caching, and multiple output formats.

Added

  • serve — local HTTP feed server (standard-library only) that turns any page URL into a feed on demand over GET (GET /feed?url=<page>&format=atom), plus /health and / endpoints. Feed URLs can be pasted straight into any RSS reader.
  • analyze — runs the dynamic heuristics once and distills them into a reusable YAML parsing spec; new --spec / -s option for extract runs fast, deterministic extraction from a pre-built spec.
  • Reusable parsing specs via newsworker.spec (FeedSpec, SpecAnalyzer, SpecExtractor) using deterministic CSS/XPath selectors.
  • Caching layer (newsworker.cache) with a spec cache and a content cache (configurable TTL). New --no-cache, --refresh, --config / -c options for extract, and --cache-dir / --content-ttl for serve.
  • Settings support (newsworker.settings) backed by a YAML config file at ~/.newsworker/config.yaml.
  • High-level newsworker.service.FeedService shared by extract and the server.
  • Multiple output formats: extract supports json/rss/atom/csv; scan supports json/rss/atom/csv/opml. New --format / -f and --output / -o options, plus a new newsworker.formats module.

Changed

  • Rewrote README.md with a modern structure and full CLI/caching documentation.
  • extract now builds and caches a parsing spec (plus page content) on first use for faster subsequent runs.
  • scan now emits structured, format-aware output.
  • Added cssselect, pyyaml, requests, urllib3 dependencies (and declared feedparser explicitly).

Fixed

  • Naive datetimes are normalized to UTC when rendering RSS/Atom feeds.

Full changelog: https://github.com/ivbeg/newsworker/blob/master/CHANGELOG.md