Skip to content

v2.2.0 — Site Recipe Engine

Choose a tag to compare

View all tags
@syswave-dev syswave-dev released this 06 May 17:05
· 61 commits to main since this release
v2.2.0
b9bd0c6

v2.2.0 — 2026-05-06

Added

  • Site Recipe Engine (#18). Declarative site-recipes.json for per-host preprocess, fetch, select, and extractor rules. Default recipes ship in the repo (site-recipes.default.json); self-hosters can mount data/site-recipes.json or set PULLMD_SITE_RECIPES to point elsewhere. Four recipe categories:
    • preprocess — DOM cleanup actions (remove-attr, remove-class, remove-element, unwrap) applied before extraction
    • fetch — render forcing (render: force|skip), wait-for selector, mobile UA
    • select — extra remove-selectors added to cleanDom
    • extractor — preferred extractor per host (readability, trafilatura, playwright)
  • New endpoint GET /api/recipes/status (public, no auth) — counts loaded/rejected recipes per source for monitoring.
  • Cache invalidation on recipe change. When recipe content changes between server boots, all cache rows become stale and re-extract on next access (lazy, on-demand).
  • Playwright sidecar accepts new optional fields: waitFor (CSS selector), waitTimeoutMs (capped at 15000), mobileUa (boolean). Backwards compatible — old fields are silently passed through.
  • Initial default recipes covering Future PLC sites (paywall + recommendation widgets) and GitHub Issues (JS-rendered comments).
  • The Playwright sidecar bundles playwright-stealth to mitigate navigator.webdriver-style headless detection on JS-driven anti-bot pages.

Known limitations

  • Sites behind cookie-based consent walls (third-party CMP frameworks like TCF v2) are not unlocked by recipes alone in this release. Such sites redirect non-consenting visitors to a JS-rendered consent UI and only return article content once HttpOnly cookies are set after a click. A future release will add a fetch.cookies recipe field so operators can paste their own consent state when they choose to. For now, write a custom recipe with whatever combination of select.remove, extractor, and fetch settings works for your specific source — the engine supports the experimentation, the defaults stay conservative.

Important — :latest tag stays on v1.x

The :latest tag in Docker Hub and GHCR remains pinned to v1.x until the scheduled flip on 2026-05-16. Self-hosters wanting the recipe engine must pin :v2.2.0 (or :2.2) explicitly for both pullmd and pullmd-playwright. Pulling :latest continues to give you v1, without the recipe engine.

services:
  pullmd:
    image: aeternalabshq/pullmd:2.2.0
  playwright:
    image: aeternalabshq/pullmd-playwright:2.2.0

Migration

  • New meta table created automatically on first boot. No action required.
  • Existing cache rows remain valid until the first recipe content change is detected.
  • See MIGRATION.md for the full upgrade path.
Assets 2
Loading