Skip to content

v1.1.0 — Playwright fallback + live SSE status

Choose a tag to compare

View all tags
@syswave-dev syswave-dev released this 27 Apr 10:05
· 123 commits to main since this release
v1.1.0
63c3258

Highlights

Playwright fallback for JavaScript-rendered pages. When Readability+Trafilatura returns body-soup or low-quality output, the pipeline now retries via headless Chromium and re-extracts on the rendered DOM. Verified against mistral.ai/pricing — the data is hydrated client-side from __NEXT_DATA__, which static extraction cannot reach. Source label is playwright and the predicate reason is preserved in metadata.extractorReason.

Live status events for the PWA via Server-Sent Events. New GET /api/stream endpoint emits fetching → extracting → (rendering) → result so the user sees what's happening when an extraction takes the slow path. Other clients (curl, Claude, MCP) continue to use /api unchanged.

What's new

  • New Python sidecar pullmd-playwright (FastAPI + Chromium, asyncio.Semaphore concurrency limit, 20s hard timeout, soft-fail networkidle wait)
  • lib/render-decision.js — predicate (i) readability fellback + thin, (ii) body-soup signature heading/paragraph ratio, (iii) low-quality safety net
  • ?render=force and ?render=skip manual overrides on both /api and /api/stream
  • Frontend status line under the spinner with i18n (DE/EN); EventSource auto-falls-back to plain fetch on transport failure; in-flight EventSource is closed on same-tab navigation
  • client_mode query-param fallback so PWA-originated SSE requests keep their attribution
  • Cache-ordering fix from v1.0.3 included here for completeness (re-fetched URLs now bubble to the top of the recents list)

Operator notes

  • Adds a ~3.7 GB Docker image for the Playwright sidecar (pulls Chromium on first build). The image is optional — leave PLAYWRIGHT_URL unset and the pipeline silently degrades to static extraction with a fallback note in metadata.extractorReason.
  • The /api/stream endpoint and ?render=force are not rate-limited — recommended to keep PullMD behind an authenticating reverse proxy if exposing to the internet.

Tests

205 / 205 green (22 new across render-decision, playwright-client, orchestrator, server SSE, render-param branches).

Assets 2
Loading