Skip to content

v3.20.0

Choose a tag to compare

@syswave-dev syswave-dev released this 20 Nov 07:37
· 3 commits to main since this release

This release contains security fixes found in an internal review. Upgrading is recommended for every instance.

Before you upgrade

  • Update pullmd-playwright together with pullmd. The sidecar's address policy is at revision 2. With an older pullmd-playwright image, /api/status reports degraded with policy-missing and answers 503; renders keep working.
  • Recommended: move the Playwright sidecar to its own network. The shipped compose files now put it on a pullmd-render network that only pullmd joins. docker compose pull doesn't change your compose file, and nothing breaks if you skip this. Docker Compose creates the network itself on docker compose up -d.
  • Recommended: run the containers with fewer privileges. The shipped compose files add no-new-privileges and cap_drop: [ALL] to all services; pullmd keeps CHOWN, DAC_READ_SEARCH, SETUID and SETGID. An older compose file keeps working.
  • Keep the database under /data. The image no longer lets the app write to /app, so a CACHE_DB that points into /app stops working. The shipped compose files already use /data/cache.db.
  • Run admin commands as the app user: docker compose exec -u app pullmd node scripts/admin.js ...
  • Monitors that read counts from /api/stats or /api/storage need an admin API key when sign-in is on or DISABLE_PUBLIC_HISTORY is set.

The compose snippets for both recommended steps are in MIGRATION.md and on Upgrading.

Security

  • The Playwright sidecar's address guard covers the whole render context. It applied to the page being rendered; now it also checks requests from workers and from any page the site opens itself. window.open returns null, links and forms aimed at another window do nothing, and SharedWorker is switched off. Dedicated workers keep working under the same address policy. The sidecar's address policy is now at revision 2, and /api/status reports a pullmd-playwright image older than 3.20.0 as degraded with policy-missing (HTTP 503). Update the pullmd-playwright image together with pullmd.
  • The Playwright sidecar removes speculation rules from page markup, on a best-effort basis. Speculative prefetch and prerender bypass the sidecar's request guard. An in-page script disarms <script type="speculationrules"> elements as they are created or parsed. Rules that a page script gets past this removal can still fire, and so can rules sent in a Speculation-Rules response header; that residual is described in SECURITY.md.
  • The shipped compose files put the Playwright sidecar on its own network. It moves from pullmd-internal to a new pullmd-render bridge network that only pullmd joins, so page scripts running in the browser cannot reach trafilatura or markitdown. A compose file you downloaded earlier keeps the old layout; see MIGRATION.md.
  • Over-nested or oversized HTML and XML is refused before it is parsed. Deeply nested markup made the parser and the extraction slow down sharply and could block the server. PullMD now measures the markup first and answers 422 with "code": "document_too_complex" when it exceeds a budget. For HTML (fetched pages, POST /api/html, saved pages, and the HTML part of e-mails): 100,000 elements, 200,000 text nodes (each &, <, > and non-breaking space in the text counts as a split), 100,000 child nodes of one element, 200,000 attributes, 1,000 attributes on one element, and a ceiling on the nesting depth summed over all elements. HTML is measured twice: as received, where there is no cap on levels and the depth sum is 100,000,000 (old markup leaves tags open that a browser closes), and again after PullMD has normalised it the way a browser would, where nesting is capped at 256 levels and the depth sum at 10,000,000 (the fallback conversion of an HTML-only e-mail has the first measurement only). For uploaded XML feeds (RSS, Atom, RDF): nesting depth 256 and 1,200,000 nodes; other .xml files are shown as code and not parsed. For OpenDocument files: nesting depth 256 and 1,200,000 nodes in content.xml (elements, attributes and text), 250,000 in styles.xml and 100,000 in meta.xml. See the Changed section for what this means for large documents.
  • Documents are converted in worker threads with a time and memory limit. The budgets above catch the common cases before a parser runs; the worker limit stops what they miss. Web pages, HTML uploads and the formats PullMD reads itself (e-mail, OpenDocument, feeds, data and code files and others) are converted in a pool of PULLMD_CONVERT_WORKERS workers (2 by default). A conversion that runs longer than PULLMD_CONVERT_TIMEOUT_MS (60 seconds of worker time) is stopped and answers 422 with "code": "conversion_timeout"; one that exhausts a worker's 512 MB heap answers 422 with document_too_complex. Reddit and Hacker News conversions and documents that go to the MarkItDown sidecar don't use the workers. PULLMD_CONVERT_WORKERS=0 converts in the server process without these limits and isn't safe for an instance others can reach. See SECURITY.md for what remains.
  • format=text conversion no longer slows down on crafted Markdown. Some bracket and parenthesis patterns made the plain-text conversion take seconds to minutes on a few kilobytes and held up the whole server meanwhile. Links and images are now converted in a single pass, so the time grows only with the length of the text. The output is unchanged.
  • A login with an unknown email costs as much as one with a wrong password. The miss ran a cheap dummy check, so response times could tell whether an account exists. It now runs a full password check against a hash made with the instance's own parameters.
  • scripts/admin.js reset-password revokes the user's OAuth refresh tokens. It already ended the user's sessions; connected OAuth clients now have to sign in again. Access tokens already issued stay valid until they expire, at most 1 hour. API keys are not touched; the user revokes them on /settings.
  • POST /oauth/revoke revokes only tokens issued to the requesting client. A token belonging to another client is left alone, and the answer is 200 either way. Revoking a token also uses up any authorization code the client holds for the same user that it has not exchanged yet.
  • POST /api/keys and POST /api/keys/:id/revoke refuse requests from other sites. Like /login and /logout, they answer 403 when the browser's Origin (or, without one, Referer) names a host other than the instance's own or PUBLIC_URL's. Requests with neither header, such as scripts, are accepted as before.
  • Every error answers JSON, without a stack trace. An unexpected failure or a rejected request body used to get Express' HTML error page, which includes the stack and file paths unless NODE_ENV=production. Errors that reach the central error handler now answer {"error": ..., "code": ...}: 500 with "code": "internal_error" and a generic message (the details go to the server log only), 413 with "code": "payload_too_large" (the message is unchanged), 400 with "code": "invalid_body" for a body that is not valid JSON, and any other 4xx with "code": "bad_request". The image also sets NODE_ENV=production. An unknown path still gets Express' short HTML 404 page.
  • Every response carries X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, a Content-Security-Policy that includes frame-ancestors 'self', and Referrer-Policy: same-origin. That includes the web app, static files, the sign-in and OAuth pages and error answers. A page on another origin can no longer show PullMD in a frame; if you embed PullMD that way, the browser now refuses it. Own pages can still frame it.
  • The address guard also blocks IPv4 ranges written as NAT64 local-use or IPv4-translated addresses. Besides the forms it already covered (IPv4-mapped, NAT64 well-known prefix, 6to4, IPv4-compatible), it now blocks the blocked IPv4 ranges when they are embedded in the low 32 bits of an address under 64:ff9b:1::/48 (NAT64 local-use) or ::ffff:0:0:0/96 (IPv4-translated), in the server and in the Playwright sidecar. The sidecar's address policy stays at revision 2.
  • Images in uploaded documents are not loaded until you ask. In the web app, a file you upload (an e-mail, a saved page, a document) no longer loads its images when you open the result. The Raw view shows the image's Markdown as text, the Rendered view shows its alt text and address, and a bar above the result offers Load images (Bilder laden) for that result. Only data: and blob: images are not held back; images with an address on the instance itself wait for the click too, and results of URL conversions, history and share links behave as before. Images in a result are now requested without a referrer.
  • The container no longer lets the app user write to the application files. In the pullmd image /app stays owned by root and only /data belongs to the app user, so the app can still write only under /data. A CACHE_DB that points into /app stops working; keep the database under /data, as the compose files do (CACHE_DB=/data/cache.db). The shipped compose files also give all four services no-new-privileges and drop all capabilities; pullmd keeps only CHOWN, DAC_READ_SEARCH, SETUID and SETGID, which its entrypoint needs to fix ownership of the data folder and then drop to the app user. Run the admin commands as the app user too (docker compose exec -u app pullmd node scripts/admin.js ...); started as root, scripts/admin.js switches to the app user itself. A compose file you downloaded earlier works as before; adding these lines is recommended, see MIGRATION.md.

Added

  • PULLMD_CONVERT_TIMEOUT_MS and PULLMD_CONVERT_WORKERS. They set the time limit per conversion (default 60000) and the number of conversion workers (default 2). The shipped compose files pass both through.
  • Connected apps on /settings. A new section lists the OAuth clients (such as claude.ai) that hold a valid refresh token for your account, with when they were connected and last used, and a Revoke button for each. Revoking ends the client's refresh tokens and any authorization code it has not exchanged yet, so it cannot get a new access token; one it already holds expires within 1 hour.

Changed

  • /api/stats and /api/storage answer with counts only to callers who may see the global history. That is the admin, or anyone when sign-in is off and DISABLE_PUBLIC_HISTORY is not set. Everyone else still gets 200, but /api/storage returns only retentionDays and /api/stats only window; the per-domain lists went the same way as the counts. A monitor that reads total or requests7d from these routes needs an admin API key. With sign-in off and public history on, nothing changes. The PWA's status light keeps working; the footer's counts show only when the answer carries them, so a signed-in user who is not the admin no longer sees the footer counts.
  • Very large pages and documents can now get 422 instead of a slow conversion. The budgets above are set well above ordinary pages, but a very long page with more than 100,000 elements or 200,000 text nodes is refused, on GET /api, /api/stream (an error event carrying the code), MCP read_url (an error result) and POST /api/html. When the automatic browser render produces a page over budget, the static result is kept; with render=force or extractor=playwright the request is refused. In OpenDocument files each filled spreadsheet cell costs five or six nodes, so a spreadsheet with more than roughly 200,000 to 240,000 filled cells (fewer when the cells carry a lot of formatting) is refused. An e-mail whose HTML part is over budget is converted from its plain-text part when it has one.
  • A conversion can now answer 503 or a timeout. When every conversion worker is busy, up to 16 conversions wait for one. A request that finds that queue full, or waits longer than 30 seconds, answers 503 with "code": "converter_busy" and Retry-After: 5 on GET /api, POST /api/html and POST /api/file; /api/stream sends an error event carrying the code, and MCP read_url returns an error result. An async file job reports it with httpStatus: 503, and the job's status response carries Retry-After too. A document that exceeds PULLMD_CONVERT_TIMEOUT_MS answers 422 with "code": "conversion_timeout". A page that is over a limit in the static pass is not retried in the browser. If only the automatic browser render finds no free worker, the static result is returned and is not cached.