Repository navigation
v3.20.0
This release contains security fixes found in an internal review. Upgrading is recommended for every instance.
Before you upgrade
- Update
pullmd-playwrighttogether withpullmd. The sidecar's address policy is at revision 2. With an olderpullmd-playwrightimage,/api/statusreportsdegradedwithpolicy-missingand answers503; renders keep working. - Recommended: move the Playwright sidecar to its own network. The shipped compose files now put it on a
pullmd-rendernetwork that onlypullmdjoins.docker compose pulldoesn't change your compose file, and nothing breaks if you skip this. Docker Compose creates the network itself ondocker compose up -d. - Recommended: run the containers with fewer privileges. The shipped compose files add
no-new-privilegesandcap_drop: [ALL]to all services;pullmdkeepsCHOWN,DAC_READ_SEARCH,SETUIDandSETGID. An older compose file keeps working. - Keep the database under
/data. The image no longer lets the app write to/app, so aCACHE_DBthat points into/appstops working. The shipped compose files already use/data/cache.db. - Run admin commands as the app user:
docker compose exec -u app pullmd node scripts/admin.js ... - Monitors that read counts from
/api/statsor/api/storageneed an admin API key when sign-in is on orDISABLE_PUBLIC_HISTORYis set.
The compose snippets for both recommended steps are in MIGRATION.md and on Upgrading.
Security
- The Playwright sidecar's address guard covers the whole render context. It applied to the page being rendered; now it also checks requests from workers and from any page the site opens itself.
window.openreturnsnull, links and forms aimed at another window do nothing, andSharedWorkeris switched off. Dedicated workers keep working under the same address policy. The sidecar's address policy is now at revision 2, and/api/statusreports apullmd-playwrightimage older than 3.20.0 asdegradedwithpolicy-missing(HTTP503). Update thepullmd-playwrightimage together withpullmd. - The Playwright sidecar removes speculation rules from page markup, on a best-effort basis. Speculative prefetch and prerender bypass the sidecar's request guard. An in-page script disarms
<script type="speculationrules">elements as they are created or parsed. Rules that a page script gets past this removal can still fire, and so can rules sent in aSpeculation-Rulesresponse header; that residual is described inSECURITY.md. - The shipped compose files put the Playwright sidecar on its own network. It moves from
pullmd-internalto a newpullmd-renderbridge network that onlypullmdjoins, so page scripts running in the browser cannot reach trafilatura or markitdown. A compose file you downloaded earlier keeps the old layout; seeMIGRATION.md. - Over-nested or oversized HTML and XML is refused before it is parsed. Deeply nested markup made the parser and the extraction slow down sharply and could block the server. PullMD now measures the markup first and answers
422with"code": "document_too_complex"when it exceeds a budget. For HTML (fetched pages,POST /api/html, saved pages, and the HTML part of e-mails): 100,000 elements, 200,000 text nodes (each&,<,>and non-breaking space in the text counts as a split), 100,000 child nodes of one element, 200,000 attributes, 1,000 attributes on one element, and a ceiling on the nesting depth summed over all elements. HTML is measured twice: as received, where there is no cap on levels and the depth sum is 100,000,000 (old markup leaves tags open that a browser closes), and again after PullMD has normalised it the way a browser would, where nesting is capped at 256 levels and the depth sum at 10,000,000 (the fallback conversion of an HTML-only e-mail has the first measurement only). For uploaded XML feeds (RSS, Atom, RDF): nesting depth 256 and 1,200,000 nodes; other.xmlfiles are shown as code and not parsed. For OpenDocument files: nesting depth 256 and 1,200,000 nodes incontent.xml(elements, attributes and text), 250,000 instyles.xmland 100,000 inmeta.xml. See the Changed section for what this means for large documents. - Documents are converted in worker threads with a time and memory limit. The budgets above catch the common cases before a parser runs; the worker limit stops what they miss. Web pages, HTML uploads and the formats PullMD reads itself (e-mail, OpenDocument, feeds, data and code files and others) are converted in a pool of
PULLMD_CONVERT_WORKERSworkers (2 by default). A conversion that runs longer thanPULLMD_CONVERT_TIMEOUT_MS(60 seconds of worker time) is stopped and answers422with"code": "conversion_timeout"; one that exhausts a worker's 512 MB heap answers422withdocument_too_complex. Reddit and Hacker News conversions and documents that go to the MarkItDown sidecar don't use the workers.PULLMD_CONVERT_WORKERS=0converts in the server process without these limits and isn't safe for an instance others can reach. SeeSECURITY.mdfor what remains. format=textconversion no longer slows down on crafted Markdown. Some bracket and parenthesis patterns made the plain-text conversion take seconds to minutes on a few kilobytes and held up the whole server meanwhile. Links and images are now converted in a single pass, so the time grows only with the length of the text. The output is unchanged.- A login with an unknown email costs as much as one with a wrong password. The miss ran a cheap dummy check, so response times could tell whether an account exists. It now runs a full password check against a hash made with the instance's own parameters.
scripts/admin.js reset-passwordrevokes the user's OAuth refresh tokens. It already ended the user's sessions; connected OAuth clients now have to sign in again. Access tokens already issued stay valid until they expire, at most 1 hour. API keys are not touched; the user revokes them on/settings.POST /oauth/revokerevokes only tokens issued to the requesting client. A token belonging to another client is left alone, and the answer is200either way. Revoking a token also uses up any authorization code the client holds for the same user that it has not exchanged yet.POST /api/keysandPOST /api/keys/:id/revokerefuse requests from other sites. Like/loginand/logout, they answer403when the browser'sOrigin(or, without one,Referer) names a host other than the instance's own orPUBLIC_URL's. Requests with neither header, such as scripts, are accepted as before.- Every error answers JSON, without a stack trace. An unexpected failure or a rejected request body used to get Express' HTML error page, which includes the stack and file paths unless
NODE_ENV=production. Errors that reach the central error handler now answer{"error": ..., "code": ...}:500with"code": "internal_error"and a generic message (the details go to the server log only),413with"code": "payload_too_large"(the message is unchanged),400with"code": "invalid_body"for a body that is not valid JSON, and any other4xxwith"code": "bad_request". The image also setsNODE_ENV=production. An unknown path still gets Express' short HTML404page. - Every response carries
X-Content-Type-Options: nosniff,X-Frame-Options: SAMEORIGIN, aContent-Security-Policythat includesframe-ancestors 'self', andReferrer-Policy: same-origin. That includes the web app, static files, the sign-in and OAuth pages and error answers. A page on another origin can no longer show PullMD in a frame; if you embed PullMD that way, the browser now refuses it. Own pages can still frame it. - The address guard also blocks IPv4 ranges written as NAT64 local-use or IPv4-translated addresses. Besides the forms it already covered (IPv4-mapped, NAT64 well-known prefix, 6to4, IPv4-compatible), it now blocks the blocked IPv4 ranges when they are embedded in the low 32 bits of an address under
64:ff9b:1::/48(NAT64 local-use) or::ffff:0:0:0/96(IPv4-translated), in the server and in the Playwright sidecar. The sidecar's address policy stays at revision 2. - Images in uploaded documents are not loaded until you ask. In the web app, a file you upload (an e-mail, a saved page, a document) no longer loads its images when you open the result. The Raw view shows the image's Markdown as text, the Rendered view shows its alt text and address, and a bar above the result offers Load images (Bilder laden) for that result. Only
data:andblob:images are not held back; images with an address on the instance itself wait for the click too, and results of URL conversions, history and share links behave as before. Images in a result are now requested without a referrer. - The container no longer lets the app user write to the application files. In the
pullmdimage/appstays owned by root and only/databelongs to the app user, so the app can still write only under/data. ACACHE_DBthat points into/appstops working; keep the database under/data, as the compose files do (CACHE_DB=/data/cache.db). The shipped compose files also give all four servicesno-new-privilegesand drop all capabilities;pullmdkeeps onlyCHOWN,DAC_READ_SEARCH,SETUIDandSETGID, which its entrypoint needs to fix ownership of the data folder and then drop to the app user. Run the admin commands as the app user too (docker compose exec -u app pullmd node scripts/admin.js ...); started as root,scripts/admin.jsswitches to the app user itself. A compose file you downloaded earlier works as before; adding these lines is recommended, seeMIGRATION.md.
Added
PULLMD_CONVERT_TIMEOUT_MSandPULLMD_CONVERT_WORKERS. They set the time limit per conversion (default60000) and the number of conversion workers (default2). The shipped compose files pass both through.- Connected apps on
/settings. A new section lists the OAuth clients (such as claude.ai) that hold a valid refresh token for your account, with when they were connected and last used, and a Revoke button for each. Revoking ends the client's refresh tokens and any authorization code it has not exchanged yet, so it cannot get a new access token; one it already holds expires within 1 hour.
Changed
/api/statsand/api/storageanswer with counts only to callers who may see the global history. That is the admin, or anyone when sign-in is off andDISABLE_PUBLIC_HISTORYis not set. Everyone else still gets200, but/api/storagereturns onlyretentionDaysand/api/statsonlywindow; the per-domain lists went the same way as the counts. A monitor that readstotalorrequests7dfrom these routes needs an admin API key. With sign-in off and public history on, nothing changes. The PWA's status light keeps working; the footer's counts show only when the answer carries them, so a signed-in user who is not the admin no longer sees the footer counts.- Very large pages and documents can now get
422instead of a slow conversion. The budgets above are set well above ordinary pages, but a very long page with more than 100,000 elements or 200,000 text nodes is refused, onGET /api,/api/stream(anerrorevent carrying the code), MCPread_url(an error result) andPOST /api/html. When the automatic browser render produces a page over budget, the static result is kept; withrender=forceorextractor=playwrightthe request is refused. In OpenDocument files each filled spreadsheet cell costs five or six nodes, so a spreadsheet with more than roughly 200,000 to 240,000 filled cells (fewer when the cells carry a lot of formatting) is refused. An e-mail whose HTML part is over budget is converted from its plain-text part when it has one. - A conversion can now answer
503or a timeout. When every conversion worker is busy, up to 16 conversions wait for one. A request that finds that queue full, or waits longer than 30 seconds, answers503with"code": "converter_busy"andRetry-After: 5onGET /api,POST /api/htmlandPOST /api/file;/api/streamsends anerrorevent carrying the code, and MCPread_urlreturns an error result. An async file job reports it withhttpStatus: 503, and the job's status response carriesRetry-Aftertoo. A document that exceedsPULLMD_CONVERT_TIMEOUT_MSanswers422with"code": "conversion_timeout". A page that is over a limit in the static pass is not retried in the browser. If only the automatic browser render finds no free worker, the static result is returned and is not cached.