x-reader v0.5.0 — Evidence Receipts and safer source reading

Added

• Hosted Evidence Receipt web reader (x-reader-web, public beta at xreader.leolabs.me): public X, YouTube and generic web URLs, with a PASS / PARTIAL / FAIL / UNKNOWN receipt for each read. Deployment configuration is in deploy/.
• Podcast episodes (Xiaoyuzhou, Apple Podcasts): show notes by default, plus opt-in local ASR in the CLI: --media-preview-seconds N (≤30 s sample, always PARTIAL), --media-full-short (≤120 s / ≤2 MiB, verified full transcript) and --media-full-long (≤90 min / ≤64 MiB, resumable, verified chunks). Needs x-reader[media], ffmpeg and ffprobe.
• X posts: the quoted post and poll options are appended to the post text when they were read; extra fields structure_status, quote_status, poll_status, article_status and thread_marker record what was checked.
• A generic URL that redirects into X, YouTube, a podcast or another supported platform is read by that platform's reader.
• Generic pages are parsed with html5lib (new dependency), so sloppy markup such as unclosed navigation drawers is read the way browsers read it instead of as an empty page.
• Hostnames are routed, checked and connected to in one canonical (IDNA/UTS46) form, at entry and on every redirect, so look-alike spellings (x。com, ｘ.com, youtube．com) go to their platform's reader and policy, and DNS checks, TLS and Host always name the same host.
• Generic reads load the page directly over pinned hops first; the extra fields redirects_verified, body_verified, embedded_media and page_signals record whether that worked, whether the text shown is that read (locally, Jina's text is shown only when it and the direct read match both ways; otherwise the direct read is shown), how many players, frames and plugins the page embeds, and which walls (login, challenge, consent, missing page, paywall) the whole page announces.
• x-reader doctor checks ffprobe and local transcription (faster-whisper), and html5lib as a core library.
• Bilibili, Xiaohongshu and browser-fallback fixes from the Evidence Receipt work.

Changed

• Podcast URLs now return source_type=podcast and media_type=audio (before: read as generic pages, manual / text). Consumers that branch on source_type should handle podcast.
• URL routing is shared by the CLI and the hosted reader and matches hosts exactly: dropbox.com is no longer treated as X, a /feedback path is no longer a feed; fxtwitter / vxtwitter / fixupx links go to the X reader.
• Evidence receipts are stricter:
  • YouTube videos are never PASS, because pictures are not analyzed; the spoken_media component is PASS only when subtitle cues cover the whole video (no uncaptioned stretch over 30 s). Unavailable, private or restricted-video pages are FAIL and pages that cannot be tied to the video are UNKNOWN (extra field page_state): page metadata counts only for a single-video URL whose page is that video's (Jina's source URL and a video title, not a consent, sign-in or bare YouTube page). Subtitles count as covering the video only when yt-dlp's metadata names the same video (extra field transcript_video_id), and a transcript never upgrades the page rating.
  • X posts use FxTwitter's structured text when that read succeeds (oEmbed text can be cut short; thread markers in either count). X posts are PASS only when the text is tied to the requested post ID (the whole status path must match, then the oEmbed permalink or the FxTwitter post ID; a rendered page cannot prove which post it shows, so browser reads are post_identity_unverified) media, the quoted post and poll were checked and are absent or read, and there is no X Article or thread continuation (a quoted post with its own layers is incomplete; a layer whose field is present but null or empty, text that is not a string, or an author handle that is not an X handle, a post or reply id that is not an ASCII post number, or a poll without two to four worded options, in the post or the quoted post, is unread, not absent; post URLs need an ASCII post id with no leading zero, a replied-to handle may carry one @, and /i/status/<id> URLs take the author FxTwitter names); text from a page reader (Jina, a page without the post element) is UNKNOWN.
  • Generic pages are PASS only when x-reader loaded the page itself, found no player, frame or plugin (hidden ones included), and read 280+ characters of sentence-like prose without login, paywall, consent, teaser, missing-page or challenge markers; pages with embedded content and teasers are PARTIAL (beside embedded content, a thin text body is UNKNOWN, not PASS); thin text, text with cookie/consent wording and pages x-reader could not load are UNKNOWN. Wall titles ("Just a moment...", "Sign in", "401 Authorization Required") FAIL; a title is checked up to 1000 characters and a longer one leaves the page UNKNOWN (page_too_large_to_check); a line of any length that opens with a login, bot-check or missing-page notice fails a page with under 600 characters of prose (the notice included) and makes a longer one UNKNOWN (possible_access_wall, possibly_missing_page), as does such wording inside a sentence. Notices are matched case-insensitively with their common punctuation, dash and status-code variants, and as people read them: letters that look like ASCII ones (Unicode's confusables data, plus small capitals and squared, circled and parenthesized letters) read as those, accents and other combining marks are dropped, characters that draw nothing are removed (Hangul fillers are read both as nothing and as a blank); prompts to comment or to open one's own account are not walls; a block that is only "Read more" is a teaser; each line is also read run on from up to two short lines before it, or the last 80 characters of a longer one (into the line's first 160, from where each earlier line starts; consent wording too), as are blocks that only hidden text parts, so a notice split over lines, paragraphs or list items reads as one. Text is laid out as browsers lay it out: inline elements and comments run on with nothing added, whitespace collapses (preformatted text keeps its lines), blocks, <br> and table cells separate it, inline display and float count, and the children of flex and grid containers run on (ruby annotations count for walls apart from the base text they sit over); inline margins, padding and borders part words at the edge that sets them, and gaps between children, and walls are also looked for with a space at every element boundary (stylesheet spacing is not known); display:contents children join their grandparent's layout, ruby annotations read whole; walls are also looked for in the text as laid out, where hidden text (a display that wins as none included) and the breaks of hidden elements take no place, with each piece of text that may draw nothing (too small, too faint, or under a style x-reader cannot settle) kept and left out in every combination (past six pieces in one block, all and none, and the page is unverified), and a block that line breaks part (<br>, preformatted newlines) also read with each break as a space, so a notice wrapped onto two or more lines reads as one; that text is judged as it is, never stripped again as markup (<notice> text, <xmp> and text/plain markup stay words). Pages served as text/html are built with a standard HTML5 parser (html5lib, scripts on); text/plain is read as the text browsers show, markup included; application/xhtml+xml, which browsers parse as XML, is UNKNOWN (page_structure_unverified). Wall notices count wherever they are, hidden markup included, since author CSS can always show it; hidden markup (the hidden attribute, display:none, popovers, a closed <details> beyond its first summary, a closed <dialog>) only leaves the page text. Text style is followed down the tree from browser defaults (black on white, blue links, smaller <small>, <sub>, <sup>, <h5>, <h6>), presentational attributes (<font color/size/face>, bgcolor, <body text/link/vlink> and its margins) and inline styles, with CSS escapes decoded (an escaped ;, : or ! is never syntax, and a value spelled with escapes outside a string is not taken for a number, color or keyword) and !important honoured (revert goes back to the browser's value for the tag, which for most text is the parent's), colors read only in the grammar browsers keep (comma or space syntax, one background color, in its final layer), heights only with values valid for each height property; a display:contents element paints no background: text must keep a font size of at least 6 px, contrast of at least 1.3 against the opaque background behind it (inherited and alpha-blended; an image behind text is not readable, except the page's own <html>/<body> backdrop, judged like external CSS; nearly transparent text never reads) and line spacing between 0.75 and 10 times its size (em and % line heights are of the element's own size). Inline CSS is judged by an allowlist, on every declaration: properties that cannot hide text pass; risky ones pass only with plain values (opacity, zoom, visibility and overflow at their defaults, readable colors, no symbol fonts, small indents, spacings, vertical shifts and decoration lines, on-screen margins and padding, lines no wider than 2000 px, no set height or max height on a box its text could overrun (content sizing passes, and percentages or inherit unless an ancestor's height is set), no font size past 2000 px, no start edge or first line moved off the screen and no line wider than 2000 px as margins, borders, padding, relative offsets, indents and percentage widths add up down the tree (by the declarations that win, logical and shorthand ones included; a border counts only when the style that wins draws one (none, hidden and the initial style draw nothing), and a border shorthand naming two widths, styles or colors is dropped; an inherited margin, border, padding or offset is not followed), no relative offsets or baseline shifts that add up to moving text over the lines around it, no text pushed off the screen by preserved spaces with nothing shown before it on its line, lines that do not wrap (preformatted ones, letters and spaces alike; nowrap, <nobr>, <td nowrap>, max-content, words joined by no-break spaces) read as overflow a reader scrolls to, which passes only while the letters past 2000 px are at most a tenth of the text judged, percentages inside math functions taken of the parent's width and auto widths less their margins and padding, and no vertical writing mode (not measured) (white-space, text-wrap and width values x-reader does not recognize leave the text unverified, so a dropped value cannot stand in for the one before it), negative top margins that add up past 2000 px down the tree (hidden boxes and other branches pull nothing), and a negative top or bottom margin of more than half a line (margins by the values browsers keep; a math margin with a minus may come out negative) leave the page unverified; on lines that do not wrap, inline boxes take room for their margins, borders and padding, a box laid out whole with a set width or minimum width (math functions included) takes that width and wraps its own lines from where the line has got to, letter spacing and word spacing (after every word separator, no-break and preformatted spaces included) widen the line, spaces collapse across text nodes and still take room, and a hidden line break starts no line; math functions count only when they parse as browsers parse them (operands and operators in place, known functions with their argument counts, var() with a name, lengths where a length is due) and are worked out (percentages of the parent's width or a 1000 px box, a variable as an outside value or its fallback), and a margin or padding that may come to more than one value leaves the text unverified; opacity, visibility, overflow (on each axis) and stacking properties count by the values that win, grid columns of at least 1em and rows that grow with their content (flexible and content-sized tracks pass), no bidi override, generated content or dark color scheme, no absolute/fixed positioning, offsets, transforms, filters, clipping, masks, containment or animation; custom properties only from the stylesheet); anything else, including styles over 20 000 characters, keeps the page from passing (visibility_unverified). So does any rendered element anywhere on the page that could paint over text or carry it out of reach: absolute, fixed or sticky positioning, a relative box that moves or does not clip its overflow, transforms and other stacking contexts, outlines over 3 px, shadows over 16 px, explicit grid placement, negative margins past 1em, geometry beyond 5000 px, an inline box whose background or border reaches past its line, a collapsed table column, an open <dialog>, an <svg> that does not clip its drawing, or anything that shows text in another order than written (flex/grid order, reversed, wrap-reversed, dense or column flow, the old flexbox's direction and ordinals, right floats and right-aligned tables, flex, grid and multi-column layouts in right-to-left text, inherited too, every declaration of these holding a value browsers keep; table captions, header and footer groups written away from where they show, in a table or in the anonymous table browsers build around table parts written outside one (display:contents hands its children on, with the caption side they inherit, and nested more than 64 deep leaves the page unverified; a caption side counts by the value browsers keep; parts with no text shown take no place, while text a hidden part shows again does, and wording from attributes (button and field values, placeholders, option labels, alt text) is text shown); rows of a right-to-left table with more than one cell of text; an <svg> placing more than one piece of text by coordinates (several positions in one attribute count) or overriding text direction, and SVG or MathML showing more than 280 characters of prose, which x-reader does not return as page text (hidden SVG and MathML, and hidden text in them, place nothing, while text shown again inside a hidden one counts and a CSS-wide display, visibility or all that wins shows them again, for the text they place too; HTML inside <foreignObject> is laid out as HTML); right-to-left MathML and MathML prescripts; right-to-left text, embeddings, <bdi>, dir=auto (which, like <bdi>, takes the direction of its first strong letter) or unicode-bidi: plaintext with left-to-right words on both sides of right-to-left letters, or of an isolate or an inline box laid out whole (controls and replaced elements whatever their display), which show reordered, and the same in text/plain lines that open right-to-left) or makes text from CSS (string list markers, quotes, and by the value that wins among those browsers keep content, hyphenate-character and string text-overflow), a layout value spelled with escapes or not known (multi-keyword displays read as their one-word names; a value browsers drop does not count as the one that wins), a layout property set to inherit/unset/revert or all, or a script, style, noscript, title, datalist or <rp> block an inline display or all may show (in a shown <head> too); direction resolves inherit/unset/revert before dir, and logical floats follow it; inside a relative box that clips its overflow on both axes (a box that generates one, by the physical and logical overflow values that win), such an element unsettles only that box's text. <bdo>, bidi overrides (unicode-bidi: *override) and directional formatting characters of any kind (embeddings, overrides, isolates) anywhere the page shows text, its title, annotations and text/plain bodies included, leave the page unverified and <marquee> is not vouched for; <meter> / <progress> fallback and <rp> are not page text; button labels, text box values, option labels and image alt text, like menus and text boxes, count for walls but not as page text; <xmp> text is page text; hidden="until-found" hides whatever the display. A page with <template> markup, which the parser does not model, is UNKNOWN (page_structure_unverified); so is a meta refresh to another URL (client_redirect). Every frame, player and plugin outside <noscript> counts, however small or hidden, since ancestors and author CSS can enlarge or show it. Parsing has budgets for tokens, attributes per tag, elements, attribute copies, nesting, tree work (each token charged for the stacks its steps walk, plus child-list copies), parse errors (counted, not kept), text work and inline-style volume (1 MB, 50 000 declarations); html5lib's own invalid code points (controls, noncharacters in every plane, lone surrogates) are capped before parsing, pages over 2.5 MiB are not read directly (a read peaks near 35 times the page's size), the letters dir=auto looks through are capped at 2 000 000 per page (past it the page is unverified), and the page text read for walls (hidden included) is capped at 1 000 000 characters, with another 1 000 000 for the further readings of it (doubtful pieces, wrapped lines, annotations), charged before they are built; a page over them, or with too much text to judge, is UNKNOWN (page_too_large_to_check). When a page marks its main content, that region is the text judged and returned, while walls anywhere on the page still count (a cookie banner beside substantial main content does not).
  • RSS items count as full only with 280+ characters of visible prose, no "read more" or paywall text, a linked article page that passes on its own, at least 80 % of that page's text in the feed body and at least 70 % of the feed body on the page, repeats included and in any script, the page counted as the words it shows (feed_body_unverified otherwise), and no player, frame or audio/video enclosure in the item (embedded_media_unread otherwise); WeChat removal/verification notices FAIL (a notice title always; a notice line anywhere beside longer text is UNKNOWN; generic wording such as 参数错误 counts only as a whole line or title), login, bot-check, missing-page and paywall wording counts as on generic pages, image-carried posts are PARTIAL, and an article passes only when the browser read counted no player, frame or WeChat media component in it (counted ones, and reads that could not count them, are PARTIAL); the browser read takes the article as the browser lays it out (its own rendered text: inline runs join, hidden text is left out, the images it shows stay in place) and is judged as the words shown, never as markup; a read that cannot vouch for what readers see (an article that is not rendered, text not drawn, too small or faint (letters filled to match the background included), filtered, turned, clipped (by its own box too) or off the page, shown in another order, made by CSS, or covered by something painted over it, wide outlines, shadows and positioned pseudo-elements included) is UNKNOWN (visibility_unverified), as is one where the browser did not end on the requested article over HTTPS on its default port (the same mp.weixin.qq.com/s/<id>, or /s? with the same __biz, mid, idx and sn; article_identity_unverified); walls anywhere on the page count (its teaser links do not; a page text too long to scan in full is UNKNOWN), and a notice wrapped onto two lines of a paragraph counts.
  • Full-audio proofs allow 0.5 s between declared and decoded duration (was 1 %), and coverage_ratio is the measured processed/declared ratio.
• Podcast previews report the seconds actually decoded (preview_seconds) next to requested_preview_seconds.
• Telegram message URLs (t.me/<channel>/<id>, t.me/s/...) read that exact message instead of the channel's latest post.
• FxTwitter posts without a media field count as having no media (before: unknown); a media, poll, quote or article field that is present but null or empty counts as unread.
• RSS feeds served as text/plain, application/octet-stream or without a Content-Type are accepted when the body is a feed.
• Pinned connections try IPv4 addresses first and move on to the next checked address when a connection fails, within the request's time budget and at most three addresses (hosts with IPv6 addresses but no IPv6 route could not read generic pages).
• With --json, usage errors print {"ok": false, "error_type": "UsageError", ...} and exit 2; an unknown command exits 2.
• The all extra includes media (faster-whisper).
• Moat benchmark: every x-reader row is scored from its Evidence Receipt; new outcome UNVERIFIED; restricted controls that time out stay out of the denominator; benchmark runs cannot write to an Obsidian vault configured in .env.

Security

• yt-dlp only receives a rebuilt https://www.youtube.com/watch?v=<id> URL with its generic extractor disabled, so consent or redirect URLs cannot send it to other hosts.
• RSS feeds are downloaded with the pinned, per-hop validated fetcher and parsed from bytes, so feed reads can no longer be redirected to private addresses.
• External tools run with a deadline and an output budget enforced while they run; on exit, timeout or overflow their whole process group is killed.
• Hosted reads: one shared router plus a child-side platform allowlist that also covers redirects; generic text comes only from the service's own pinned fetch (no Jina Reader); the child skips .env, writes no inbox or Markdown, inherits no proxies, starts no external tools (no yt-dlp, ffmpeg or browser), and runs under address-space / CPU / file-size / open-file ceilings; CLI output is size-capped.
• Hosted server: at most 3 reads at once (a large page can take about 120 MiB), absolute 20 s request-read deadline watched by one thread, connection cap that survives thread-start failures, JSON-only POSTs, JSON for every error including unparseable request targets, a fixed set of public reason codes, control characters refused in URLs, no interpreter version in headers, stricter CSP.
• The manual preview workflow no longer opens a public tunnel; it runs the live canary against the server on the CI runner.
• yt-dlp calls skip playlists, cap audio downloads at 25 MB and bound subtitle / metadata files; short-audio decoding stops at the PCM budget and long-audio decoding has an idle watchdog.