You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Breaking (behaviour): PDF reading moves from Unpdf 0.48.0 to 0.52.1, and extracted PDF text changes in three ways. Hyperlinks now come out as Markdown links
([text](https://...); a 16-link model card went from plain text to links), so a consumer that chunks or embeds PDF text sees URLs in it. Table rows whose decimals were split
across lines (0. 899) read whole, and a form row whose label and options were interleaved across columns reads in order. A page set in a fixed-pitch font stays prose
(Unpdf 0.50.0 - 0.51.0 rendered it as a single code block). On 18 real PDFs (8 forms, 10 press releases) 16 extractions are
byte-identical and 2 differ only by the form-row and heading gains. Migration: strip [text](url) to text before chunking if URLs are unwanted.
Breaking: CSV/TSV parsing moves to CsvHelper 34.0.0 (a major release that replaced CsvParser/CsvConfiguration with a new CsvReader). The extracted tables are unchanged on 56 constructed cases and 86 real CSV/TSV files except for two malformed inputs:
a quoted field followed by more text keeps its escaped quote ("x""y"z reads x"yz, was xyz), and a U+FEFF at the very start of
already-decoded text is dropped. Migration: a consumer that references CsvHelper directly alongside FileFlux must move to 34.x
(the new reader API; CsvHelper.Configuration no longer exists).
Fixed
A paragraph that starts with a link keeps its link text: in Markdown, plain-text and HTML input (<p><a href="u">FileFlux</a> reads ...), [FileFlux](https://...) reads came out as - (https://...) reads - the link text lost and the paragraph turned into a list item; [1](...) became a numbered item. A ] followed by (, [ or : closes a link, a
reference or a definition, not a list marker. Present in every earlier release; PDF links (Unpdf 0.52.1) make it more likely to be met.
Sentences and headings that start like a list marker keep their text (plain-text, Markdown and PDF input): Dr. Smith arrived came out as - Smith arrived, v1.2 was released as - 2 was released, U.S. policy as - S. policy, No. 5 is next as - 5 is next, 3.14 is pi as 3. 14 is pi; in PDFs a numbered heading read 2. 1 Quantization and a decimal 93. 4.
A list marker is now a number of up to three digits, one letter or a short roman numeral closed by ) (1), (a), ii)), or a bracketed [1]/[a] followed by a space;
letters or digits followed by . are text (1. item was and stays Markdown). Behaviour change for a consumer that relied on a. item or Note) item becoming - item.
Present in every earlier release.
HTML: the word after an inline link or inline code is no longer fused to it: Read <a href="u">the docs</a> for more extracted as Read [the docs](u)for more and Run <code>x</code> first as Run xfirst. The reader wrote every text node followed by a space but left none after a link or inline code. Present in every earlier release.
Re-pinned third-party package Undoc 0.27.0 -> 0.27.1 (Word, Excel, PowerPoint reading): extracts the same text on 21 public Office files.
Re-pinned sibling package(s) LMSupply.Captioner 0.136.0 -> 0.137.0, LMSupply.Core 0.136.0 -> 0.137.0, LMSupply.Embedder 0.136.0 -> 0.137.0, LMSupply.Generator 0.136.0 -> 0.137.0, LMSupply.Ocr 0.136.0 -> 0.137.0, LMSupply.Transcriber 0.136.0 -> 0.137.0. LMSupply 0.137.0 changes the token estimate it falls back to when a local server reports no usage: Korean text now counts about 2.5x more tokens than before, English slightly fewer, digits more.
