You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ImageInfo.Properties["reference_name"] names the picture the way the extracted text does. The Markdown of a Word, PowerPoint or Excel file refers to a picture by the media part's file name () while ImageInfo.Id is the resource id (rId11), and nothing connected the two: a consumer that stores the image under its id could not find the file a reference names. Every reference in a Word document's text now matches the reference_name of exactly one image (PowerPoint and Excel set the same key from the same source); the text itself is unchanged.
fileflux extract / chunk / process show the reader's notes and a note for a file that gave no text. A non-empty file that produced 0 characters (a workbook or document in a layout the reader does not understand, a blank page) printed "Extracted successfully" with a table row of zero and nothing else; RawContent.Warnings was never shown. The notes now print as Warning: lines under the result and are written to info.json as a warnings array (only when there is something to say, so the info.json of a clean run is unchanged). The exit code is still success.
Library: ExtractionResult.Warnings carries them from ExtractToDirectoryAsync / ChunkToDirectoryAsync; ExtractionWarnings.Collect(rawContent, fileSize, text) builds the list for a caller that runs its own pipeline.
Any reader's "read but found nothing" is now marked, not only the PDF reader's.ExtractAsync (and the stateful processor) set Hints["extraction_failure_reason"] = "empty_extraction" and add a warning ending in [empty_extraction] when a non-empty source gives no text, no image and no table and the reader has given no reason of its own (the PDF reasons no_text_layer, blank_page, … are never overwritten). A consumer that stores the hints (FluxFeed passes them through) can tell an empty read from a short document. EmptyExtraction.Note(rawContent, sourceLength) is the same mark for a caller that runs a reader directly.
Breaking (behaviour): PDF reading moves from Unpdf 0.52.1 to 0.53.1, and more PDF tables now come out as Markdown tables where they were flowing text.
A table ruled across only (horizontal rules, no vertical rules; Word draws each such rule as one rectangle per cell), a table drawn with double rules, records ruled
apart inside a frame (a short key with a wrapped description) and a table whose title spans its columns are now read as tables; a header band
that does not line up in columns stays in its table, and a chart's axis labels, a diagram's boxes or an equation read as a grid of mostly empty cells are written as one
paragraph per row. A page turned by /Rotate is read as it is shown, and a page body set in two columns is no longer read as a table. On 188 table pages of a public
document-parsing benchmark, 44 pages came out with no table before and 39 now; across its five non-math sets the benchmark's own scorer goes from 53.4 % to 54.5 %
(71 checks gained, 5 lost; the 5 are tables with narrow gaps between columns in 3 files that now read as fewer columns), and 99 of its 845 outputs differ.
Styled text that contains a link is written as one emphasis around the whole run (**Read [the docs](url) now**) instead of separate bold pieces around the link; the words are the same (7 of 845 benchmark PDFs differ, none in any check).
The words of a chart or figure (axis labels, bar values) can come out in a different order, and a value that used to be dropped from a chart is kept; the header cell that spans several columns of a table can sit one column over.
Migration: none for code; a consumer that stored PDF chunks keeps the old text until the document is re-indexed.
Fixed
Breaking by observation: the rule-based metadata extractor labels a document by its language, not by one character.RuleBasedMetadataExtractor (the metadata fallback when no AI enricher answers) reported language: "ko" for any text containing a single Hangul syllable, "zh" for any text containing a CJK ideograph (checked before kana, so ordinary Japanese with kanji read as Chinese) and "en" for every other script, French and German included. It now uses the shared LanguageDetector (script ratios for Korean, Japanese and Chinese; a statistical identifier for the rest) and reports "unknown" when it cannot tell instead of guessing English. The stored language of English documents that quote one Korean name, of Japanese documents with kanji, and of non-English Latin-script documents changes: re-extract rather than diff.
A nested Markdown list, and anything else under a list item, is no longer dropped. The reader kept only the first paragraphs of each top-level bullet, so - Mike with - November and - Oscar under it was stored as Mike alone - every nested item, and a code block, quote or table inside an item, vanished without a trace. Each level is now written indented under its parent (• Mike / • November / • Oscar), and an item's second paragraph, code block and quote follow it indented; only the outermost list carries the LIST_START / LIST_END markers. Chunks you already stored keep the old text until the document is re-indexed.
An HTML line break inside Markdown text, or inside a heading, link, caption or table cell of an HTML page, no longer fuses the words around it.Golf<br>Hotel in Markdown was stored as GolfHotel and <td>Anna<br>Berta</td> in HTML as AnnaBerta; the same held for two paragraphs in one cell. A <br> in Markdown text is now a line break, in a Markdown table cell it stays <br>, and in the HTML text of a heading, link, caption or cell a line break or the edge of a block element counts as a space. Markup inside a word (un<b>believ</b>able) still adds none.
A text, Markdown or HTML file saved in CP949 / EUC-KR keeps its Korean. The readers decoded every file as UTF-8 with the replacement fallback (the intended fall-back to another code page never fired, because that decoder does not throw), so an older Korean .txt, .md or web page came out as a run of U+FFFD with no error; only the CSV reader had the CP949 fall-back. One decoder now serves the text, Markdown, HTML and CSV readers: a byte order mark (UTF-8, UTF-16, UTF-32), then strict UTF-8, then - for HTML - the charset the page declares (<meta charset>, ks_c_5601-1987, windows-1252, ...), then CP949. A file that is valid UTF-8 reads as before. Chunks you already stored from such a file hold the replacement characters until the document is re-indexed.
An HTML nested list or a table inside a cell is no longer written twice.<ul><li>Jakob<ul><li>Kaufmann</li></ul></li></ul> came out with Kaufmann once as an item of the outer list and again as the inner list; a nested ordered list was numbered 1. Norbert, 1. Otto, 2. Otto; the rows of a table inside a cell were repeated after the outer table. Each item and row is now written once, in its own list or table (the inner table's text stays with its cell). Chunks you already stored keep the repeated text until the document is re-indexed.
Three more Markdown alterations are gone. A list that starts at 3. is numbered 3, 4, ... (it was renumbered from 1); the term of a definition list (Term then : definition) is written with its definitions (only the definitions came out); and a literal pipe in a table cell (a \| b) stays escaped (it was written bare, which cut the cell in two and gave the row a column the header does not have).
Breaking by observation: the words on either side of a soft line break in a Word or PowerPoint file stay two words. A Shift+Enter line break inside a paragraph (an address, a form label, a verse, a signature block) was dropped by the Markdown conversion, so Street 1<break>City was stored as Street 1City and a search for either word missed; a Word page break fused the same way. The words are now separated by a space (a hard break is not written by default; Undoc's preserve_line_breaks render option exists for callers that want lines kept). The change applies to .docx, .pptx and the legacy .doc/.ppt readers (via Undoc 0.27.3, which also keeps a full stop inside bold text where it was written). Over 246 public and generated Office files only one output changed; stored text of files that contain a soft line break changes - re-index them.
An .xlsx written by the Open XML SDK is no longer read as an empty document. A workbook whose spreadsheet XML spells its elements with a prefix (x:worksheet, x:row, x:c; what the SDK writes for a part built from typed elements, so any .NET report generator produces it) gave no sheets, no rows and no error with Undoc 0.27.1; it now reads like its unprefixed twin. A Word document with another prefix (ns0:, x:) reads as well. A workbook that declares no sheet, or a sheet whose part is missing, still reads as empty.
A sentence that names an image marker is no longer replaced by an image. In plain text and Markdown, a line such as Write [image:diagram] where a picture belongs., a line mentioning [img_5], or one mentioning an <!-- IMAGE_START:IMG_1 --> comment was stored as just 
(or the image reference of the first marker), dropping every other word of the line. Only a line that is a placeholder by itself is converted now; a line that already holds an embedded image reference is kept as written. Chunks you already stored keep the old text until the document is re-indexed.
A # inside a line no longer cuts the paragraph or the table around it. Plain text, Markdown, HTML, PowerPoint and CSV content went through a blank-line rule that took # anywhere in a line for a heading: I write C# daily followed by and enjoy it. was split into two paragraphs, and a table row whose cell began with # (a CSV cell # not a heading) ended the table there, so the rest of its rows came out under a <!-- table: Complex table structure (missing separator) --> hint without a header. The rule applies to a heading at the start of a line only (and the same for a bare code fence). Words were never lost; chunk boundaries and table structure were wrong.
