Skip to content

v0.69.0

Latest

Choose a tag to compare

@github-actions github-actions released this 16 Nov 01:57

Added

  • A picture file (.png, .jpg, .jpeg, .webp, .bmp, .gif, .tif, .tiff) is read through the registered IImageToTextService. ImageDocumentReader (registered by AddFileFlux) turns what the service returns as text into the document, so a photographed receipt or a scanned page saved as an image becomes searchable like the PDF that holds the same scan. It claims only the picture extensions the service lists in SupportedImageFormats (compared with or without a leading dot, any case) and none when no service is registered, so a consumer without one keeps «no reader» for pictures — AddLMSupplyOcr() reads no TIFF, so with it .tif/.tiff stay unsupported. A read that found no text carries Hints["extraction_failure_reason"] = "no_text_in_image" and a warning; a service that answers with an error message is a failed read (Status = Failed, error code image_read_failed); a service that throws propagates. The supported extensions can be re-read at any time (IDocumentReaderFactory.GetAllReaders(); a DI-less DocumentReaderFactory takes RegisterReader(new ImageDocumentReader(service)) and GetSupportedExtensions() changes at once).

Fixed

  • Breaking by observation: .tmp is no longer a supported extension of the text reader. TextDocumentReader claimed .tmp next to .txt, .md and .json, so a consumer that builds its «can be memorized» set from the readers' SupportedExtensions took every scratch file or unfinished write for a document (a watched folder memorized the scratch files a program dropped every few seconds). IDocumentReaderFactory.IsExtensionSupported(".tmp") is false now, GetExtensionReaderMapping() has no .tmp entry and GetReader("x.tmp") returns no reader, so a .tmp file is an unsupported format like any other unknown extension. Migration: a consumer that really keeps text in .tmp files copies or renames them to .txt (or reads them itself) before handing them to FileFlux.

  • An HTML <pre> code block comes out as a fenced block with the code as it was written. The reader added one empty line before the closing fence to every block whose code ended with a line break (almost every highlighted block), and it opened every block with three backticks, so a code sample that held a fence line (a Markdown sample in a page) closed the block early. The line break that ends the code is now the one before the closing fence, and the fence is longer than any run of backticks that starts a line of the code. Measured over seven real pages with code (71 indented lines, blank runs, comment-looking text): every line was already kept; only the trailing empty line and the fence collision were wrong. A line break right after a bare <pre> start tag (<pre>\ncode) is dropped as browsers do instead of becoming an empty first line of the code; a line break inside <code> stays.

  • A line that starts with (1) and continues a line ending in a colon, comma or semicolon is no section title. The numbered-section step read a paragraph wrapped at a column (... trigger: / (1) The server is exposed to an untrusted network ...) as a level-3 heading, so the chunk heading path of everything after it named a clause of a sentence. A numbered line directly under a line that has not finished its sentence stays text; one after a blank line, a finished sentence or a heading is a title as before.

  • A code block that is larger than a chunk keeps its HTML comments, its indentation and its blank lines in both parts. A block cut across two chunks is not a closed fence in either (the head has an unclosed fence, the tail starts with a closing fence), so the removal of internal markers from the chunk text took every HTML comment out of it and collapsed its blank lines. A chunk that is part of a cut block - found from the chunk position in the text it was cut from, so a piece from the middle of a block that holds no fence line is covered too, and from the chunk text itself (an unclosed fence, an odd number of fence lines) for chunks without a position - now loses only FileFlux own structural markers (the closed set the readers write: <!-- CODE_START -->, <!-- HEADING_START:H2 -->, <!-- LIST_START:UNORDERED -->, <!-- TABLE_START -->, the image and image-section markers, the three table notes, the source-span marker); any other comment - also one with the shape of a marker, like <!-- FOOTER_END --> - and the blank lines stay, and the first line of the tail keeps its indentation. A chunk without a cut fence is cleaned as before (every comment outside code goes, closed code blocks are untouched). The same removal also repairs a structural marker that a chunker had glued to the fence line after it (<!-- CODE_START -->```xml, from the overlap seam fixed in FluxCurator).

  • A text, Markdown, HTML or CSV file that another program holds open for writing is read on Windows. The readers opened the file asking to deny writers, so a document its editor had open was refused with «being used by another process» exactly when it was most often asked about; they open it shared for read, write and delete now, as do the format probe and the Word, Excel and PowerPoint paths that read the file's bytes, the slide renderer and the metadata cache key. Word, PDF, Excel and PowerPoint documents were read through parsers that already shared, so only the text-based readers changed behaviour; nothing changes on macOS or Linux, which do not enforce share modes.

  • A text, Markdown, HTML or CSV file saved as UTF-16 without a byte order mark is read as text. It used to come out with a NUL between every character (NUL is valid UTF-8, so the strict UTF-8 step accepted it) or, for a Korean-only file, as garbage; no warning was given. The shared decoder now recognises UTF-16 (either byte order) by its NULs when there is no byte order mark; a file with a byte order mark, UTF-8 and CP949 are read as before.

  • A text, Markdown, HTML or CSV file saved as UTF-32 without a byte order mark is read as text too. Like UTF-16 before it, such a file came out with three NULs between every ASCII character (NUL is valid UTF-8, so the strict UTF-8 step accepted it). The shared decoder now recognises UTF-32 (either byte order) when every character has the shape of a code point (its highest byte 0, the next at most 0x10); UTF-8 and CP949 text, which has no NUL, is never taken for it, and a file with a byte order mark is read by the mark as before. An HTML page saved as UTF-16 whose <meta charset> says utf-8 was already read by its bytes (checked, now pinned by facts).

  • Breaking by observation: cancelling a read that goes through an image-to-text service now cancels, even when that service answers the cancelled call with an error message. ImageDocumentReader and the PDF page reading (ExtractOptions.PageReading) counted such an answer as a failed read (a PDF came back with every selected page marked read_failed, a picture as a failed extraction), and the image description of the Word, PowerPoint, Excel and PDF readers, RefiningOptions.ProcessImagesToText and the image processor of the output writer took it for a picture without text - a call nobody was waiting for any more returned as a finished one. All of them now throw OperationCanceledException when the caller's token is cancelled. Services that let the cancellation propagate (the LMSupply OCR and captioner services do) are unaffected; an adapter that turns every exception into an error message needs no change.

  • A line that starts with # inside a fenced code block (a shell comment, a preprocessor directive) is no longer read as a heading. The chunks after such a block carried a Location.HeadingPath (and HierarchyPath) made of the comment text, with the real heading above it lost. Every step that reads Markdown headings now skips fenced blocks - backtick and tilde fences, a fence closed only by an equal or longer one, an unclosed fence running to the end of the text, a fence inside a list item: the section/heading-path calculation of both processors, the numbered-section conversion, the Markdown normalizer's heading rules (it no longer removes a lone #, demotes a # (note) or re-levels a ##### line inside a block) and the converter's blank-line rule. Chunk boundaries were not affected (measured with the paragraph strategy).

  • The default refining no longer deletes the closing fence of a code block. FluxDocumentProcessor removed any short line without lowercase letters that occurs three or more times as a «running header» - so a document with three or more fenced code blocks lost every closing ``` (and the <!-- CODE_END --> marker), which left the first block unclosed to the end of the text. The refining steps now leave fenced blocks alone (a line of code that looks like a page number, a separator or a repeated all-caps word is code), a line with no letter and a structural marker comment are never taken for a header, and ConvertBlocksToMarkdown no longer adds a blank line before a block's closing fence.

  • A fenced code block comes out of both processors exactly as it went in. The cleaning, normalizing and deduplicating steps treated a block as prose: every line lost its indentation (a Python or YAML sample changed meaning), a line that repeats (} } }, END END) was collapsed to one, runs of blank lines inside the block were cut to one (the legacy parser also cut the block at its blank lines and trimmed each piece), trailing spaces and tabs were stripped. The refining steps of FluxDocumentProcessor and the stateful processor, the Markdown normalizer, the parser's section rebuild and the chunk content cleanup now hide each fenced block from the step (one placeholder line) and put it back as written; a step that loses the placeholder is skipped for that text instead of losing code. The Markdown reader still writes every fence with backticks (a ~~~ fence comes out as a backtick fence; a backtick in a tilde fence's info string is dropped, which would have turned the opening line into text). The same hiding now covers the converter's whitespace step (blank lines inside a block), the chunk content cleanup (an HTML comment that is part of a sample is no longer deleted from the chunk text) and the table-of-contents demotion; the converter finds a block by its fence (a md ```` block that shows a fence is no longer cut at its inner fence lines); a closing fence indented more than three columns deeper than its opening is code, not the end of the block; the Markdown normalizer's action line numbers and the legacy parser's section offsets refer to the text that was passed in, not to a text with the blocks folded away. Only a closed fence is hidden from a text step, and a bare line of four or more tildes is a separator.

  • The LLM refinement (LlmRefiner) no longer sends fenced code to the model, and no longer rewrites it. Each pass told the model to join lines that end without punctuation, fix look-alike characters, drop what repeats and re-level # lines, and a model followed that on code: with a local model the OCR pass rewrote the identifier O0 to 00 in a code block. Each closed block is now hidden from every pass (one marker line each) and put back exactly as written (a fence that is never closed in the text sent is not a block); a pass is no longer started because code looks like a scan error (O0, 100, rn), a broken line (cd build above make all) or a heading (# comment), and code does not count toward the length a pass needs. An answer that loses, repeats, moves or wraps a marker is not applied, and a text that already holds a marker line of its own is not sent: the pass is reported as Failed with the new reason LlmRefinementPass.CodeBlockLost (code_block_lost) and the text is kept. A text with no fenced block gets the prompts it always got.

  • A {...} in the text of a Markdown file is kept. The Markdown reader's Markdig pipeline had the generic-attributes extension on, which reads a {...} after a word or in a heading as an attribute list and takes it out of the text: a heading such as ### Options `{...}` or ### GET /messages/{id} came out as «Options » / «GET /messages/», so the chunks' Location.HeadingPathandSectionnamed a heading that does not exist, and a paragraph's{placeholder}vanished. The reader uses no attributes, so the extension is off; **Breaking by observation:** a{#id}or{.class}after a heading, or a{width=50%}/{target=_blank}` after an image or a link, which used to be removed, now stays in the text as the document wrote it.

  • A numbered line that holds two sentences is no longer promoted to a heading. With BuildSections a line starting with ③, (1) or 1. counts as a section title when it is short and does not end in a full stop; a paragraph hard-wrapped at its points («… ② …⏎③ …한다. 추천의 근거는 …») put a clause of that paragraph into the chunks' heading path. A title is one phrase now: a sentence end followed by more words (한다. 추천의, done. Then) rules the line out; a short abbreviation before the stop (vs., e.g., Fig., Dr.) or a stop followed by a digit is no break, so Closed vs. open model refusal comparison, Fig. 3 Results and Version 2.0 notes are still titles.

  • A numbered item of a list in a Markdown file is no longer promoted to a heading. With BuildSections the refining turns a numbered line that looks like a title into a heading; an item written as «1. open the page» with a detail line under it, which is not a run of consecutive numbered lines, was taken for a section title, so a chunk's Location.HeadingPath carried the whole item as its last element (37 of 49 wrong path elements over 204 Markdown files). Lines inside the Markdown reader's list markers are items now and are left alone; numbered lines of text without structure are promoted as before.

  • A fenced code block under a list item, a definition or a quote keeps its blank lines. The Markdown reader writes such a block indented (or prefixed with > ) and dropped every empty line while doing so, so a sample in a numbered step lost the blank lines between its statements (found by reading 204 Markdown files with 2088 code blocks through the default path: the nine files with a changed block were all blocks under a list item). Blank lines between blocks are still dropped.

  • The default refining no longer deletes every line that starts with 주. The company-name header rule was written as ^(주)|(주식회사)\s*\S+$ - a regular-expression alternation whose first branch matched any line beginning with 주 (주요 기능, 주의: …), so those lines vanished from FluxDocumentProcessor output. It now matches the header forms (주) <name> and 주식회사 <name> only.

  • A chunk that starts at a heading has only that heading's section in its heading path. A section ended on the character where the next heading of its level starts and was still counted as containing it, so a chunk starting at ## C got A > B > C instead of A > C.

  • The Markdown reader writes a code block that shows a fence with a longer fence. A ```markdown sample inside a ```` block ended the block at its first inner fence line; the fence of the written block is now longer than any run of backticks in its code.

Changed

  • Breaking: the detected language of short Latin-script text is unknown instead of a guess, and its confidence now means something. LanguageDetector.Detect (document Language / LanguageConfidence in the metadata, a chunk's language, the LMSupply OCR service's DetectedLanguage) labelled a few words with whatever NTextCat's best candidate was — Invoice and Importo totale: 42,00 EUR came out pt, Factuur fr — and reported a confidence of 0.50-0.51 for every Latin-script text, right or wrong, so IsLanguage(text, code) with its default 0.6 could never be true for one. It now answers ("unknown", 0.0) when the best candidate is not measurably better than the second (a relative distance gap under 0.002; every wrong label measured sat at 0.0012 or less, every sentence of 42+ characters at 0.0043 or more), and the confidence spends the observed range (about 0.6 for a short sentence, 0.8-0.97 for a paragraph). Sentences keep their label; Hangul/Kana/Han detection is unchanged. LanguageDetector.DetectMultiple is calibrated the same way: its confidences were about 0.33 each for three candidates whatever the text (inverse NTextCat distances, all near 4000, normalised); they are now a distribution over the candidates that sums to 1 - near 1 for a paragraph, spread evenly for a few words - and a Korean, Japanese or Chinese text is answered by that one language, as Detect does, instead of by Latin-profile guesses. Migration: treat unknown as «not enough text to tell» (it was already a possible value); a threshold you set on the old 0.5-scale confidence should be re-read — a sentence-length text now clears 0.6.
  • Breaking: a language hint the speech model does not decode now throws. LMSupplyTranscriberService and LMSupplyTranscriberOptions.Language pass the hint to LMSupply 0.140.0, which checks it: an ISO 639-1 code, an English name ("korean") or a code with a region ("ko-KR") becomes the code; null, empty or white space mean identify the language; any other value ("xx", "kor"), or a code the chosen model does not transcribe, throws an ArgumentException listing the accepted codes. Before, such a value was dropped, the model chose a language of its own and AudioTranscript.Language echoed the wrong code. Migration: pass null for automatic detection.

Dependencies

  • Re-pinned sibling package(s) FluxImprover 0.23.0 -> 0.24.0, FluxImprover.LMSupply 0.23.0 -> 0.24.0.
  • Re-pinned sibling package(s) LMSupply.Captioner 0.139.0 -> 0.140.0, LMSupply.Core 0.139.0 -> 0.140.0, LMSupply.Embedder 0.139.0 -> 0.140.0, LMSupply.Generator 0.139.0 -> 0.140.0, LMSupply.Ocr 0.139.0 -> 0.140.0, LMSupply.Transcriber 0.139.0 -> 0.140.0.
  • Re-pinned third-party packages: xunit.v3 4.0.2 -> 4.0.3.
  • Re-pinned sibling package(s) LMSupply.Captioner 0.140.0 -> 0.141.0, LMSupply.Core 0.140.0 -> 0.141.0, LMSupply.Embedder 0.140.0 -> 0.141.0, LMSupply.Generator 0.140.0 -> 0.141.0, LMSupply.Ocr 0.140.0 -> 0.141.0, LMSupply.Transcriber 0.140.0 -> 0.141.0.
  • Re-pinned sibling package(s) FluxImprover 0.24.0 -> 0.24.1, FluxImprover.LMSupply 0.24.0 -> 0.24.1.
  • Re-pinned sibling package(s) LMSupply.Captioner 0.141.0 -> 0.142.0, LMSupply.Core 0.141.0 -> 0.142.0, LMSupply.Embedder 0.141.0 -> 0.142.0, LMSupply.Generator 0.141.0 -> 0.142.0, LMSupply.Ocr 0.141.0 -> 0.142.0, LMSupply.Transcriber 0.141.0 -> 0.142.0.
  • Re-pinned sibling package(s) FluxCurator 0.19.0 -> 0.20.0, LMSupply.Captioner 0.142.0 -> 0.143.0, LMSupply.Core 0.142.0 -> 0.143.0, LMSupply.Embedder 0.142.0 -> 0.143.0, LMSupply.Generator 0.142.0 -> 0.143.0, LMSupply.Ocr 0.142.0 -> 0.143.0, LMSupply.Transcriber 0.142.0 -> 0.143.0.