You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MarkdownFences (FluxCurator.Core.Infrastructure.Refining): the fenced-code-block scan, public.FindRanges, Contains, LineMask and Shield / Shielded(text, step) - runs a text-level step of your own over a Markdown text with each fenced block (backtick or tilde, a longer fence closed only by an equal or longer one, an unclosed fence to the end, a fence in a list item) hidden behind one placeholder line and put back exactly as written; a step that loses the placeholder is skipped for that text instead of losing code. Only a closed fence is hidden (an unclosed one is left to the step), a bare line of four or more tildes is a separator and not a fence, a closing fence indented more than three columns deeper than its opening is code, the placeholder is not markup (a <!--.*?--> pattern leaves it) and a block always comes back on lines of its own even when a whitespace preset glued the placeholder to the prose around it; FenceShield maps offsets and line numbers of the shielded text back to the original.
FenceShield can hide code from a language model: ModelText, RestoreModelAnswer(answer), UnsafeToSend. A model does not see the private-use characters of the placeholder (measured with a local model: it answered every one as a backtick fence), so ModelText shows each block as a plain marker line instead; RestoreModelAnswer puts the blocks back and returns null unless the answer carries every marker exactly once, in order and on a line of its own and the code of the restored text is the code that was hidden (a marker the model dropped, repeated, moved, wrapped in backticks or set in a sentence, or a fence it added, is refused - keep the text you sent). RestoreModelAnswer refuses (null) every answer for a text that is UnsafeToSend. UnsafeToSend is true when the text holds fenced code and already holds the placeholder delimiters or a marker-looking line of its own, so the markers could not be told from the text: do not send such a text. The marker spelling is not a contract. Scan facts moved here from a consumer that kept a second copy of the scan.
Fixed
Breaking by observation: a paragraph chunk that carries an overlap no longer fuses the overlap with its own first line. With OverlapSize > 0 and TrimWhitespace (the default of FileFlux) the previous chunk was trimmed before its tail was taken as the overlap, so the line break between the overlap and the next paragraph was lost: ...about beta. + <!-- CODE_START --> came out as ...about beta.<!-- CODE_START -->, and a marker line followed by a code fence came out as <!-- CODE_START -->```xml (the fence line was no fence any more). On a 30-section test document the old text fused 55 lines in 29 chunks. The overlap now carries the whitespace that separated the previous chunk from the next text, unless the next text already starts with a line break of its own (an indented fence starts with spaces only, and keeps the break); OverlapFromPrevious still holds the overlap text without that break. The Sentence, Hierarchical and Token chunkers were measured and do not fuse. Chunk text of stored documents that were chunked with an overlap changes at the seam (re-chunk to get the unfused text).
TextRefiner.Refine (and Curator.RefineText / WithTextRefinement) leaves a Markdown fenced code block exactly as written, with every preset. The refiner tidies prose, and applied its steps inside code too: lines lost their indentation, a line that repeats (}}}) was collapsed to one, blank-line runs were cut, trailing spaces were stripped, RemovePatterns / ReplacePatterns rewrote code. Fenced blocks are now hidden from every step (MarkdownFences.Shielded). Observable: a custom pattern no longer matches inside a fenced block; text without a fence is refined as before.