You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Today — an invalid byte occupies no cell, produces no U+FFFD, does not advance the cursor, and leaves no trace anywhere on Screen. Measured against 0.8.0:
The shift is exactly one column per invalid byte: in caf<0xE9> latte |col22 the pipe sits at column 12 when the bad byte renders as one glyph, and termlens reports 11.
This is specific to genuinely invalid sequences. Valid multibyte text spanning read(2) boundaries is handled correctly — 40000 four-byte emoji across many read boundaries, zero replacement characters — so buffer-boundary handling is not part of this finding.
Why it is worth fixing — an encoding bug is one of the main reasons to put a TUI under a real PTY: a latin-1 log line, a filename from a non-UTF-8 filesystem, strerror() under a C locale, a buffer sliced mid-character. termlens renders all of them as if the bytes were never sent, so a snapshot of a corrupted screen is byte-identical to a snapshot of a clean one, and the test written to catch the corruption goes green. contains("caf latte") is true on a screen where the application actually emitted caf<0xE9> latte.
The damage does not stop at the missing glyph. Because the byte occupies no cell, everything to its right reports one column too far left, so find() and cell() addressing disagree with the terminal the user is testing for — silently, by one column per invalid byte, on exactly the rows where something has already gone wrong.
This is a gap rather than a stated boundary. The crate has a written position on invalid UTF-8 in two other places and neither is silence: Link::uri (screen.rs:210) replaces rather than refuses, "while still showing what arrived"; close_link (emu/seq.rs:536) refuses rather than replaces, and the label then reports None, which is observable. The grid, the crate's primary surface, does neither. The README's one adjacent sentence covers grapheme clusters, which these are not, and the unicode-torture fixture contains no invalid byte at all.
Fix — three options, cheapest first.
Write it down. A line under the README's Known limitations beside the grapheme-cluster sentence, saying that invalid bytes are dropped and that columns to their right shift left, plus a raw non-UTF-8 byte added to the unicode-torture fixture so the behaviour is pinned rather than incidental.
Be faithful. Feed U+FFFD to the parser for each invalid sequence, so the grid shows what a terminal shows and the columns line up. The only option that fixes the shift, and the only one that moves existing snapshots — taking (2) first does not close the door on it.
Done when — a test drives a child that emits a raw 0xE9 mid-word and can tell from the Screen alone that a byte was dropped, via a counter or via a replacement character in the cell. If the decision is to keep deleting them, the README's Known limitations says so and says the columns after the byte shift left, and unicode-torture carries a line with a raw non-UTF-8 byte so a future parser change cannot move this silently.
Today — an invalid byte occupies no cell, produces no U+FFFD, does not advance the cursor, and leaves no trace anywhere on
Screen. Measured against 0.8.0:The shift is exactly one column per invalid byte: in
caf<0xE9> latte |col22the pipe sits at column 12 when the bad byte renders as one glyph, and termlens reports 11.This is specific to genuinely invalid sequences. Valid multibyte text spanning
read(2)boundaries is handled correctly — 40000 four-byte emoji across many read boundaries, zero replacement characters — so buffer-boundary handling is not part of this finding.Why it is worth fixing — an encoding bug is one of the main reasons to put a TUI under a real PTY: a latin-1 log line, a filename from a non-UTF-8 filesystem,
strerror()under a C locale, a buffer sliced mid-character. termlens renders all of them as if the bytes were never sent, so a snapshot of a corrupted screen is byte-identical to a snapshot of a clean one, and the test written to catch the corruption goes green.contains("caf latte")istrueon a screen where the application actually emittedcaf<0xE9> latte.The damage does not stop at the missing glyph. Because the byte occupies no cell, everything to its right reports one column too far left, so
find()andcell()addressing disagree with the terminal the user is testing for — silently, by one column per invalid byte, on exactly the rows where something has already gone wrong.This is a gap rather than a stated boundary. The crate has a written position on invalid UTF-8 in two other places and neither is silence:
Link::uri(screen.rs:210) replaces rather than refuses, "while still showing what arrived";close_link(emu/seq.rs:536) refuses rather than replaces, and the label then reportsNone, which is observable. The grid, the crate's primary surface, does neither. The README's one adjacent sentence covers grapheme clusters, which these are not, and theunicode-torturefixture contains no invalid byte at all.Fix — three options, cheapest first.
unicode-torturefixture so the behaviour is pinned rather than incidental.invalid_bytes()accessor onScreenbesidebells()andrepaints(), so "nothing was silently dropped" becomes an assertion. Same move Observe BEL — often the only feedback a rejected input produces #102 made for BEL, and it changes no existing snapshot.Done when — a test drives a child that emits a raw
0xE9mid-word and can tell from theScreenalone that a byte was dropped, via a counter or via a replacement character in the cell. If the decision is to keep deleting them, the README's Known limitations says so and says the columns after the byte shift left, andunicode-torturecarries a line with a raw non-UTF-8 byte so a future parser change cannot move this silently.