Skip to content

A byte that is not UTF-8 is deleted from the grid, and every column after it shifts left #217

Description

@vyncint

Today — an invalid byte occupies no cell, produces no U+FFFD, does not advance the cursor, and leaves no trace anywhere on Screen. Measured against 0.8.0:

$ cargo test --test edge_badbytes -- --nocapture --test-threads=1

latin1_byte_in_a_word_vanishes    # child ran: printf 'caf\351 latte  |col22'
row_text(0)   = "caf latte  |col22"
text()        = "caf latte  |col22"
contains("caf latte")  = true
contains("caf\u{fffd} latte") = false
find("|col22")= Some((0, 11))
cursor        = (0, 17, true)
cells 0..8    = ["c", "a", "f", " ", "l", "a", "t", "t"]

every_flavour_of_invalid          # each printed as A<bad bytes>B
lone 0xff            -> row0="AB" cursor_col=2
lone continuation    -> row0="AB" cursor_col=2
surrogate ED A0 80   -> row0="AB" cursor_col=2
truncated 3-byte     -> row0="AB" cursor_col=2
overlong C0 AF       -> row0="AB" cursor_col=2
5-byte FE            -> row0="AB" cursor_col=2
(a terminal that emits U+FFFD would show A�B and cursor_col=3)

utf8_split_across_reads
emitted 40000 crabs, grid holds 40000, replacement chars = 0

The shift is exactly one column per invalid byte: in caf<0xE9> latte |col22 the pipe sits at column 12 when the bad byte renders as one glyph, and termlens reports 11.

This is specific to genuinely invalid sequences. Valid multibyte text spanning read(2) boundaries is handled correctly — 40000 four-byte emoji across many read boundaries, zero replacement characters — so buffer-boundary handling is not part of this finding.

Why it is worth fixing — an encoding bug is one of the main reasons to put a TUI under a real PTY: a latin-1 log line, a filename from a non-UTF-8 filesystem, strerror() under a C locale, a buffer sliced mid-character. termlens renders all of them as if the bytes were never sent, so a snapshot of a corrupted screen is byte-identical to a snapshot of a clean one, and the test written to catch the corruption goes green. contains("caf latte") is true on a screen where the application actually emitted caf<0xE9> latte.

The damage does not stop at the missing glyph. Because the byte occupies no cell, everything to its right reports one column too far left, so find() and cell() addressing disagree with the terminal the user is testing for — silently, by one column per invalid byte, on exactly the rows where something has already gone wrong.

This is a gap rather than a stated boundary. The crate has a written position on invalid UTF-8 in two other places and neither is silence: Link::uri (screen.rs:210) replaces rather than refuses, "while still showing what arrived"; close_link (emu/seq.rs:536) refuses rather than replaces, and the label then reports None, which is observable. The grid, the crate's primary surface, does neither. The README's one adjacent sentence covers grapheme clusters, which these are not, and the unicode-torture fixture contains no invalid byte at all.

Fix — three options, cheapest first.

  1. Write it down. A line under the README's Known limitations beside the grapheme-cluster sentence, saying that invalid bytes are dropped and that columns to their right shift left, plus a raw non-UTF-8 byte added to the unicode-torture fixture so the behaviour is pinned rather than incidental.
  2. Count it. An invalid_bytes() accessor on Screen beside bells() and repaints(), so "nothing was silently dropped" becomes an assertion. Same move Observe BEL — often the only feedback a rejected input produces #102 made for BEL, and it changes no existing snapshot.
  3. Be faithful. Feed U+FFFD to the parser for each invalid sequence, so the grid shows what a terminal shows and the columns line up. The only option that fixes the shift, and the only one that moves existing snapshots — taking (2) first does not close the door on it.

Done when — a test drives a child that emits a raw 0xE9 mid-word and can tell from the Screen alone that a byte was dropped, via a counter or via a replacement character in the cell. If the decision is to keep deleting them, the README's Known limitations says so and says the columns after the byte shift left, and unicode-torture carries a line with a raw non-UTF-8 byte so a future parser change cannot move this silently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions