Vim terminal does not properly handle emojis if output was delayed

[ychin]

Steps to reproduce

1. Compile the following C program:

   #include <stdio.h>
   #include <unistd.h>
   int main()
   {
       setbuf(stdout, NULL);
       printf("abc%c%c", 0xf0, 0x9f); // first half of 😅
       sleep(1);
       fflush(stdout);
       printf("%c%c", 0x98, 0x85); // second half
       printf("\n");
       return 0;
   }

   clang -o prog test.c

2. Run Vim, open terminal by typing :terminal

3. Run the compiled prog program.

4. Observe that Vim shows a broken emoji because the time delay of printing the emoji "😅" caused the terminal emulator to improperly split it into two half:

    ➜ ychin ~/testbed/vim/broken_terminal_output% ./prog
    abc��
    �%                                                                                                                                                                                                       
   

Expected behaviour

It should look like this:

➜ ychin ~/testbed/vim/broken_terminal_output% ./prog
abc😅

Note that I tried this in Apple Terminal / Ghostty / iTerm2 / Kitty, and also Neovim's own terminal and they all work correctly.

I did notice that an old version of Neovim that I had lying around (0.9.5) would render this incorrectly. So they probably fixed it after then.

Version of Vim

9.1.1061

Environment

OS: macOS 15.2
Terminal: Tested on different ones, doesn't matter (since the bug happens in Vim's own libvterm terminal emulator)

Logs and stack traces

[zeertzjq]

Related neovim/neovim#16245

Note that the libvterm patch proposed there causes some libvterm tests to fail, while neovim/neovim#27922 doesn't make changes to libvterm.

[ychin]

I haven't looked closely enough yet, but just filing the bug first.

I noticed this issue because it's actually causing random syntax tests failures in MacVim CI ( I just turned them off for now). Vim tests heavily rely on screendump tests, and they are unreliable because of this bug. Even in local testing I could reliably get syntax tests to fail maybe every 5 tests or so, and the reason is that it's using screendumps to validate unicode printouts, and occasionally Vim prints it out just a little slowly and this libvterm bug causes it to print the wrong thing and we fail the screen dump test.

You can usually hack it by injecting a sleep before every VerifyScreenDump call but it's not a long term solution.

[chrisbra]

Perhaps we need to sync libvterm?

[zeertzjq]

No, the bug still exists in libvterm.

[ychin]

From skimming it seems like we need to port the fix you did to Neovim then (in the linked issue)? Seems like Neovim is pre-processing the bytes and withholding them from libvterm to prevent this bug?

[zeertzjq]

I've tried to port it before but it's a bit complicated. The data goes from channel_get -> channel_get_all -> write_to_term -> term_write_job_output -> vterm_input_write, and I can't decide where the data should be withheld.

[ychin]

Is this actually a bug in vterm in a desired behavior sense? If so I wonder if we should fix it in libvterm.

[zzzyxwvut]

There has been some effort to keep non-Latin-1 characters
filtered out when comparison is done for syntax tests, while
not touching non-ASCII characters and, incidentally, one
combining character (more below). Manually sampling every
week a portion of failure reports tentatively suggests that
such filtering is successful (until recently when a new
batch of C syntax tests has been added), i.e. the failures
are either elsewhere, not in syntax tests, or syntax dumps
are out of date for a particular syntax file.

See #14767 and #16560.

@ychin, do you ever see these particular tests fail on your
Mac machine:

input/java_contextual_keywords.java
input/java_numbers.java
input/markdown_conceal.markdown

?

The listed files contain multibyte characters for which NO
filtering is done: the Java files contain § (0xC2 0xA7)
(U+00A7 SECTION SIGN) and the Markdown file contains some
ASCII characters, where each is followed by (0xCC 0xB6)
(U+0336 COMBINING LONG STROKE OVERLAY).

Again, I haven't seen a single syntax failure report related
to this problem since #14767 and until the current C streak;
but then I haven't looked at every failure report.

I confirm, the emoji example breaks for U+00A7 and U+0053
combined with U+0336 as well when used with :terminal
(v9.1.1039) on a Linux box (GCC 14.2.0).

Note that on a Linux box, the tests also pass with filtering
disabled (after the removal of dumps/*.vim) and artificial
load imposed (via yes(1) instances):

cd runtime/syntax
rm testdir/dumps/*.vim

yes >/dev/null &
yes >/dev/null &
...

make clean c java markdown test
jobs
kill %1 %2 ...

git restore testdir/dumps/\*

[ychin]

There has been some effort to keep non-Latin-1 characters filtered out when comparison is done for syntax tests, while not touching non-ASCII characters and, incidentally, one combining character (more below).

Personally I don't love that. Is this just to work around misc test failures? Vim works natively with Unicode, and it's a core functionality. Core functionality should be validated and tested, not dodged. If things don't work, then it's a bug, and we can make temporary adjustments to keep tests stable but they should be committed to be of a temporary status, not a permanent one. Just because tests are passing doesn't mean we are bug-free if we aren't validating them.

But I would concede that this is a bug in the terminal code. Syntax tests shouldn't really fail if another part of Vim is being flaky (but that does mean we should find a way to fix the libvterm integration).

do you ever see these particular tests fail on your Mac machine:

The one that I see failing on my Mac and in MacVim CI is c_character_constant.c. I don't know if there are other ones as well because of the CI spam (more on that later). In particular, this line seems to be a problem with the (🍌 text:

// implementation-defined (🍌 maps to two 16-bit characters)

Again, I haven't seen a single syntax failure report related to this problem since #14767 and until the current C streak; but then I haven't looked at every failure report.

I don't think Vim CI runs syntax tests in the macOS runners though. MacVim by definition does do that so we were accidentally validating this code path for Vim. The interesting thing is I can get these syntax tests to fail on my personal Mac so I can't just blame it on typical GitHub Actions CI underpowered VMs. There may be some OS-specific implementation details with Vim and libvterm that I'm not aware of.

---

About the CI spam that I mentioned above, one issue with the syntax tests is that they are the only Vim tests that don't redirect output to /dev/null. Regular script tests have a default REDIR_TEST_TO_NULL redirect that makes sure the tests don't spam the console with live Vim previews. Is there a reason why syntax tests don't have that? Right now running syntax tests would mean the developer's terminal is overtaken by Vim, and in CI it would output a lot of control character spam making it hard to go through the test logs. If it was just an oversight, should we just add that? Local testing seems to show that it still works fine with it.