ui_attach: ill-formed redraw event with Neovide and noice.nvim

[ryo33]

Describe the bug

Related to folke/noice.nvim#17, neovide/neovide#1751, and maybe #21265
Neovide version (patched for debugging): ryo33/neovide@db6ba6d
Noice.nvim version: folke/noice.nvim@d8a1f30
nvim-notify version: rcarriga/nvim-notify@bdd647f
nui.nvim version: MunifTanjim/nui.nvim@d147222

Abstract

Using Neovide GUI with noice.nvim suddenly crashes occasionally. I've found that, in most cases, it's caused by Neovim emitting an Ill-formed redraw event and subsequent rpc messages are broken (see Appendix 2.) Some people report the sudden crash in the same case, and one reported the almost same backtrace as mine.

The event from neovide_backtrace.log with printing in ron format at: (File: src/bridge/handler.rs; Line: 77, Column: 25)

[
    "msg_show",
    [
        [],
        [
            true,
        ],
        [
            "msg_showcmd",
            [
                [],
            ],
        ],
    ],
]

The situation

• Neovide call ui_attach with ext_linegrid and rgb.
• noice.nvim call ui_attach with ext_messages, ext_cmdline, and ext_popupmenu.
• noice.nvim sometimes trigger redraw while handling events.
  • Neovide not playing nice with noice neovide/neovide#1751 (comment)
  • Neovide not playing nice with noice neovide/neovide#1751 (comment)

Appendix 1.

Starting Neovim with nvim -u NONE and running the following lua code without -- in front of print("hello") crashes neovim silently. This might be related to this issue. If someone tells me that this is an unexpected behavior, I'll open another issue.

local first = true
local ns = vim.api.nvim_create_namespace "test"
vim.ui_attach(ns, {
  ext_messages = true,
}, function(event, kind, content, replace_last)
  if first then
    -- print("hello")
    first = false
  end
end)

Edited: I noticed there is #21265

Appendix 2.

To ensure that the MessagePack and RPC implementation Neovide uses isn't doing wrong, I write the following Python script:

import sys
import pprint
from pynvim import attach
path = sys.argv[1]
nvim = attach("socket", path = path)
nvim.ui_attach(1000, 1000, True, ext_linegrid = True)
pp = pprint.PrettyPrinter(indent=2, width=40)
def on_notify(name, args):
    if name != "redraw":
        return
    str = pp.pformat(args)
    if "'msg_show'" not in str:
        return
    print(str)

def on_request(name, args):
    pass

nvim.run_loop(on_request, on_notify)

It prints:

Click me

[ ['cmdline_hide', [1]],
  [ 'msg_show',
    [ 'emsg',
      [[4, 'E32: No file name']],
      False]],
  ['mode_change', ['normal', 0]],
  ['mouse_on', []],
  ['flush', []]]
[ [ 'win_viewport',
    [ 2,
      <Window(handle=1000)>,
      0,
      10,
      1,
      0,
      6]],
  [ 'grid_line',
    [1, 2, 0, [[' ', 0]]],
    [1, 3, 0, [[' ', 0]]],
    [1, 5, 0, [['a', 0]]],
    [1, 6, 0, [['~', 7], [' ', 7, 71]]],
    [1, 7, 0, [['~', 7], [' ', 7, 71]]],
    [1, 8, 0, [['~', 7], [' ', 7, 71]]],
    [1, 26, 54, [['2', 9]]]],
  [ 'hl_attr_define',
    [ 383,
      { 'background': 16711935,
        'foreground': 11140968},
      { 'background': 13,
        'foreground': 0},
      []],
    [ 384,
      { 'background': 16711935,
        'foreground': 5203794},
      { 'background': 13,
        'foreground': 0},
      []]],
  [ 'grid_line',
    [ 1,
      0,
      20,
      [ ['╭', 65],
        ['─', 65, 50],
        ['╮']]],
    [1, 1, 20, [['│', 65]]],
    [1, 1, 71, [['│', 65]]],
    [1, 2, 20, [['│', 65]]],
    [1, 2, 71, [['│', 65]]],
    [1, 3, 20, [['│', 65]]],
    [1, 3, 71, [['│', 65]]],
    [ 1,
      4,
      20,
      [ ['╰', 65],
        ['─', 65, 50],
        ['╯']]],
    [ 1,
      1,
      21,
      [ [' ', 41],
        ['\uf057', 343],
        [' '],
        ['E'],
        ['r', 343, 2],
        ['o'],
        ['r'],
        [' ', 343, 5],
        [' ', 41, 31],
        ['2', 343, 2],
        [':'],
        ['4'],
        ['3'],
        [' ', 41]]],
    [1, 2, 21, [['━', 342, 50]]],
    [ 1,
      3,
      21,
      [ ['E', 382],
        ['3'],
        ['2'],
        [':'],
        [' '],
        ['N'],
        ['o'],
        [' '],
        ['f'],
        ['i'],
        ['l'],
        ['e'],
        [' '],
        ['n'],
        ['a'],
        ['m'],
        ['e'],
        [' ', 41, 33]]]],
  [ 'msg_show',
    ['', [[0, '3 fewer lines']], True]],
  ['msg_showmode', [[]]],
  [ 'win_viewport',
    [ 2,
      <Window(handle=1000)>,
      0,
      7,
      1,
      0,
      6]],
  [ 'grid_line',
    [ 1,
      5,
      20,
      [ ['╭', 67],
        ['─', 67, 50],
        ['╮']]],
    [ 1,
      6,
      20,
      [ ['│', 67],
        [' ', 41],
        ['\uf05a', 383],
        [' '],
        ['M'],
        ['e'],
        ['s', 383, 2],
        ['a'],
        ['g'],
        ['e'],
        ['s'],
        [' ', 41, 33],
        ['2', 383, 2],
        [':'],
        ['4'],
        ['3'],
        [' ', 41],
        ['│', 67]]],
    [ 1,
      7,
      20,
      [ ['│', 67],
        ['━', 384, 50],
        ['│', 67]]],
    [ 1,
      8,
      20,
      [ ['│', 67],
        ['3', 41],
        [' '],
        ['f'],
        ['e'],
        ['w'],
        ['e'],
        ['r'],
        [' '],
        ['l'],
        ['i'],
        ['n'],
        ['e'],
        ['s'],
        [' ', 41, 37],
        ['│', 67]]],
    [ 1,
      9,
      20,
      [ ['╰', 67],
        ['─', 67, 50],
        ['╯']]]],
  ['grid_cursor_goto', [1, 26, 55]],
  ['flush', []]]
[ [ 'msg_show',
    [ [],
      [True],
      ['msg_showcmd', [[]]]]],
  ['grid_cursor_goto', [1, 1, 0]],
  ['mode_change', ['normal', 0]],
  ['flush', []],
  [ 2,
    'redraw',
    [ [ 'grid_line',
        [ 1,
          5,
          20,
          [ ['╭', 67],
            ['─', 67, 50],
            ['╮']]],
        [1, 6, 20, [['│', 67]]],
        [1, 6, 71, [['│', 67]]],
        [1, 7, 20, [['│', 67]]],
        [1, 7, 71, [['│', 67]]],
        [1, 8, 20, [['│', 67]]],
        [1, 8, 71, [['│', 67]]],
        [ 1,
          9,
          20,
          [ ['╰', 67],
            ['─', 67, 50],
            ['╯']]],
        [ 1,
          6,
          21,
          [ [' ', 41],
            ['\uf05a', 383],
            [' '],
            ['M'],
            ['e'],
            ['s', 383, 2],
            ['a'],
            ['g'],
            ['e'],
            ['s'],
            [' ', 41, 33],
            ['2', 383, 2],
            [':'],
            ['4'],
            ['3'],
            [' ', 41]]],
        [1, 7, 21, [['━', 384, 50]]],
        [ 1,
          8,
          21,
          [ ['3', 41],
            [' '],
            ['f'],
            ['e'],
            ['w'],
            ['e'],
            ['r'],
            [' '],
            ['l'],
            ['i'],
            ['n'],
            ['e'],
            ['s'],
            [' ', 41, 37]]],
        [ 1,
          0,
          20,
          [ ['╭', 65],
            ['─', 65, 50],
            ['╮']]],
        [1, 1, 20, [['│', 65]]],
        [1, 1, 71, [['│', 65]]],
        [1, 2, 20, [['│', 65]]],
        [1, 2, 71, [['│', 65]]],
        [1, 3, 20, [['│', 65]]],
        [1, 3, 71, [['│', 65]]],
        [ 1,
          4,
          20,
          [ ['╰', 65],
            ['─', 65, 50],
            ['╯']]],
        [ 1,
          1,
          21,
          [ [' ', 41],
            ['\uf057', 343],
            [' '],
            ['E'],
            ['r', 343, 2],
            ['o'],
            ['r'],
            [' ', 343, 5],
            [' ', 41, 31],
            ['2', 343, 2],
            [':'],
            ['4'],
            ['3'],
            [' ', 41]]],
        [1, 2, 21, [['━', 342, 50]]],
        [ 1,
          3,
          21,
          [ ['E', 382],
            ['3'],
            ['2'],
            [':'],
            [' '],
            ['N'],
            ['o'],
            [' '],
            ['f'],
            ['i'],
            ['l'],
            ['e'],
            [' '],
            ['n'],
            ['a'],
            ['m'],
            ['e'],
            [' ', 41, 33]]]],
      ['msg_showmode', [[]]],
      [ 'win_viewport',
        [ 2,
          <Window(handle=1000)>,
          0,
          7,
          1,
          0,
          6]],
      ['flush', []]]]]

Steps to reproduce

neovide -- -u NONE
:packadd nvim-notify
:packadd nui.nvim
:packadd noice.nvim
:lua require("noice").setup {}
" 1. input several lines
" 2. type 'd2k' in normal mode
" repeat 1 and 2 a few times

Expected behavior

Neovim emits a well-formed redraw event.

Neovim version (nvim -v)

NVIM v0.9.0-dev-1004+gbfe6b4944-dirty

Vim (not Nvim) behaves the same?

no

Operating system/version

Ubuntu 22.04.1 LTS

Terminal name/version

GNOME Terminal Version 3.44.0 for GNOME 42

$TERM environment variable

xterm-256color

Installation

homebrew (linuxbrew)

[ryo33]

It might be also weird that Neovide receives that “msg_show” although Neovide does not subscribe “ext_messages”.

[polachok]

I did some research on this and found that in ui_call_event msg_show args are valid before calling lua callbacks and become corrupt after. Then UI_CALL is called with bad args and we see ill-formed events later.

Bad args vs good args:

ui_call_event end: msg_show 
// args after lua callbacks
 arr[0]: 5
   empty array
 arr[1]: 5
   array @ 0x16b4fcac0
   arr[0]: 1775635200
 arr[2]: 1
 array @ 0x16b4fcbe0

// original args
 arr[0]: 4
  str: confirm
 arr[1]: 5
   array @ 0x16b4fcac0
   arr[0]: 5
     array @ 0x16b4fc9a0
     arr[0]: 2
     arr[1]: 4
      str:
Swap file "~/.local/state/nvim/swap//%Users%censored%/sample.lua.swp" already exists!
[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort:
 arr[2]: 1

Here's a diff I used:

diff --git a/src/nvim/ui.c b/src/nvim/ui.c
index 45959b7b6..77501ad46 100644
--- a/src/nvim/ui.c
+++ b/src/nvim/ui.c
@@ -656,10 +656,39 @@ void ui_grid_resize(handle_T grid_handle, int width, int height, Error *err)
   }
 }
 
+static void dump_array(Array args, int depth) {
+  char buf[128] = {};
+  for (int i = 0; i < depth; i++) {
+    strcat(buf, "  ");
+  }
+  if (args.size == 0) {
+    printf("%s empty array\n", buf);
+    return;
+  }
+  for(size_t i = 0; i < args.size; i++) {
+    printf("%s arr[%d]: %d\n", buf, i, args.items[i].type);
+    if (args.items[i].type == kObjectTypeString) {
+      printf("%s  str: %s\n", buf, args.items[i].data.string);
+    }
+    if (args.items[i].type == kObjectTypeArray) {
+      dump_array(args.items[i].data.array, depth + 1);
+    }
+  }
+}
+
 void ui_call_event(char *name, Array args)
 {
   UIEventCallback *event_cb;
   bool handled = false;
+
+  printf("ui_call_event start: %s\n", name);
+
+  if (strcmp("msg_show", name) == 0) {
+    dump_array(args, 0);
+  }
+
+  Array orig_args = copy_array(args, NULL);
+
   map_foreach_value(&ui_event_cbs, event_cb, {
     Error err = ERROR_INIT;
     Object res = nlua_call_ref(event_cb->cb, name, args, false, &err);
@@ -672,8 +701,14 @@ void ui_call_event(char *name, Array args)
     api_clear_error(&err);
   })
 
+  printf("ui_call_event end: %s\n", name);
+
+  if (strcmp("msg_show", name) == 0) {
+    dump_array(args, 0);
+    dump_array(orig_args, 0);
+  }
   if (!handled) {
-    UI_CALL(true, event, ui, name, args);
+    UI_CALL(true, event, ui, name, orig_args);
   }
 
   ui_log(name);

[polachok]

It seems like what happens is:

1.ui_call_event("msg_show", args = { size: 3, items: 0xWHATEVER }) is called, where 0xWHATEVER is the addr of static call_buf.items
2.it enters lua callbacks loop
3.vim command redraw is called from a lua callback
4.msg_ext_flush_showmode is called from updatescreen or w/e
5. ui_call_msg_showmode reuses the same call_buf for its argument, so call_buf.items[0] becomes empty array
7. we return to ui_call_event("msg_show", args = { size: 3, items: ...}), where items[0] is an empty array, not a string, which was expected
8. UI_CALL is called with wrong arguments

[polachok]

I made a dirty hack which works around the problem for me. Not sure if it would make sense to make a PR, but if anyone is looking for a solution here, it may (or may not) work.

[bfredl]

3.vim command redraw is called from a lua callback

This is the culprit. we need to make this strictly verboten and fix the use cases that use that some other way.

[folke]

It seems that from the ui_attach lua callback, I can control whether the attached GUI also receives the ext event.

I'm further doing some tests, but it seems I can make it work so Neovide no longer crahses, even with --multigrid enabled

[Felix-Kyun]

@folke on the latest noice version, with --multigrid option on neovide im still getting an error:

You're using a GUI that uses ext_multigrid. Noice can't work when the GUI has ext_multigrid enabled.

did i mess smth up or is that expected with multigrid