excessive wakeups/cpu usage with mode=raw jobs with close_cb but not out_cb

[liskin]

Steps to reproduce

1. :let job = job_start('sh -c "echo x; cat"', {"mode": "raw", "close_cb": {->0}})
2. strace -f -p <pid-of-vim>

Expected behaviour

I expected to see the same output as I see after :let job = job_start('sh -c "echo x; cat"', {"mode": "raw", "close_cb": {->0}, "out_cb": {->0}}), that is:

pselect6(11, [0 6 8 10], [], [0], NULL, NULL

but what I got instead was a lot of

pselect6(14, [0 6 8 10], [], [0], {tv_sec=0, tv_nsec=10000000}, NULL) = 0 (Timeout)

I investigated this a bit in gdb and found it being caused by

vim/src/ui.c

Lines 370 to 371 in b74e046

	if (channel_any_readahead())
	wait_time = 10L;

What is the point of this polling? Is it useful for raw channels? I gitblamed it back to 8a8199e but can't say I understand the motivation.

(This is a real issue in https://github.com/dense-analysis/ale introduced recently by an addition of close_cb to a job without an err_cb, but it'll need to be fixed there anyway because a) needs a fix for older vim versions and b) accumulating stderr in some unbounded buffer somewhere is bad)

Version of Vim

9.0.0057

Environment

OS: Linux (Debian testing)
the rest is irrelevant I believe

Logs and stack traces

No response

[brammool]

### Steps to reproduce 1. `:let job = job_start('sh -c "echo x; cat"', {"mode": "raw", "close_cb": {->0}})` 2. `strace -f -p <pid-of-vim>` ### Expected behaviour I expected to see the same output as I see after `:let job = job_start('sh -c "echo x; cat"', {"mode": "raw", "close_cb": {->0}, "out_cb": {->0}})`, that is: ``` pselect6(11, [0 6 8 10], [], [0], NULL, NULL ``` but what I got instead was a _lot_ of ``` pselect6(14, [0 6 8 10], [], [0], {tv_sec=0, tv_nsec=10000000}, NULL) = 0 (Timeout) ``` I investigated this a bit in gdb and found it being caused by https://github.com/vim/vim/blob/b74e04649168aec579183dfef70b8784d2afe993/src/ui.c#L370-L371 What is the point of this polling? Is it useful for raw channels? I gitblamed it back to 8a8199e but can't say I understand the motivation.

The idea here, if I remember correctly, is that if there is a partly received message, then we should check again soon to see if the rest of the message is now available. Perhaps in your specific situation there is readahead but checking for more will never read anything?

…

-- hundred-and-one symptoms of being an internet addict: 115. You are late picking up your kid from school and try to explain to the teacher you were stuck in Web traffic. /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[liskin]

The idea here, if I remember correctly, is that if there is a partly received message, then we should check again soon to see if the rest of the message is now available.
Perhaps in your specific situation there is readahead but checking for more will never read anything?

The specific situation is mode=raw and no callback, so "there is readahead" just means there's pending input that hasn't been read and will not be read because there's no callback to read it, and also there's no known format/structure of the input that would suggest it's a partial input or not.

Still, my question was a bit more general: when there is a known format/structure (e.g. json), why does vim need to do frequent polling instead of just waiting for (p)select to tell it there's new input?

[brammool]

> The idea here, if I remember correctly, is that if there is a partly received message, then we should check again soon to see if the rest of the message is now available. > Perhaps in your specific situation there is readahead but checking for more will never read anything? The specific situation is mode=raw and no callback, so "there is readahead" just means there's pending input that hasn't been read and will not be read because there's no callback to read it, and also there's no known format/structure of the input that would suggest it's a partial input or not.

Vim doesn't know about that. Besides a callback there are other ways to read the text. You could add a callback that discards the text, then there won't be this unused typahead.

Still, my question was a bit more general: when there is a known format/structure (e.g. json), why does vim need to do frequent polling instead of just waiting for (`p`)`select` to tell it there's new input?

It might be possible to just depend on select() waking up, but it's very tricky. There are a lot of things going on.

…

-- "The sun oozed over the horizon, shoved aside darkness, crept along the greensward, and, with sickly fingers, pushed through the castle window, revealing the pillaged princess, hand at throat, crown asunder, gaping in frenzied horror at the sated, sodden amphibian lying beside her, disbelieving the magnitude of the frog's deception, screaming madly, "You lied!" - Winner of the Bulwer-Lytton contest (San Jose State University), wherein one writes only the first line of a bad novel /// Bram Moolenaar -- ***@***.*** -- http://www.Moolenaar.net \\\ /// \\\ \\\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[liskin]

You could add a callback that discards the text, then there won't be this unused typahead.

Sure, that's what I ended up doing, it just seemed weird having to do this, with the docs not saying something like "the input must be read or else!" :-)
But then yeah, it's fairly obvious there'd be a buffer somewhere filling up, and the excessive wakeups is just a minor issue in many cases.