After IO buffer is overrun, IO buffering is broken for future commands

[mqudsi]

all-the-package-names has recently passed the 10MiB boundary, which uncovered some problems:

mqudsi@Blitzkrieg /m/d/r/fish> math (all-the-package-names | wc -c) / 1024 / 1024
10.534367

When the io buffer boundary is reached, the shell can be left unusable:

mqudsi@Blitzkrieg /m/d/r/fish> fg
/usr/local/share/fish/config.fish (line 279): Too much data emitted by command substitution so it was discarded

    builtin fg (__fish_expand_pid_args $argv)
               ^
in function “fg”
        called on standard input

Obviously, $argv is empty here and wouldn't have returned any data, let alone > 10MB.

To reproduce,

rubish.fish:

function foo1
    for i in (seq 1 20)
        cat /usr/share/dict/words
    end
end

# Entire list of packages is too long to be used efficiently in a `complete` subcommand.
# Search it for matches instead.
function foo2
    foo1 | string match -re -- ^(commandline -ct) #| head -n 50
end

complete -c rubbish -xa '(foo2)'

followed by the following:

source ./rubbish.fish
touch ./rubbish
chmod +x ./rubbish
./rubbish <TAB>
fg

[mqudsi]

@faho: With regards to milestone, etc: I'm pretty sure this was caused by 4197420, which never shipped in any release.

Before that, there was no limit imposed on the command substitution size, so I don't think the shell could have reached this state.

[faho]

I'm pretty sure this was caused by 4197420, which never shipped in any release.

Sure, yeah.

More specifically, the "read too much" flag on the buffer is probably never reset.

Also, since we now have something that actually outputs 10MiB.... should we increase the limit?

I mean that entire thing came about because somebody noticed that reading from (IIRC) /dev/zero would "hang" fish and be generally unhelpful. So it just needs to be large enough that it's "hang detection", not something to discourage you from using the shell. So 100MiB or even 1GiB would probably be alright on most systems (obviously if you ran this on a raspberry pi with 256MiB of RAM and no swap the latter won't really work).

[faho]

Oh, wait, there actually is an underlying limit: ARG_MAX.

If we're over that the OS won't accept the arguments anyway. On my system getconf ARG_MAX returns 2097152, which is 2MiB.

So that 10MiB won't work anyway.

[mqudsi]

If we're over that the OS won't accept the arguments anyway. On my system getconf ARG_MAX returns 2097152, which is 2MiB.

No, the problem is that IO buffering is not only used for command substitution but also where it shouldn't be used such as piping output from a function's stdout to another function or process' stdin.

If you look at the example you'll note that there's no command substitution (apart from (commandline -ct), that is) going on. It's all stdout buffering.

IMHO this is fish's biggest job-related failure (way worse than the keepalive process), all function/block output is buffered when all of it should be using anonymous pipes connected directly to the read end instead. Moreover, the IO buffer isn't really a "buffer" in the traditional sense in that it's written to then read from (such as the OS' own pipe buffer) but rather a write-only buffer that becomes available for reading only after the write is completely finished, i.e. it must buffer the entire content.

[faho]

If you look at the example you'll note that there's no command substitution

The (foo2) in

complete -c rubbish -xa '(foo2)'

is a command substitution.

[faho]

I can only confirm this with rubbish <TAB>, but not rubbish a<TAB>, and if I replace (foo2) with (foo2 | string match -re ^a), it also works.

So it really does seem like it's the command substitution buffer being used for a command substitution.

[mqudsi]

Oh, I completely missed that, I was just focusing on the function definition. I'm pretty sure function output is buffered even without command substitution, but I think this bug (shell unusable) only affects command substitution.

[faho]

I'm pretty sure function output is buffered even without command substitution

Yup - #1396.

[mqudsi]

Ah, yes. That's the one.

The reason function stdout buffering doesn't run into this bug is that functions there's no limit on how big the function buffer can get, they're explicitly allowed an uncapped buffer.

[faho]

This behaves kind of weird.

$ rubbish <TAB>
fish: Too much data emitted by command substitution so it was discarded #....
$ fg
fish: Too much data emitted by command substitution so it was discarded
$ echo (echo abc)
$ fg
fg: There are no suitable jobs

Do we somehow share buffers sometimes?

[mqudsi]

I just ran into this working on the yarn completions again. It's definitely not fixed.

mqudsi@Blitzkrieg /m/d/r/fish> fg
Send job 1, “nvim share/completions/yarn.fish” to foreground
Job 1, 'nvim share/completions/yarn.fish' has stopped
mqudsi@Blitzkrieg /m/d/r/fish> fg
/usr/local/share/fish/config.fish (line 267): Too much data emitted by command substitution so it was discarded

    builtin fg (__fish_expand_pid_args $argv)
               ^
in function “fg”
        called on standard input

mqudsi@Blitzkrieg /m/d/r/fish> exit
There are still jobs active:

   PID  Command
 26038  nvim share/completions/yarn.fish

A second attempt to exit will terminate them.
Use 'disown PID' to remove jobs from the list without terminating them.

[faho]

@mqudsi: This works for me. Please confirm your $version.

[mqudsi]

It’s a recent master build but I know the test as documented here was fixed since I confirmed as much at the time. But the issue can be triggered differently.

[faho]

But the issue can be triggered differently.

@mqudsi: Any idea how? What did you do differently?

[mqudsi]

This worked (it's an even simpler test than the original one):

mqudsi@Blitzkrieg /m/d/D/CPSA Board> function foo
                                         bar (all-the-package-names)
                                     end
mqudsi@Blitzkrieg /m/d/D/CPSA Board> function bar
                                     end
mqudsi@Blitzkrieg /m/d/D/CPSA Board> foo
fish: Too much data emitted by command substitution so it was discarded

bar (all-the-package-names)
    ^
in function “foo”
        called on standard input

mqudsi@Blitzkrieg /m/d/D/CPSA Board> fg
/usr/local/share/fish/config.fish (line 267): Too much data emitted by command substitution so it was discarded

    builtin fg (__fish_expand_pid_args $argv)
               ^
in function “fg”
        called on standard input

[faho]

Okay, this works for me with 29c627.

In addition, I'm going to increase the read limit to 100MiB. That still might not blow up your little Raspi, and you could possibly pass it in multiple passes. I don't think fish is used on smaller machines, so we don't need to optimize for those.