Redirections and pipes involving blocks (like for-loops) buffer output

[CGamesPlay]

When I run this script in fish 2.1.0:

for i in (seq 1 5); sleep 1; echo $i; end | cat

I should except to see a slow count to 5, one line per second. Instead, I see no output, and then, five seconds later, all 5 lines appear. This appears to be the same issue as #1302.

This appears to happen regardless of the amount of output:

for i in (seq 1 5); sleep 1; cat /usr/share/dict/words; end | cat

[ridiculousfish]

Yes, redirections and pipes involving blocks are run serially, not in parallel.

[CGamesPlay]

Thanks for getting back to me so quickly. Is there a way to work around this? I'm running a series of long-running scripts and tee'ing the output to a log file. I'd like to monitor the log as it happens, but the shell just stores up all of the output until the very end.

[ridiculousfish]

It's a longstanding and annoying issue.

[xfix]

There is a workaround for it, but it's really ugly, and probably not a good idea (besides, it ignores scopes, unless you are going to use universal variables removed after leaving the scope).

fish -c 'for i in (seq 1 5)
    sleep 1
    echo $i
end' | cat

It would be really nice to fix it, but nobody made a fix yet.

[tannhuber]

It's annoying, indeed. Does anybody have an idea how to fix it?

[ridiculousfish]

The long-term plan is to support full multithreaded execution. See #563

[tannhuber]

Would be great. Is this also the reason, why

function fish_prompt
        echo -n "prompt: "
        sleep 3 # in reality there's a time consuming command after printing the prompt
end

shows the prompt with 3 seconds delay?

[lilyball]

@tannhuber No, that's a different issue. That's also correct behavior. Which is to say, the prompt isn't supposed to paint until all the data is available. In any case, you can think of fish_prompt as being executed in a manner similar to

set -l left_prompt $default_prompt
if functions -q fish_prompt
    set left_prompt (fish_prompt)
end
set -l right_prompt ''
if functions -q fish_right_prompt
    set right_prompt (fish_right_prompt)
end
internal_read_line --left-prompt $left_prompt --right-prompt $right_prompt

In this pseudocode it should be obvious that nothing is painted until the entire prompt command has finished (and the right prompt command, if provided).

[tannhuber]

Makes sense. I know how I could do it without this delay. Thank you for clarifying.

[xfix]

It's actually more tricky, even ignoring right prompts. Fish shell automatically shortens too long prompts.

xfix@damson ~> function fish_prompt
                   perl -e 'print "x" x 1e5'
               end
>

For this to work, fish has to wait.

[enricotagliavini]

Just adding an explicit comment here in case somebody, like me, encounter the same problem and searches for already open bugs. When piping large amount of data via scripts and functions fish might exhaust available RAM memory, producing a system crash or triggering the OOM killer (on platforms having one).

One notable case: grep. If grep is defined as a function, usually to automatically add --color=auto like in bash aliases, don't process large files (producing large outputs). I triggered this by processing apache httpd log files (many GBs) and boom, OOM.

More details in issue #2803 .

Limiting the buffer size and / or forcing a flush would be nice, if possible.

[jonhoo]

This just came back to bite me again today :( In particular, I have a function that sets some secret keys in environment variables, and then ran a long-running program that produces log output. But the function causes no log output to be produced until the whole program exits, which is pretty unfortunate.

For those tracking this issue, I tried to figure out the current state of affairs: #1396 (comment) suggested that this would be supported with "full multithreaded execution" and references #563. That issue has now been closed as a duplicate of #238. In #238 (comment), @krader1961 states that there isn't a plan to add multi-threading support any time soon, so looks like we'll just have to wait patiently (or someone has to step up and write a PR).

[mqudsi]

Just a ping for those subscribed: @ridiculousfish committed some improvements to this as discussed in #5635

[zanchey]

Actually all the examples given seem to work now - is there still work to be done here?

[ridiculousfish]

The way this was done was through a hack which covered the majority of cases, but not all, for example:

for i in (seq 1 5); sleep 1; echo $i; end | begin ; cat ; end

[exploide]

I tried to fix #5340 again and this case also doesn't work, yet.

When using the following function for ip:

#
# Enable colored output for ip when available
#

if command -sq ip; and command ip -color=auto l >/dev/null 2>/dev/null
    function ip
        command ip -color=auto $argv
    end
end

ip monitor | grep dev does not work. (I.e. piping slowly out of a fish function into a fish function)