Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redirections and pipes involving blocks (like for-loops) buffer output #1396

Open
CGamesPlay opened this issue Apr 2, 2014 · 20 comments
Open

Redirections and pipes involving blocks (like for-loops) buffer output #1396

CGamesPlay opened this issue Apr 2, 2014 · 20 comments

Comments

@CGamesPlay
Copy link

@CGamesPlay CGamesPlay commented Apr 2, 2014

When I run this script in fish 2.1.0:

for i in (seq 1 5); sleep 1; echo $i; end | cat

I should except to see a slow count to 5, one line per second. Instead, I see no output, and then, five seconds later, all 5 lines appear. This appears to be the same issue as #1302.

This appears to happen regardless of the amount of output:

for i in (seq 1 5); sleep 1; cat /usr/share/dict/words; end | cat
@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Apr 3, 2014

Yes, redirections and pipes involving blocks are run serially, not in parallel.

@CGamesPlay
Copy link
Author

@CGamesPlay CGamesPlay commented Apr 3, 2014

Thanks for getting back to me so quickly. Is there a way to work around this? I'm running a series of long-running scripts and tee'ing the output to a log file. I'd like to monitor the log as it happens, but the shell just stores up all of the output until the very end.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Apr 6, 2014

It's a longstanding and annoying issue.

@xfix
Copy link
Contributor

@xfix xfix commented Apr 7, 2014

There is a workaround for it, but it's really ugly, and probably not a good idea (besides, it ignores scopes, unless you are going to use universal variables removed after leaving the scope).

fish -c 'for i in (seq 1 5)
    sleep 1
    echo $i
end' | cat

It would be really nice to fix it, but nobody made a fix yet.

@tannhuber
Copy link
Contributor

@tannhuber tannhuber commented Sep 4, 2014

It's annoying, indeed. Does anybody have an idea how to fix it?

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Sep 4, 2014

The long-term plan is to support full multithreaded execution. See #563

@tannhuber
Copy link
Contributor

@tannhuber tannhuber commented Sep 4, 2014

Would be great. Is this also the reason, why

function fish_prompt
        echo -n "prompt: "
        sleep 3 # in reality there's a time consuming command after printing the prompt
end

shows the prompt with 3 seconds delay?

@lilyball
Copy link
Contributor

@lilyball lilyball commented Sep 4, 2014

@tannhuber No, that's a different issue. That's also correct behavior. Which is to say, the prompt isn't supposed to paint until all the data is available. In any case, you can think of fish_prompt as being executed in a manner similar to

set -l left_prompt $default_prompt
if functions -q fish_prompt
    set left_prompt (fish_prompt)
end
set -l right_prompt ''
if functions -q fish_right_prompt
    set right_prompt (fish_right_prompt)
end
internal_read_line --left-prompt $left_prompt --right-prompt $right_prompt

In this pseudocode it should be obvious that nothing is painted until the entire prompt command has finished (and the right prompt command, if provided).

@tannhuber
Copy link
Contributor

@tannhuber tannhuber commented Sep 5, 2014

Makes sense. I know how I could do it without this delay. Thank you for clarifying.

@xfix
Copy link
Contributor

@xfix xfix commented Sep 6, 2014

It's actually more tricky, even ignoring right prompts. Fish shell automatically shortens too long prompts.

xfix@damson ~> function fish_prompt
                   perl -e 'print "x" x 1e5'
               end
>

For this to work, fish has to wait.

@ridiculousfish ridiculousfish added this to the fish-future milestone Oct 12, 2014
@jonhoo jonhoo mentioned this issue Dec 6, 2014
@faho faho changed the title For loop buffers output Redirections and pipes involving blocks (like for-loops) buffer output Jan 18, 2016
@enricotagliavini
Copy link

@enricotagliavini enricotagliavini commented Mar 8, 2016

Just adding an explicit comment here in case somebody, like me, encounter the same problem and searches for already open bugs. When piping large amount of data via scripts and functions fish might exhaust available RAM memory, producing a system crash or triggering the OOM killer (on platforms having one).

One notable case: grep. If grep is defined as a function, usually to automatically add --color=auto like in bash aliases, don't process large files (producing large outputs). I triggered this by processing apache httpd log files (many GBs) and boom, OOM.

More details in issue #2803 .

Limiting the buffer size and / or forcing a flush would be nice, if possible.

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Apr 13, 2018

This just came back to bite me again today :( In particular, I have a function that sets some secret keys in environment variables, and then ran a long-running program that produces log output. But the function causes no log output to be produced until the whole program exits, which is pretty unfortunate.

For those tracking this issue, I tried to figure out the current state of affairs: #1396 (comment) suggested that this would be supported with "full multithreaded execution" and references #563. That issue has now been closed as a duplicate of #238. In #238 (comment), @krader1961 states that there isn't a plan to add multi-threading support any time soon, so looks like we'll just have to wait patiently (or someone has to step up and write a PR).

@paddor

This comment has been minimized.

@zanchey

This comment has been minimized.

@faho

This comment has been minimized.

@paddor

This comment has been minimized.

@mqudsi
Copy link
Contributor

@mqudsi mqudsi commented Mar 28, 2019

Just a ping for those subscribed: @ridiculousfish committed some improvements to this as discussed in #5635

@zanchey
Copy link
Member

@zanchey zanchey commented Sep 3, 2019

Actually all the examples given seem to work now - is there still work to be done here?

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Sep 8, 2019

The way this was done was through a hack which covered the majority of cases, but not all, for example:

for i in (seq 1 5); sleep 1; echo $i; end | begin ; cat ; end

@exploide
Copy link
Contributor

@exploide exploide commented Sep 24, 2019

I tried to fix #5340 again and this case also doesn't work, yet.

When using the following function for ip:

#
# Enable colored output for ip when available
#

if command -sq ip; and command ip -color=auto l >/dev/null 2>/dev/null
    function ip
        command ip -color=auto $argv
    end
end

ip monitor | grep dev does not work. (I.e. piping slowly out of a fish function into a fish function)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests