New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"... &" background statement cannot work with functions? #238

Open
sdothum opened this Issue Jul 19, 2012 · 20 comments

Comments

Projects
None yet
@sdothum

sdothum commented Jul 19, 2012

Does the "... &" statement only work with simple commands and not functions? For example, if I have defined

function q; echo $argv; sleep 10s; echo $argv; end

and execute

q 1 &; q 2 &; q 3 &

the output is

1
1
2
2
3
3
Job 3, “q 3 &” has ended
Job 2, “q 2 &” has ended
Job 1, “q 1 &” has ended

with the observed performance (delay) indicating that each successive "q" command is not executed until the previous command is completed. I use in POSIX sh "(sleep [some delay] && [command]) &" quite commonly in startup scripts e.g. for window managers to allow for the initialization requirements of certain apps.

If there is a correct "fish" way of spawning multiple background processes, my second question is: is there a way to spawn a block of commands in background similar to POSIX "(command1 && command2 && command3...) &". I tried "begin; ...statements...; end &" but fish does not like that!

@siteshwar

This comment has been minimized.

Show comment
Hide comment
@siteshwar

siteshwar Jul 21, 2012

Member

On Linux, executing "vi &" doesn't take vi to background, and at exit I get following message :

Job 1, “vi &” has ended

Is anyone else able to reproduce it ?

Member

siteshwar commented Jul 21, 2012

On Linux, executing "vi &" doesn't take vi to background, and at exit I get following message :

Job 1, “vi &” has ended

Is anyone else able to reproduce it ?

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Jul 21, 2012

Member

fish probably doesn't spawn a subshell for functions executed as background jobs. If it did, those jobs probably could not do things like update global variables.

Member

ridiculousfish commented Jul 21, 2012

fish probably doesn't spawn a subshell for functions executed as background jobs. If it did, those jobs probably could not do things like update global variables.

@smlx

This comment has been minimized.

Show comment
Hide comment
@smlx

smlx Aug 12, 2012

Contributor
$ vi &
Job 1, “vi &” has stopped

Seems to work correctly here using fish packaged in debian:
Version: 1.23.1+20120106.git8b407a3-1

Could be a regression?

Contributor

smlx commented Aug 12, 2012

$ vi &
Job 1, “vi &” has stopped

Seems to work correctly here using fish packaged in debian:
Version: 1.23.1+20120106.git8b407a3-1

Could be a regression?

@siteshwar

This comment has been minimized.

Show comment
Hide comment
@siteshwar

siteshwar Aug 12, 2012

Member

I had written a function named "vi" (alias like) to invoke vi editor, so I was not able to invoke vi in background. This should be fixed, we need to add support for executing functions in background.

Member

siteshwar commented Aug 12, 2012

I had written a function named "vi" (alias like) to invoke vi editor, so I was not able to invoke vi in background. This should be fixed, we need to add support for executing functions in background.

@robacarp

This comment has been minimized.

Show comment
Hide comment
@robacarp

robacarp Apr 26, 2013

This bit me in a rather confusing way just now.

Several weeks ago I defined an alias: alias redis="redis-server /usr/local/etc/redis.conf" today, I was trying to run redis and elasticsearch together in the same console, both as background jobs, so that their output would be in the same shell and typed redis & elasticsearch &. Instead of what I was expecting, I got just the redis server and on exit, it spawned elasticsearch in a background job.

I understand what you're saying about subshells for functions being overkill, but I think that, as with the !! syntax, if there is a way to achieve this functionality without changing things, it'll meet the expectations of more people migrating from bash/zsh, because you can background just about everything in a bash.

Thanks again for all your work on this awesome shell!

robacarp commented Apr 26, 2013

This bit me in a rather confusing way just now.

Several weeks ago I defined an alias: alias redis="redis-server /usr/local/etc/redis.conf" today, I was trying to run redis and elasticsearch together in the same console, both as background jobs, so that their output would be in the same shell and typed redis & elasticsearch &. Instead of what I was expecting, I got just the redis server and on exit, it spawned elasticsearch in a background job.

I understand what you're saying about subshells for functions being overkill, but I think that, as with the !! syntax, if there is a way to achieve this functionality without changing things, it'll meet the expectations of more people migrating from bash/zsh, because you can background just about everything in a bash.

Thanks again for all your work on this awesome shell!

@JanKanis

This comment has been minimized.

Show comment
Hide comment
@JanKanis

JanKanis Apr 30, 2013

Member

This is a known problem, see also issue #563.

Unlike most other shells fish is a multithreaded program, so it can not
just be forked into a new subshell without coordinating the multiple
threads. That could be done but no-one has taken the time to do so. The
other (preferable) solution is to have background functions execute in
different threads, this should be possible but would also require some work
to be implemented.

Member

JanKanis commented Apr 30, 2013

This is a known problem, see also issue #563.

Unlike most other shells fish is a multithreaded program, so it can not
just be forked into a new subshell without coordinating the multiple
threads. That could be done but no-one has taken the time to do so. The
other (preferable) solution is to have background functions execute in
different threads, this should be possible but would also require some work
to be implemented.

zanchey added a commit to zanchey/fish-shell that referenced this issue May 12, 2013

ridiculousfish added a commit that referenced this issue May 13, 2013

@geoff-codes

This comment has been minimized.

Show comment
Hide comment
@geoff-codes

geoff-codes Oct 16, 2013

Contributor

Ah. Well that would also explain #1040 then, I think.

Contributor

geoff-codes commented Oct 16, 2013

Ah. Well that would also explain #1040 then, I think.

@maxnordlund

This comment has been minimized.

Show comment
Hide comment
@maxnordlund

maxnordlund Oct 23, 2015

Contributor

Hm, what if calls to builtins, like set, were synchronized? @ridiculousfish Then subshells could update global/universal variables but allow everything else to run concurrently, or am I missing something else?

Contributor

maxnordlund commented Oct 23, 2015

Hm, what if calls to builtins, like set, were synchronized? @ridiculousfish Then subshells could update global/universal variables but allow everything else to run concurrently, or am I missing something else?

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Oct 23, 2015

Member

@maxnordlund The biggest obstacle is that fish uses the C stack for builtins, functions, and command substitutions. Therefore executing two of these concurrently means multiple threads, multiple processes, or something like coroutines.

I have a branch that tries to support multithreaded execution, but it's very complex so I'm not optimistic that it will work.

Member

ridiculousfish commented Oct 23, 2015

@maxnordlund The biggest obstacle is that fish uses the C stack for builtins, functions, and command substitutions. Therefore executing two of these concurrently means multiple threads, multiple processes, or something like coroutines.

I have a branch that tries to support multithreaded execution, but it's very complex so I'm not optimistic that it will work.

@maxnordlund

This comment has been minimized.

Show comment
Hide comment
@maxnordlund

maxnordlund Oct 23, 2015

Contributor

I was leaning towards multiple threads, and come to think of maybe it would be easier if script execution was always running in the background. Every time it needed a builtin it synchronized/sent a message to the main thread to do the work.

Sort of like most GUI frameworks keep one thread responsible for rendering, and you always have to ask it if you need an update.

Contributor

maxnordlund commented Oct 23, 2015

I was leaning towards multiple threads, and come to think of maybe it would be easier if script execution was always running in the background. Every time it needed a builtin it synchronized/sent a message to the main thread to do the work.

Sort of like most GUI frameworks keep one thread responsible for rendering, and you always have to ask it if you need an update.

@Lucretiel

This comment has been minimized.

Show comment
Hide comment
@Lucretiel

Lucretiel Oct 29, 2015

Contributor

Fish shell already uses libevent, right? Couldn't we keep it single threaded, but switch builtin and function execution to an asynchronous or coroutine model? That solves the global state atomicity issue- only one piece of code is ever running at a time, so changes to variables, working directory, etc all happen atomically. Once all pipelines are converted to true unix pipes, rather than being fully buffered between functions (see #936), the pipe's file descriptors can be selected over in the libevent event loop, allowing for single-threaded concurrency in the same vein as node.js or python's asyncio.

We should probably assume that users who are backgrounding fish functions are aware and accepting of the possibility of global state mutation, rather than attempting to simulate multiple working directories (for instance, with openat). This is already the behavior of background jobs that write to stdout. One of the main fish features is that it never forks except when launching a program, so cd and set and related commands never don't work as expected. Therefore, when a user writes:

function contrived
    sleep 5
    set -g MY_VAR hello
    cd $argv[1]
end

contrived /another/directory

The user clearly wants the current shell to switch to /another/directory and set MY_VAR globally to hello after 5 seconds. We should keep this behavior for background jobs- if the user instead does contrived /another/directory &, the working directory and variable should still be changed after 5 seconds, even though it happened in the background. This is the same as background jobs sharing stdout/stderr with the foreground. Fish doesn't currently have a mechanism for a "local" working directory, in the same way it has "local" variables; that should probably be a separate (though related) discussion. cd -l, anyone?

Contributor

Lucretiel commented Oct 29, 2015

Fish shell already uses libevent, right? Couldn't we keep it single threaded, but switch builtin and function execution to an asynchronous or coroutine model? That solves the global state atomicity issue- only one piece of code is ever running at a time, so changes to variables, working directory, etc all happen atomically. Once all pipelines are converted to true unix pipes, rather than being fully buffered between functions (see #936), the pipe's file descriptors can be selected over in the libevent event loop, allowing for single-threaded concurrency in the same vein as node.js or python's asyncio.

We should probably assume that users who are backgrounding fish functions are aware and accepting of the possibility of global state mutation, rather than attempting to simulate multiple working directories (for instance, with openat). This is already the behavior of background jobs that write to stdout. One of the main fish features is that it never forks except when launching a program, so cd and set and related commands never don't work as expected. Therefore, when a user writes:

function contrived
    sleep 5
    set -g MY_VAR hello
    cd $argv[1]
end

contrived /another/directory

The user clearly wants the current shell to switch to /another/directory and set MY_VAR globally to hello after 5 seconds. We should keep this behavior for background jobs- if the user instead does contrived /another/directory &, the working directory and variable should still be changed after 5 seconds, even though it happened in the background. This is the same as background jobs sharing stdout/stderr with the foreground. Fish doesn't currently have a mechanism for a "local" working directory, in the same way it has "local" variables; that should probably be a separate (though related) discussion. cd -l, anyone?

@ridiculousfish

This comment has been minimized.

Show comment
Hide comment
@ridiculousfish

ridiculousfish Oct 29, 2015

Member

No, fish does not use libevent.

Making the internals thread safe is part of it. There's also the issue of replicating process state. For example, we need to have multiple variable stacks, since two separately executing functions need independent local variables. We need separate values of $status. We need to have separate notions of "last backgrounded process" since functions may use that. We need separate working directories - I don't think it's acceptable to prohibit cd for two simultaneously executing functions.

I did some work on this in the multithreaded_execution branch, but I think as it stands it adds too much complexity to merge, and if we want to go down this path, it will involve a lot of rewriting.

Member

ridiculousfish commented Oct 29, 2015

No, fish does not use libevent.

Making the internals thread safe is part of it. There's also the issue of replicating process state. For example, we need to have multiple variable stacks, since two separately executing functions need independent local variables. We need separate values of $status. We need to have separate notions of "last backgrounded process" since functions may use that. We need separate working directories - I don't think it's acceptable to prohibit cd for two simultaneously executing functions.

I did some work on this in the multithreaded_execution branch, but I think as it stands it adds too much complexity to merge, and if we want to go down this path, it will involve a lot of rewriting.

@roeebar

This comment has been minimized.

Show comment
Hide comment
@roeebar

roeebar Mar 14, 2017

Are there any plans to add multithreading support to functions in the near future?
Since an alias is actually a function, a large piped 'grep' can take forever... (since 'grep' is usually an alias)

roeebar commented Mar 14, 2017

Are there any plans to add multithreading support to functions in the near future?
Since an alias is actually a function, a large piped 'grep' can take forever... (since 'grep' is usually an alias)

@krader1961

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 Mar 14, 2017

Contributor

Are there any plans to add multithreading support to functions in the near future?

No. With 530 open issues as I type this patches are welcomed.

Contributor

krader1961 commented Mar 14, 2017

Are there any plans to add multithreading support to functions in the near future?

No. With 530 open issues as I type this patches are welcomed.

@pirate

This comment has been minimized.

Show comment
Hide comment
@pirate

pirate May 20, 2017

Powershell just released proper backgrounding, hopefully fish gets it soon! :)
Until then, this hack is still working well for me (though backgrounded tasks don't have access to local vars from the current shell):

function background --description "Run a fish command in the background using a bash subshell"
    bash -c "fish -c '""$argv""' &"
end

background "sleep 5; echo hi"

pirate commented May 20, 2017

Powershell just released proper backgrounding, hopefully fish gets it soon! :)
Until then, this hack is still working well for me (though backgrounded tasks don't have access to local vars from the current shell):

function background --description "Run a fish command in the background using a bash subshell"
    bash -c "fish -c '""$argv""' &"
end

background "sleep 5; echo hi"
@Lucretiel

This comment has been minimized.

Show comment
Hide comment
@Lucretiel

Lucretiel May 20, 2017

Contributor
Contributor

Lucretiel commented May 20, 2017

@pirate

This comment has been minimized.

Show comment
Hide comment
@pirate

pirate May 20, 2017

@Lucretiel notice the & on the end is part of the bash command. Bash provides backgrounded subshells with consistent behaviour, fish does not.

Edit: maybe I misunderstood, did you mean call fish -c 'command' & from within fish? It produces a line: 'fish -c 'sleep 10; echo hi' &' has ended which I'd rather suppress.

pirate commented May 20, 2017

@Lucretiel notice the & on the end is part of the bash command. Bash provides backgrounded subshells with consistent behaviour, fish does not.

Edit: maybe I misunderstood, did you mean call fish -c 'command' & from within fish? It produces a line: 'fish -c 'sleep 10; echo hi' &' has ended which I'd rather suppress.

@krader1961

This comment has been minimized.

Show comment
Hide comment
@krader1961

krader1961 May 20, 2017

Contributor

Bash provides backgrounded subshells with consistent behaviour, fish does not.

I have no idea where you got that idea. What exactly is wrong with simply using fish to do this? This is commonly known as the double-fork trick and it works fine with fish.

Contributor

krader1961 commented May 20, 2017

Bash provides backgrounded subshells with consistent behaviour, fish does not.

I have no idea where you got that idea. What exactly is wrong with simply using fish to do this? This is commonly known as the double-fork trick and it works fine with fish.

@pirate

This comment has been minimized.

Show comment
Hide comment
@pirate

pirate May 20, 2017

@krader1961, sorry I mistakenly thought that because fish had issues backgrounding functions, that I wouldn't be able to trust it to background other commands consistently as well.
fish -c '...' & seems to work fine, although I'm going to keep using the bash double-fork because I don't want the 'fish -c 'sleep 10; echo hi' &' has ended line when fish bg jobs finish.

pirate commented May 20, 2017

@krader1961, sorry I mistakenly thought that because fish had issues backgrounding functions, that I wouldn't be able to trust it to background other commands consistently as well.
fish -c '...' & seems to work fine, although I'm going to keep using the bash double-fork because I don't want the 'fish -c 'sleep 10; echo hi' &' has ended line when fish bg jobs finish.

@sabinem

This comment has been minimized.

Show comment
Hide comment
@sabinem

sabinem Jun 20, 2018

Contributor

I think it is acceptable that functions cannot be put in the background, but it should be better communicated to the user. Now it just runs in the foreground anyway, even though the user requested the background: it is blocking the shell and in the end the user sees: Job 1, 'hello &' has ended even though there never was a job running in the background.

I think in case of a request hallo &, where hello is a function, there should be a syntax error or other message, that tells the user that this command is a function and won't run in the background.

What do you think about this? Fish stands for a friendly shell: it should tell the user what it does in my humble opinion.

Contributor

sabinem commented Jun 20, 2018

I think it is acceptable that functions cannot be put in the background, but it should be better communicated to the user. Now it just runs in the foreground anyway, even though the user requested the background: it is blocking the shell and in the end the user sees: Job 1, 'hello &' has ended even though there never was a job running in the background.

I think in case of a request hallo &, where hello is a function, there should be a syntax error or other message, that tells the user that this command is a function and won't run in the background.

What do you think about this? Fish stands for a friendly shell: it should tell the user what it does in my humble opinion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment