Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compatibility with GNU Parallel #1084

Closed
ogoid opened this issue Nov 1, 2013 · 10 comments
Closed

Compatibility with GNU Parallel #1084

ogoid opened this issue Nov 1, 2013 · 10 comments
Assignees
Labels
bug
Milestone

Comments

@ogoid
Copy link

@ogoid ogoid commented Nov 1, 2013

I tried parallelizing some programs with GNU Parallel (http://www.gnu.org/software/parallel/), but fish produces some warnings/errors. For instance:

~> seq 1 6 | parallel echo
3
4
2
1
5
6
fish: Could not connect to universal variable server, already tried manual restart (or no command supplied). You will not be able to share variable values between fish sessions. Is fish properly installed?
~> 

Or:

~> seq 1 6 | parallel --pipe echo
fish: Expected a command name, got token of type 'Run job in background'. Did you mean 'COMMAND; and COMMAND'? See the help section for the 'and' builtin command by typing 'help and'.
Standard input:              test ! -s "/var/folders/w9/9jx0r60d6c1g5rrbkps8rrr00000gn/T/Y6K17QPi1f.chr" && rm -f "/var/folders/w9/9jx0r60d6c1g5rrbkps8rrr00000gn/T/Y6K17QPi1f.chr" && exec true;
                                                                                                          ^
~> 

I'm running OSX 10.9 with fish installed by Homebrew and set as my default shell.

@zanchey
Copy link
Member

@zanchey zanchey commented Nov 5, 2013

For the second problem, Parallel tries to execute a POSIX script in the current $SHELL, but if fish is your login shell then that may not work. You could try setting your shell to bash temporarily:

function parallel;
  set -lx SHELL bash
  command parallel $argv
end

The first problem is proving harder to replicate. It occurs less than 1% of the time (try seq 1 1000 | parallel --gnu -n 1 echo) on one of my machines running fish 2.1.0 and parallel 20120422, and never on my machine running fish 2.1.0-31-g3c65cd4 and parallel 20121122. There is probably a race condition in connecting to fishd which parallel is happening to tickle.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Nov 5, 2013

If we ever do threaded execution #563 we could provide a totally kick-ass builtin parallel.

@zanchey
Copy link
Member

@zanchey zanchey commented Nov 11, 2013

If you start a huge stack of fish processes, fishd eventually dies with SIGPIPE. Does the fish side of things get done on a background thread or something? I wonder if fish is starting, finishing its work before getting a reply from fishd and tearing everything down. Would it be safe to set SIGPIPE as ignored in the startup of fishd? It will still get EPIPE but that can be safely ignored.

@zanchey
Copy link
Member

@zanchey zanchey commented Nov 11, 2013

This appears to fix the problem:

diff --git a/fishd.cpp b/fishd.cpp
index 5e2a364..30ded3c 100644
--- a/fishd.cpp
+++ b/fishd.cpp
@@ -667,13 +667,14 @@ static void daemonize()
             setup_fork_guards();

             /*
-              Make fishd ignore the HUP signal.
+              Make fishd ignore the HUP and PIPE signals.
             */
             struct sigaction act;
             sigemptyset(& act.sa_mask);
             act.sa_flags=0;
             act.sa_handler=SIG_IGN;
             sigaction(SIGHUP, &act, 0);
+            sigaction(SIGPIPE, &act, 0);

             /*
               Make fishd save and exit on the TERM signal.

Not sure if it's ok though.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Nov 11, 2013

I think this fix is the right thing to do. But before we apply it I want to understand why we're getting SIGPIPE. My understanding is that all communication with fishd happens on the main thread and is synchronous, and furthermore that all data sent by fishd is in response to a request from a fish instance (so fishd never initiates). Based on that I'd expect to never get SIGPIPE unless a fish instance crashes, and if we're crashing I very much want to know :)

zanchey, can you share your technique for starting a huge stack of fish instances?

@lledey
Copy link
Contributor

@lledey lledey commented Nov 11, 2013

I'm not sure this zanchey's technique (sorry if it's not), but the following makes things go really wrong :

for i in (seq 10)
    fish -c exit &
end

This produces A LOT of error messages and sometimes even backtraces.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Nov 11, 2013

Ooh, nice find. Thanks!

@zanchey
Copy link
Member

@zanchey zanchey commented Nov 12, 2013

As above - I use seq 1 1000 | parallel --gnu -n 1 echo but @lledey's suggestion seems to work too.

The failure is disappointingly intermittent; on my local machine, I am yet to be able to reproduce it, but it is much more frequent on some remote machines.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Feb 14, 2014

Now I get it - it's very simple. If fishd begins a write before fish starts reading it, and fish exits, then fishd will get SIGPIPE. Ignoring SIGPIPE is the right thing to do.

@ridiculousfish
Copy link
Member

@ridiculousfish ridiculousfish commented Feb 15, 2014

Should be fixed by de8bae3 . Thanks for reporting this.

@zanchey zanchey modified the milestones: next-minor, next-major Feb 15, 2014
@zanchey zanchey modified the milestones: 2.1.1, next-minor Sep 24, 2014
@zanchey zanchey self-assigned this Sep 24, 2014
@zanchey zanchey added the bug label Sep 24, 2014
ridiculousfish added a commit that referenced this issue Sep 24, 2014
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.