Job handling is broken.

[layus]

I have an issue running the command sudo echo lol | grep lol. This command is not as trivial as it seems because grep is a fish function, calling command grep --color=auto $argv behind the scene.

Now, a simple run of that command gives

layus@klatch ~/p/fish> sudo echo lol | grep lol
# Nothing happens, job is blocked. Entering ^Z.
Job 1, 'sudo echo lol | grep lol' has stopped
layus@klatch ~/p/fish> jobs
Job	Group	CPU	State	Command
1	7318	0%	stopped	sudo echo lol | grep lol
2	7319	0%	stopped	command grep --color=auto $argv
layus@klatch ~/p/fish> fg
Send job 1, 'sudo echo lol | grep lol' to foreground
[sudo] password for layus : 
layus@klatch ~/p/fish> fg
Send job 2, 'command grep --color=auto $argv' to foreground
lol
layus@klatch ~/p/fish> 

This behavior is not changed by #3922, and may relate to #206. We see that fish starts two jobs for one user job, and that these jobs are managed separately. Digging in the source code, we can find that one job is created for the top-level pipeline.

exec_job starts processes for each step of the pipeline. When it reaches grep, it detects a fish function and calls internal_exec_helper to start its body.
That function parses the function body and starts a new job to to handle its content.
That new job is not aware of its parent context, and start with a new pgid. Fish gives that process control over the terminal, removing it from sudo.

When sudo starts, it receives a SIGTTOU and gets stopped. Fish blocks because the grep job is still running, and unrelated to the sudo job.

Running ps to compare fish to other shells gives

 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
===== ===== ===== ===== ======== ===== ===== ==== ====== =========
    1   764   764   764 ?           -1 Ss       0   0:00 sshd [...]
  764 31954 31954 31954 ?           -1 Ss       0   0:00  \_ sshd: me [priv]
31954 31956 31954 31954 ?           -1 S     1000   0:00  |   \_ sshd: me@pts/0
31956 31957 31957 31957 pts/0     7319 Ss    1000   0:00  |       \_ -zsh
31957  7306  7306 31957 pts/0     7319 S     1000   0:00  |           \_ ./fish -d4
 7306  7317  7306 31957 pts/0     7319 S     1000   0:00  |               \_ ./fish -d4
 7306  7318  7318 31957 pts/0     7319 T        0   0:00  |               \_ sudo echo lol
 7306  7319  7319 31957 pts/0     7319 S+    1000   0:00  |               \_ grep --color=auto lol
  764  7390  7390  7390 ?           -1 Ss       0   0:00  \_ sshd: me [priv]
 7390  7392  7390  7390 ?           -1 S     1000   0:00  |   \_ sshd: me@pts/4
 7392  7393  7393  7393 pts/4     7470 Ss    1000   0:00  |       \_ -zsh
 7393  7460  7460  7393 pts/4     7470 S     1000   0:00  |           \_ zsh
 7460  7470  7470  7393 pts/4     7470 S+       0   0:00  |               \_ sudo echo lol
 7460  7471  7470  7393 pts/4     7470 S+    1000   0:00  |               \_ grep --color=auto lol
  764  7474  7474  7474 ?           -1 Ss       0   0:00  \_ sshd: me [priv]
 7474  7476  7474  7474 ?           -1 S     1000   0:00      \_ sshd: me@pts/5
 7476  7477  7477  7477 pts/5     7530 Ss    1000   0:00          \_ -zsh
 7477  7519  7519  7477 pts/5     7530 S     1000   0:00              \_ bash
 7519  7530  7530  7477 pts/5      7530 S+       0   0:00                  \_ sudo echo lol
 7519  7531  7530  7477 pts/5      7530 S+    1000   0:00                  \_ grep lol

We can see that the current TPGID is 7319 while the sudo command has a PGID of 7318. It's status is T, which means that the process is "stopped, either by a job control signal or because it is being traced". Here it is because of the aforementioned SIGTTOU.
We see that other shells give the same PGID to all the subcommands.

The extra fish process is the "keepalive fork" used by fish. I still wonder why fish needs it while other shells do not, but it is not important.

From a job control point of view, fish shows two running jobs in jobs output, while bash and zsh will show only one, even if zsh seems to distinguish between the processes of the job. (^Z was added for clarity, but does not appear in the terminal)

ZSH:
--(layus@klatch)--> grep () { command grep --color=auto "$@"; }   
--(layus@klatch)-->  sudo echo lol | grep lol                      
[sudo] password for layus: ^Z
zsh: running    sudo echo lol | grep --color=auto lol
--(layus@klatch)--> jobs
[1]  + suspended           sudo echo lol | 
       suspended (signal)  grep --color=auto lol

BASH:
layus@klatch:~$ grep () { command grep --color=auto "$@"; }
layus@klatch:~$ sudo echo lol | grep lol
[sudo] password for layus: ^Z

[1]+  Stopped                 sudo echo lol | grep lol
layus@klatch:~$ jobs
[1]+  Stopped                 sudo echo lol | grep lol

We need to make function invocation part of the same job as their context. I believe that one command line should create one job in the jobs view. That would also immediately fix the SIGTTOU issue for sudo because both processes will have the same PGID and will both have control over the terminal.
Because they are handled exctly like function, begin/end blocks would be fixed at the same time.

---

On current master (version 2.5.0-259-gc0de8afa-dirty, aka c0de8af).

[krader1961]

Job control in fish is definitely idiosyncratic. I can't be bothered to decide if this is a duplicate of one of the already open issues on this topic.

Your description of the problem, @layus, is fantastic. Any chance we can entice you to take a stab at fixing some of the job control issues even if not this specific one? For that matter, your approach to such matters suggests to me that you could contribute meaningful improvements to fish in practically any area if you are comfortable making changes to C/C++ code.

[layus]

Thanks @krader1961 :-).

I can't be bothered to decide if this is a duplicate of one of the already open issues on this topic.

Me neither, that's why I opened a new Issue.

What bothers me is that the parsing and execution are tighly coupled. I would rather refactor that code to separate parsing and execution. What I hoped to find in fish is a cleaner shell implementation than zsh. I am a bit disappointed to find this very complex code in exec_job.

I will probably take time to improve on this issue, but I secretly hoped that sharing my discovery would trigger someone else to do it. Fish is not my top priority, and I already spent way too much time on these issues. that being said, leaving things as-is will probably itch me too much ;-).

[krader1961]

I would rather refactor that code to separate parsing and execution.

Yes, please. I made a huge number of lint cleanups a year ago including some refactoring. But I mostly left the job management code untouched because, frankly, many of the functions are so large and have so many deeply nested blocks they're borderline incomprehensible. Refactoring that code before making functional changes is a very good idea. In fact, I consider it borderline mandatory. If you make lint-all you'll see what I mean.

[ThomasAH]

I guess this is the same issue:
ssh foo.example.com ps auxww | grep something
does not print anything and seems to hang when it wants the passphrase of an ssh key or the password of the target server.

After Ctrl-C I get a new prompt, but nothing else is printed.
When I now try to exit this shell, I get a warning, that a job is running in the background, which is above command.

Workaround for me is to load the ssh key into ssh-agent, which I usually do anyway, but I have more than one key and some are loaded with an automatic timeout.