Hang when exiting subshell

[dnicolson]

If the following code is run as go run subshell.go, the terminal session in iTerm2 can become unresponsive and cause the fish process to run at 99% CPU.

package main

import "os"
import "os/exec"

func main() {
	cmd := exec.Command(os.Getenv("SHELL"), "--login")
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	cmd.Stdin = os.Stdin
	cmd.Run()
}

1. go run subshell.go
2. Hit Control-C
3. Hit enter
4. Hit enter

This results in the following output before the terminal session becomes unresponsive:

$ go run subshell.go
$ signal: interrupt
$ ⏎
$
$
$

The output of the Terminal app is similar, except the terminal session closes and fish stops running.

Performing the above steps with fish 2.7.1 does not have the same issue.

fish 3.1.2
Darwin Kernel Version 19.5.0: root:xnu-6153.121.1~7/RELEASE_X86_64 x86_64
xterm-256color
iTerm 3.3.11beta1

[mqudsi]

That just launches a bash environment for me and nothing breaks (tested under conhost and kitty).

What is your $SHELL?

[dnicolson]

$SHELL is /usr/local/bin/fish.

[mqudsi]

I was able to reproduce this and then suddenly I wasn't any more. I believe it's a race condition with the terminal owning pgrp (which is the general issue with our pgrp bugs), as I got the following error one of the times that it reproduced for me:

And now that I have all my ducks in a row to be able to debug this, it won't reproduce for me any more. Go figure.

I still had my default $SHELL set to /usr/bin/bash on one of the times that I was able to reproduce it and the bash session that it launched had incorrect terminal flags (it seemed that ECHO was off) but could read and write from the terminal OK (commands were still executed as blindly typed).

[mqudsi]

Ok, I think this one is caused by the fact that fish doesn't forcibly create a new process group with control of the terminal. When you run the go code in your shell, the shell creates a new process group for the go process. go is the only command in this process group and is therefore the process group leader.

When go launches fish, its a naive execution that doesn't do any form of job control. fish is launched into go's process group and inherits all its handles. The go process is still the process group leader, and the process group has full control of the terminal.

When the go process exits (when you hit ^C), the process group is now orphaned (it has no members whose parents are members of another process group in the same session session). Whatever shell you ran that command in (bash, fish, etc) was waiting for go to exit so that it could retake control of the terminal. And retake control of the terminal it does, since it was the parent of that process group leader.

Two different issues are coming into play here:

1. fish doesn't create a new process group for itself on startup (except under some exceptional circumstances, e.g. firejail) so the shell that spawned go is free to regain control of the terminal
2. when go exits and is reaped by its parent and the subshell fish becomes a member of an orphaned process group that no longer has control of the terminal, its call to read/readch from STDIN triggers a SIGTTIN (attempting to read from a tty it does not control). The fish sighandler catches this and basically does nothing fruitful about it, ultimately letting the read call continue after failing (with EIO, I believe) immediately afterwards.

The first point is somewhat deliberate, although it might be worth revisiting given the code cleanup that we've done since that time. The basic gist is that by not creating our own process group, we assume that we are being launched by a shell or something else smart enough to do its own job control. But under that assumption, the right thing to do with a SIGTTIN would be to suspend (SIGSTOP) ourselves until we've been given control of the terminal again (not sure how that would happen apart from being directly fg'd by the user, but that is the assumption). If we didn't override the SIGTTIN or we handled it by cleaning up and then pausing, it would be OK (for some definition of that word): the fish subshell, no longer in control of the terminal and not in the foreground, would stop trying to read from STDIN or write to STDOUT and CPU usage would remain at 0.

But what ends up happening is that we (similarly no longer in control of the terminal and similarly no longer in the foreground) ignore the SIGTTIN, keep trying to read from STDIN, keep getting SIGTTIN, rinse and repeat - i.e. the worst of both worlds.

(If we did create our own process group on launch, we'd still be in control of the terminal although go wouldn't be any more and you can imagine the problems that would cause if the go process tried to read/write to the terminal. It's really a lose-lose.)

[ridiculousfish]

See this blast from the past.

fish has special logic to try to notice when it is orphaned. Perhaps it needs to be updated.

[mqudsi]

Ah yes, that SO post. I remember coming across it many moons ago!

One option, even if not the most elegant, is to just SIGHUP; logically the situation isn’t too different from a terminal window closing on us, with similarly slim-to-no chance of regaining control of the tty. (We wouldn’t do this automatically on SIGTTIN, we’d have to first ascertain we’ve been actually orphaned, otherwise we could have been backgrounded instead - in which case we should probably SIGSTOP until we have control of the terminal again).

[ridiculousfish]

This is pretty amusing:

1. fish launches go launches fish. Call these "fish1" and "fish2". go and fish2 are in the same process group.
2. SIGINT is delivered. go exits. fish2 is interactive, it swallows SIGINT. Now fish2 is orphaned.
3. fish1 sees that go has exited and reclaims the terminal.
4. fish2 soon after decides to exit gracefully - probably it thinks stdin is closed. One of the last things it does is "helpfully" restore the foreground process group to what it initially saw, with is go's orphaned group.
5. This in turn steals the tty from fish1 without fish1 noticing. So we get the SIGTTIN.

The right fixes here are:

1. Probably fish should not restore the foreground process group if it is orphaned. But it's hard to know that for sure - probably kill(0, getpgrp()) and check for ESRCH.

2. Definitely fish should be better about handling SIGTTIN, but it's not obvious how. The usual approach when you get SIGTTIN is to wait until you have ownership of the terminal, but here ownership won't ever arrive.

[mqudsi]

See my comment above about alternative options. We were posting simultaneously.

I think an easy win is on graceful exit to use tcgetpgrp before using tcsetpgrp. If we’re no longer in control of the tty (pgrp doesn’t match ours or we get ENOTTY) then we shouldn’t issue a tcsetpgrp. (I also don’t see how tcsetpgrp can succeed if “fish2” issues it and doesn’t have control of the tty and is merely a process in the same session?)

[ridiculousfish]

I also don’t see how tcsetpgrp can succeed if “fish2” issues it and doesn’t have control of the tty and is merely a process in the same session?

Heh, it's a bit of Unix sillyness. tcsetpgrp is allowed by a non-foreground proc if SIGTTOU is blocked, which fish does. So then that's the right idea here - unblock SIGTTOU and then tcsetpgrp will fail. Clever.

[ridiculousfish]

Fixed as 3ae91f1. Awesome debugging @mqudsi! Process groups sure are "fun".

[ridiculousfish]

Also thanks to @dnicolson for filing.