"setpgid: No such process" on FreeBSD

[raichoo]

Hi,

I'm getting this error from time to time when I issue a command:

$ pkg info | grep seat
[…]
Could not send own process 1895, 'grep' in job 1,  'command grep --color auto $argv' from group -1 to group 1894
setpgid: No such process

fish, version 3.1.2-1589-g36337fc45
FreeBSD lain 12.2-STABLE FreeBSD 12.2-STABLE r365792 GENERIC  amd64
alacritty 0.6.0-dev (78f6cc4)

Kind regards,
raichoo

[faho]

If I understand you correctly:

• This doesn't happen in 3.1.2

• This only happens sometimes, even with the same commands.

In that case it seems like a race condition in setting up the processes (it's running and has already finished before we give it a pgroup). I'm not entirely sure which path FreeBSD ends up taking in this context (we do sometimes use posix_spawn).

If you could git bisect this, that'd be fantastic. Otherwise I'll have to fire up my FreeBSD vm and see if I can trigger it.

Also @ridiculousfish might know more about this.

[raichoo]

I'm afraid the bug is too unreliable to trigger (yes I'm assuming a race condition), so doing a bisect wouldn't
really help. I don't think I've seen the same happening in 3.1.2.

[mqudsi]

The first command in the job (here, pkg) is exiting before the second command (here, the command grep called by the function grep) finishes its post-fork work and calls exec().

Normally, this isn't an issue: pids stick around until they're reaped. So even after the group leader exits, it remains a zombie allowing subsequent commands in the job to join its group. But we must be triggering the reap midway through the execution of the job; probably something to do with executing the function grep. (I seem to recall a regression that was around for a very short window of time that also involved early reaping, but if I'm not mistaken, a check was added to prevent exactly this.)

And we did introduce some changes to the reaping logic since 3.1.2, so this is probably not barking up the wrong tree.

[faho]

Okay, I managed to reproduce in my FreeBSD vm, here's the debug log:

exec-fork: Fork #17, pid 2880: external command for 'pkg'
Could not send own process 2880, 'pkg' in job 1, 'pkg info | grep seat' from group -1 to group 2880
setpgid: No such process
topic-monitor: Updating topic 1 to 17
exec-fork: Fork #18, pid 2882: external command for 'sleep'
exec-job-exec: Executed job 1 from command 'sleep 1' with pgrp 2880
proc-job-run: Start job 1, gid 2880 (sleep 1), UNCOMPLETED, INTERACTIVE
topic-monitor: TID 1 local  0,17,3 : current 0,17,3
topic-monitor: TID 1 becoming reader
fd-monitor: Thread exiting
iothread: pthread 0x800ab5700 exiting
topic-monitor: TID 1 local 0,17,3 read() complete, current is 0,17,3
topic-monitor: Updating topic 1 to 18
topic-monitor: TID 1 local  0,17,3 : current 0,18,3
proc-reap-external: Reaped external process 'sleep' (pid 2882, status 0)
proc-pgroup: fish reclaiming terminal after job pgid 2880
uvar-file: universal log sync
uvar-file: universal log sync elided based on fast stat()
uvar-file: universal log no modifications
exec-fork: Fork #19, pid 2883: external command for 'grep'
Could not send own process 2883, 'grep' in job 1, 'command grep $argv' from group -1 to group 2880
setpgid: No such process
exec-job-exec: Executed job 1 from command 'command grep $argv' with pgrp 2880
proc-job-run: Start job 1, gid 2880 (command grep $argv), UNCOMPLETED, INTERACTIVE
topic-monitor: Updating topic 1 to 19
proc-reap-external: Reaped external process 'grep' (pid 2883, status 1)
proc-pgroup: fish reclaiming terminal after job pgid 2880
uvar-file: universal log sync
uvar-file: universal log sync elided based on fast stat()
uvar-file: universal log no modifications
exec-job-status: Set status of job 1 (pkg ...) to 256 using short circuit
exec-job-exec: Executed job 1 from command 'pkg info | grep seat' with pgrp 2880
proc-job-run: Start job 1, gid 2880 (pkg info | grep seat), UNCOMPLETED, INTERACTIVE
proc-reap-external: Reaped external process 'pkg' (pid 2880, status 0)
proc-pgroup: fish reclaiming terminal after job pgid 2880

I changed the grep function to include a sleep 1 before the actual grep call in hopes that would trigger it, but in this case we already get the issue with the pkg!

(this was with fish 3.1.2-1621-g61d322d40 on FreeBSD 12.1-RELEASE r354233)

[mqudsi]

FreeBSD doesn't consider zombie processes when setpgid(2) is called, leading to a race between the first process finishing execution and the next process in the pipeline trying to join its pgrp. (This is very reminiscent of the earliest WSL releases!)

I'm not sure why it's also happening when setpgid(2) is invoked by the very first child in the pipeline ("Could not send own process...."), as regardless of the known parent/child setpgid race, the child's call should always succeed (at worst, it is assigned to the same pgrp it is already in).

[mqudsi]

Unlike the old WSL issue, FreeBSD allows processes to join a pgrp consisting of only zombies - it just doesn't allow the parent to setpgid(2) the zombie child and returns ESRCH instead of the EACCES that we expect (and Linux returns). This is handled in e4c0523 by assuming that we are not pulling pids out of a hat and therefore ESRCH indicates a valid-but-dead child rather than a truly non-existent process.

The "own process" is a bug in the calling of report_setpgid_failure, which presumably was only called by the child at one point but is now called by both the parent and the child. The incorrect error message is fixed in 06f1b34.

[raichoo]

Whoop whoop! Thanks a lot.