Reset signal behavior before starting children with std::process

[geofft]

UNIX specifies that signal dispositions and masks get inherited to child processes, but in general, programs are not very robust to being started with non-default signal dispositions or to signals being blocked. For example, libstd sets SIGPIPE to be ignored, on the grounds that Rust code using libstd will get the EPIPE errno and handle it correctly. But shell pipelines are built around the assumption that SIGPIPE will have its default behavior of killing the process, so that things like head work:

geofft@titan:/tmp$ for i in `seq 1 20`; do echo "$i"; done | head -1
1
geofft@titan:/tmp$ cat bash.rs
fn main() {
        std::process::Command::new("bash").status();
}
geofft@titan:/tmp$ ./bash
geofft@titan:/tmp$ for i in `seq 1 20`; do echo "$i"; done | head -1
1
bash: echo: write error: Broken pipe
bash: echo: write error: Broken pipe
bash: echo: write error: Broken pipe
bash: echo: write error: Broken pipe
bash: echo: write error: Broken pipe
[...]

Here, head is supposed to terminate the input process quietly, but the bash subshell has inherited the ignored disposition of SIGPIPE from its Rust grandparent process. So it gets a bunch of EPIPEs that it doesn't know what to do with, and treats it as a generic, transient error. You can see similar behavior with find / | head, yes | head, etc.

This PR resets Rust's SIGPIPE handler, as well as any signal mask that may have been set, before spawning a child. Setting a signal mask, and then using a dedicated thread or something like signalfd to dequeue signals, is one of two reasonable ways for a library to process signals. See carllerche/mio#16 for more discussion about this approach to signal handling and why it needs a change to std::process. The other approach is for the library to set a signal-handling function (signal() / sigaction()): in that case, dispositions are reset to the default behavior on exec (since the function pointer isn't valid across exec), so we don't have to care about that here.

As part of this PR, I noticed that we had two somewhat-overlapping sets of bindings to signal functionality in libstd. One dated to old-IO and probably the old runtime, and was mostly unused. The other is currently used by stack_overflow.rs. I consolidated the two bindings into one set, and double-checked them by hand against all supported platforms' headers. This probably means it's safe to enable stack_overflow.rs on more targets, but I'm not including such a change in this PR.

r? @alexcrichton
cc @Zoxc for changes to stack_overflow.rs

[alexcrichton]

Oh dear, thanks for the eagle eyes here! Glad to see this duplication reduced.

[alexcrichton]

Aha! So back in 33a2191 I was cleaning out some old code in the unix process spawning code, and I removed a call to rust_unset_sigprocmask because there were no comments as to why it existed and I also had no idea why it existed. I think that this explains it, however!

To clarify, however, I've got a few questions:

• The old implementation used sigprocmask instead of pthread_sigmask, is there a reason that you opted to use the pthread version here instead? At this point there's only one thread, so there shouldn't be any multithreading worries.
• The old implementation also did nothing with SIGPIPE or other signals, so I'm somewhat wary here of explicitly setting it back to the default behavior. I can see how because our runtime explicitly resets this away from the default that it may be the only one we explicitly reset back, however, so I think I'm OK with that.

[alexcrichton]

Also, according to the execve manpage:

Signals set to be ignored in the calling process are set to be ignored
in the new process. Signals which are set to be caught in the calling
process image are set to default action in the new process image.
Blocked signals remain blocked regardless of changes to the signal
action. The signal stack is reset to be undefined (see sigaction(2)
for more information).

Could you clarify in this comment that specifically the ignore behavior of SIGPIPE is the only part that's inherited? Any registered handlers will be reset and we're manually unmasking signals, so the only thing that's inherited is ignored signals.

[geofft]

The old implementation used sigprocmask instead of pthread_sigmask

Yeah, either is fine here. pthread_sigmask is a strict superset of sigprocmask: it's defined to do the same thing if the process is single-threaded, and has well-defined behavior if the process is multithreaded, so I've just been making myself forget that sigprocmask exists. It's also one fewer thing to bind, which also reduces the temptation for a future commit to call it in a maybe-multithreaded context (e.g., if you wanted to mask signals around the call to fork, as libc system does). I guess there's an argument that it requires -lpthread, so if we expect to have UNIXish ports where we aren't using pthreads (???) we'd prefer sigprocmask, plus a warning comment about threads.

The old implementation also did nothing with SIGPIPE or other signals

Yes, I think it's libstd's responsibility to unset any ignored-signal dispositions it sets, if that disposition is for internal use. SIGPIPE is the only one that's ignored; the other two signals libstd touches get handlers. (As it happens, I think there's also an argument that SIGPIPE is the only one it ever makes sense to ignore-but-not-inherit, since it's the only one with a corresponding EPIPE. If you're ignoring stuff like SIGHUP, you probably want the rest of your process group to ignore them too.)

Ignoring SIGPIPE is relatively recent (#13158, March 2014). The old sigprocmask code is much older (ea5dc54, March 2011), and Rust was not setting any signal handlers at that time.

Could you clarify in this comment

Just the antecedent of "both of these"? I intend it as saying "SIGPIPE being ignored, and any signal mask", but I can rework that comment to be clearer, sure.

[alexcrichton]

pthread_sigmask is a strict superset of sigprocmask: it's defined to do the same thing if the process is single-threaded, and has well-defined behavior if the process is multithreaded, so I've just been making myself forget that sigprocmask exists. It's also one fewer thing to bind, which also reduces the temptation for a future commit to call it in a maybe-multithreaded context (e.g., if you wanted to mask signals around the call to fork, as libc system does).

Sounds good to me

Just the antecedent of "both of these"?

Ah I think I just didn't reread after learning more about what's going on here, I think it's fine.

---

So just to check I poked around at some other libraries to see what they do. In libuv at least, nothing with signals is handled in terms of forking, but they also don't by default disable SIGPIPE. I couldn't make heads or tails of what Ruby does, but it didn't look like Python did much with signal masks or forking despite ignoring SIGPIPE like we are.

Given this information, it strikes me as somewhat odd that we're resetting the signal mask? Shouldn't that be up to the application instead of the library here instead? For example this makes it so it's impossible to spawn a process with a signal mask in place through the Command API. This info also makes me uneasy about ignoring SIGPIPE by default, but I think that setting it to SIG_DFL here is fine.

[alexcrichton]

This is some fascinating investigation @geofft, very nice work!

[geofft]

So just to check I poked around at some other libraries to see what they do. In libuv at least, nothing with signals is handled in terms of forking, but they also don't by default disable SIGPIPE

libuv appears to be processing signals via signal handlers. I can look in more detail if you think it's interesting, but if they're doing handlers instead of a thread and asking library users to ignore SIGPIPE themselves, strictly speaking none of this is their responsibility (and I'm also a little unsure if they've thought through their signals-and-subprocesses story enough to be a worthwhile reference). Rust libstd ignores SIGPIPE, so it incurs that responsibility, and while there's no signal-handling API in libstd, mio (for instance) wants to avoid handler functions to avoid EINTR problems. But without cooperation from libstd, unintentionally masking all signals in children is a bad plan.

I couldn't make heads or tails of what Ruby does

Seems to get this right:

howe-and-ser-moving:~ geofft$ ruby -e 'open("/proc/self/status").each_line { |line| puts line if line.include? "SigIgn" }'
SigIgn: 0000000000001000
howe-and-ser-moving:~ geofft$ ruby -e 'puts `grep SigIgn /proc/self/status`'
SigIgn: 0000000000000000

but it didn't look like Python did much with signal masks or forking despite ignoring SIGPIPE like we are.

Python 3 (at least 3.4) appears to have fixed this relative to Python 2.7:

howe-and-ser-moving:~ geofft$ python2 -c 'print [line for line in open("/proc/self/status") if "SigIgn" in line][0]'
SigIgn: 0000000001001000

howe-and-ser-moving:~ geofft$ python2 -c 'import subprocess; print subprocess.check_output(["grep", "SigIgn", "/proc/self/status"])'
SigIgn: 0000000001001000

howe-and-ser-moving:~ geofft$ python3 -c 'print([line for line in open("/proc/self/status") if "SigIgn" in line][0])'
SigIgn: 0000000001001000

howe-and-ser-moving:~ geofft$ python3 -c 'import subprocess, sys; sys.stdout.buffer.write(subprocess.check_output(["grep", "SigIgn", "/proc/self/status"]))'
SigIgn: 0000000000000000

Shouldn't that be up to the application instead of the library here instead? For example this makes it so it's impossible to spawn a process with a signal mask in place through the Command API. This info also makes me uneasy about ignoring SIGPIPE by default, but I think that setting it to SIG_DFL here is fine.

Yeah, I do actually mostly agree: there is definitely a use case for intentionally starting a child with ignored signals, e.g., writing nohup in Rust. (I'm less convinced there's a use case for starting a child with a mask, but posix_spawn supports it, so whatever.) What would be nice is a method on the Command object to change signal handling, just like the methods to change standard I/O inheritance -- and a static method that allows libraries to reset signal behaviors (or perhaps run a general-purpose unsafe fn to clean things up) post-fork. But I don't think we need that immediately. I was planning on non-urgently sending in a PR or writing a third-party crate to add some methods to do these sorts of things, but this current PR fixes the immediate problem with SIGPIPE, and also enables libraries like mio to turn on signal masks safely.

If you're worried about the regression possibility with signal masks (although the docs don't promise anything either way, and I can't imagine any users will ever care, let alone have cared), we could set things up to reset SIGPIPE, and also offer a static method that an external library like mio can call to undo its signal mask. That would also solve my immediate problem, although it's more complexity and unsafe code surface, and I don't think it gains us anything in practice. If you're just worried about the ability to set signal masks one way or another, I can get you a separate PR for that, though probably not immediately.

[alexcrichton]

Yeah I think the change here to reset SIGPIPE back to SIG_DFL is definitely the right way to go (e.g. we differ from libuv here). My only pause now is whether to reset the sigmask on a spawn. For example the test here does not require the procmask to be reset, and nor does anything else in the standard library.

Perhaps a test could be added at least which requires procmask to be reset on spawn? I think it's fine to do, but it'd be nice to have a regression test to ensure it's not accidentally removed as well!

[geofft]

OK, I've added a #[test] to process.rs (so I can use all the internal bindings) that masks SIGINT, spawns cat, writes something, and makes sure that cat dies instead of replying. If I remove the five lines in that file to clear the signal mask, it fails make check-stage1-std, so it's appropriately testing for this.

Is this ready for merge? Are you happy with the changes to the bindings?

[alexcrichton]

Looks good to me, thanks @geofft! Can you also tag the test with #[cfg(unix)]? Other than that through r=me.

[geofft]

Doesn't its presence in libstd/sys/unix/ make that implicit, since that path is only brought in by libstd/lib.rs's #[cfg(unix)] #[path = "sys/unix/mod.rs"] mod sys;? Like, code that references libc::fork is not really going to compile on Windows.

[alexcrichton]

Welp, I got this.

@bors: r+ 8e2e0ee

[bors]

⌛️ Testing commit 8e2e0ee with merge 52f12c9...

[bors]

💔 Test failed - auto-linux-64-x-android-t

[geofft]

Oops, fixed. Can I test-build Android without having a local Android dev environment? make check-stage1-T-arm-linux-androideabi-H-x86_64-unknown-linux-gnu-std doesn't seem to like me.

[alexcrichton]

@bors: r+ 824a928

You'll need to pass --target=arm-linux-androideabi to ./configure to enable a target like that.

[bors]

⌛️ Testing commit 824a928 with merge 2b457ad...

[bors]

💔 Test failed - auto-linux-64-x-android-t

[geofft]

Bah, I think I should go and set up build toolchains for all platforms. A #![cfg_attr(target_os = "linux", allow(dead_code))](or just reintroducing #![allow(dead_code)]...) will probably fix the immediate problem, but maybe you want a nicer fix, and maybe something's broken on Bitrig or whatever.

[alexcrichton]

Ah feel free to throw up just a blanket #![allow(dead_code)] on c.rs, we're not that worried about dead code in that module anyway.

[alexcrichton]

@bors: r+ 9a676f1

[bstrie]

This probably means it's safe to enable stack_overflow.rs on more targets, but I'm not including such a change in this PR.

Can you file a bug for this so we remember to look at it later?

[bors]

⌛️ Testing commit 9a676f1 with merge 623072d...

[bors]

💔 Test failed - auto-linux-64-x-android-t

[geofft]

OK, I've gotten this passing against Android API level 18 locally, and rebased against current HEAD. The big problem was that, until Android API level 21 (introduced in Android 5.0), several of the signal-handling functions were only exposed as C inline functions: see Android 4.4's <signal.h>. So we need to translate those to Rust ourselves. I've kept that as a separate commit to make it easy to revert once the API level is bumped, but that's unlikely to be a good idea for a while. This feels ugly and I'm open to style suggestions, but I don't think it'll ever be non-ugly.

Also, my run-pass test case wasn't working on Android for lack of either yes or head, but it turns out that Android's dynamic linker unconditionally sets a SIGPIPE handler on all processes (to tell debuggerd to take a crash dump), so the question of SIGPIPE inheritance is completely irrelevant on Android. I suppose this makes sense, given that Android is primarily a mobile platform, not a UNIX, and if your app dies with SIGPIPE you'd want to know. So I disabled the test on Android. It mostly exists to catch changes in std::process's semantics, so it doesn't need to run on all platforms.

[alexcrichton]

@bors: r+ 2a93dca

Thanks!

[bors]

🔒 Merge conflict

[bors]

☔️ The latest upstream changes (presumably #25641) made this pull request unmergeable. Please resolve the merge conflicts.

[sfackler]

@bors r=alexcrichton

[bors]

📌 Commit 3a3d864 has been approved by alexcrichton

[bors]

⌛️ Testing commit 3a3d864 with merge 00b862b...

[bors]

💔 Test failed - auto-linux-64-x-android-t

[geofft]

Hrrm. I don't think this test failure is related to my code, despite being on Android again.

rustc: x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd
../src/libstd/lib.rs:152:21: 152:35 error: unused or unknown feature, #[deny(unused_features)] on by default
../src/libstd/lib.rs:152             feature(num_bits_bytes))]
                                             ^~~~~~~~~~~~~~
error: aborting due to previous error

Seems related to #26192, but the Android tests passed there, so I'm confused.

[alexcrichton]

Could this use as foo instead of mem::transmute? The transmute may be a bit of a heavy hammer for this operation.

[geofft]

Transmute checks size, and SIG_ERR is ~0. Really the problem is that liblibc and c.rs have different sighandler_t types. c.rs uses *mut c_void since that's what we need for struct sigaction (well, really a function pointer); liblibc calls it a size_t, which is probably the same but I didn't want to assume that. Ideally we should synchronize these.

(Actually what I really want is an extended nullable-pointer optimization where I can say that 0, 1, and ~0 are invalid and force an enum {SIG_DFL = 0, SIG_IGN = 1, SIG_ERR = ~0, handler(extern fn(c_int)) to be represented as a single pointer, but I assume that's super hard.)

[alexcrichton]

Wow that definitely sounds like a mess, this already landed so it's fine for now, but I may do a drive-by fix at some point to just cast these both to a usize and compare that (not critical at all though)

[alexcrichton]

Ah unfortunately that constant is unstable currently, and it just looks like you deleted the usage of it, so you may be able to delete the associated cfg_attr entirely.

[sfackler]

@bors r=alexcrichton

[bors]

📌 Commit a8dbb92 has been approved by alexcrichton

[bors]

⌛️ Testing commit a8dbb92 with merge 4e2a898...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt