withCreateProcess async exception safety

[joeyh]

I'm concerned that withCreateProcess may fail to clean up if an async exception is received at just the wrong time. I do not have a test case to prove it, but my reasoning follows.

See Control.Exception's discussion of "Interruptible operations". The masking of async exceptions by bracket when cleanupProcess is running only prevents non-interruptable operations from receiving an async exception. So if any operation in cleanupProcess turns out to be interruptable, an async exception received during that operation will prevent the rest of it from running.

Looking at the operations that get run, there's a takeMVar, but I think the MVar is always full, so it's not interruptable. Then it closes the IO handles, and as far as I can see, hClose is interruptable, or at least is not documented not to be.

So, an async exception received just as hClose is run will terminate the thread, and waitForProcess will never wait on the process. (IO handles could also potentially be left open, although generally GC will auto-close them when they go out of scope.) It's probably a rather narrow race condition.

My analysis could be wrong, but eh, if I've been staring at this code for an hour and I don't know if it it's correct, it seems like it would be a good idea to make sure this doesn't happen by using uninterruptibleMask somewhere?

(The closest I have come to a test case is modifying cleanupProcess to add threadDelay 100, and then it's easy to time the async exception to interrupt it, leaving the process running. But of course it might be threadDelay itself that's the interruptable operation in that case.)

[joeyh]

hClose is indeed interruptable, see the comment here
https://hackage.haskell.org/package/base-4.14.0.0/docs/src/GHC.IO.Handle.Internals.html#hClose_help

An alternative to uninterruptibleMask would be to work off a worker thread to do all the closing etc, and wait on it before returning. That insulates it from receiving async exceptions, and won't prevent the program from being interrupted by ctrl-c while it's running.

[snoyberg]

This topic is an ongoing debate in the Haskell community. I agree with uninterruptible being the right behavior for cleanups, and wish bracket worked that way. I'm not sure if it's a good idea to change an existing function in a core library with a difficult-to-notice semantics change, however.

[nh2]

Investigation of "acquire" part of bracket

Slightly tangential to this, I looked into whether the "acquire" part of bracket in withCreateProcess_ uses Interruptible Operations.

Details

Call chain starting from callProcess in process-1.6.26.1:

withCreateProcess_ fun c action =
    C.bracketOnError (createProcess_ fun c) cleanupProcess  -- calls `createProcess_`

createProcess_ msg proc_ = unwrapHandles `fmap` createProcess_Internal msg proc_ -- calls `createProcess_Internal`

The createProcess_Internal code looks correct to me so far, at least I couldn't spot any interruptible operations after the call to c_runInteractiveProcess.

But I think that funciton, and createProcess_, should both gain Haddocks that say This function does not use Interruptible Operations (link) after the process is launched, so when used in "Control.Exception.bracket", the "release" function is guaranteed to run.

Downstream streaming-process use

The same invariant then needs to be added to streamingProcess_ from streaming-commons, as it, too, is used in bracket "acquire"s, and it calls createProcess_. However, streamingProcess_ cannot just state this, because it is not under its sole control: It does

    let (getStdin, stdinStream) = isStdStream

    createProcess_ ...

    (,,,)
      A.<$> getStdin stdinH
      A.<*> getStdout stdoutH
        <*> getStderr stderrH
        <*> return (StreamingProcessHandle ph ec close)

at the end, where the functions getStdin are arbitrary IO actions from typeclass InputSource that the caller can supply ( isStdStream :: (Maybe Handle -> IO a, Maybe StdStream)).

So InputSource and OutputSink should really also have Haddocks that say that these must not use Interruptible Operations.

However, a 1 of the 4 provided instances of those, ClosedStream, already violates this requirement, as it calls hClose, which was found interruptible above.

instance InputSource ClosedStream where
    isStdStream = (\(Just h) -> hClose h >> return ClosedStream, Just CreatePipe)

instance OutputSink ClosedStream where
    osStdStream = (\(Just h) -> hClose h >> return ClosedStream, Just CreatePipe)

So this suggests that streaming-commons can leak its subprocess under async exceptions already in the "acquire" part of bracket.

---

(I know this is the process issue tracker, not the one for streaming-commons, but posting this info here anyway because it's a common way to call process, e.g. conduit-extra calls streamingProcess in 3 places.)

typed-process has a similar setup which I haven't fully investigated yet (and there's also that fpco/typed-process#38).