Clarify Binary properties of the streams

[ndmitchell]

The process library is largely silent on whether it gets binary or text handles to the process input/output. The one place where it is explicit is runInteractiveCommand, and there it seems to be wrong (as far as I can tell). I think this confusion has reared it's head as ndmitchell/hoogle#194, commercialhaskell/stack#2582, #59 and #36.

The command ghc-pkg dump is documented by GHC HQ to return UTF8 content, and to a first approximation, let's believe them (I have no reason to doubt it). If we install QuickCheck then the copyright field will include Björn Bringert. In Linux, if I set my $LANG variable to the empty string, then run the code:

import System.Process
import System.IO

main = do
    (inp,out,err,pid) <- runInteractiveCommand "ghc-pkg dump"
    hClose inp
    src <- hGetContents out
    print $ length src

With the command:

runhaskell -package=process-1.4.3.0 ProcessTest.hs

I get the output:

ProcessTest.hs: fd:13: hGetContents: invalid argument (invalid byte sequence)

If I leave the $LANG variable as normal (e.g. en_GB.UTF-8) then it works. That suggests the handles are impacted by the $LANG variable, and thus they aren't really binary. If I modify the code to add:

    hSetBinaryMode out True

Then it works once more.

It would be nice if there was a general statement at the top about whether the handles are in binary format or not, and it would be nice if the statements that were there were accurate.

[snoyberg]

I'd be happy if someone could send a PR with such improved docs. I'd have to ask someone more familiar to review the comments as well, but I think what you're getting at here is true: all functions that get a String or Text out of the Handles will default to the system locale.

[simonmar]

Yes, the Handles created by createProcess (and therefore everything defined in terms of it) are created as text Handles using the default locale encoding.

[ndmitchell]

So runInteractiveCommand has docs that are completely wrong. That comment was introduced in https://ghc.haskell.org/trac/ghc/ticket/2129, so 9 years ago it was in binary mode (confirmed with the code), and that has changed since.

[simonmar]

Yeah, looks like it was me, when I added Unicode support to the IO library: b3eb59a

We should just fix the docs.