Adding Functionality to Data.Conduit.Text

[jhance]

There's a lot to be wishful for in Data.Conduit.Text. Some shortcomings (in my opinion) of the text API make a few of these a bit difficult to implement (see haskell/text#25).

The most evident to me is that there should be a way to sink and source Text in a similar way to ByteStrings. However, you would have to use one of the specific decodings or manually detect the system locale, which is ugly. We could do it that way (which is what the patch, when I finish it hopefully tonight, will do), but it would be preferable to avoid the ByteStrings all together.

I propose the following, in conflict with the Data.Conduit.ByteString API (which makes since given that Data.Text and Data.ByteString also conflict each other):

sourceFile :: MonadResource m => FilePath -> Source m Text
sourceHandle :: MonadResource m => Handle -> Source m Text
sourceIOHandle :: MonadResource m => IO Handle -> Source m Text
sinkFile :: MonadResource m => FilePath -> Sink Text m ()
sinkHandle :: MonadResource m => Handle -> Sink Text m ()
sinkIOHandle :: MonadResource m => IO Handle -> Sink Text m ()
conduitFile :: MonadResource m => FilePath -> Conduit Text m Text
lines :: Monad m => Conduit Text m Text

Notably I left out sourceFileRange because that seems more specialized toward binary data.

(Also, lack of conduitHandle and friends seems odd. Any reason for this?)

Any comments are welcome.

[jhance]

I actually have most of the code written for this now but have to wait for haskell/text#26 so I don't know if I can get this in for 0.4.1.

[snoyberg]

Personally, I never like using the default system encoding, and prefer to have to explicitly convert to/from Unicode. But that's just me. One issue you'll run into for this is implementing sourceFile properly for Windows. We have separate code that doesn't go via a Handle to avoid some file locking issues, and there's no easy way to determine the correct encoding. We could of course just assume UTF-8.

[jhance]

How about a sourceFileWith and such which takes a Codec as a parameter and similar functions for the rest? These would just be trivial wrappers but would be nice for completing the API.

Assuming UTF-8 for Windows on the non-explicit functions sounds fair.

I should be able to send in a patch for the With-family soon since it won't require the patch to text.

[snoyberg]

Sounds good to me, thank you!

[snoyberg]

Any updates here?

[jhance]

Oh, I had completely forgot about this since I went out of state for a week or two and didn't have time to work on it.

I think its mostly done or done but not submitted; I'll check this out later.