Incorrect result for decodeUtf8 of "\194" ByteString with text 1.0

[snoyberg]

Consider the following code:

{-# LANGUAGE OverloadedStrings #-}

import qualified Data.Text.IO as TIO
import Data.Text.Encoding (decodeUtf8)

main :: IO ()
main = TIO.putStrLn $ decodeUtf8 "\194"

The behavior with text pre-1.0, and the expected behavior, is that it throws an exception, in particular:

text-bug.hs: Cannot decode byte '\xc2': Data.Text.Encoding.decodeUtf8: Invalid UTF-8 stream

Beginning with release 1.0, this prints out an empty string. This behavior was initially reported as a bug in conduit's text decoding at snoyberg/conduit#127.

[bos]

This bisects back to aca0971 which was written by @bgamari. Sorry for the regression, I'm looking into it now.

[snoyberg]

Thanks Bryan, let me know if I can be of any assistance.

[bos]

OK, please test the 1.0 branch. It works for me. If it looks good to you, I'll release it.

The fix is in 7c09f3c, and a much improved test case is in 9c11b44. The new test case reveals the regression that was introduced in aca0971, and is fixed by the alleged fix :-)

[snoyberg]

I can confirm that with the 1.0 branch, the conduit test suite passes, so I believe the issue is resolved. Thank you!

[bos]

OK, it's up: http://hackage.haskell.org/package/text-1.0.0.1

A couple of last invalid UTF-8 generators added, too, for good measure, in 494d7d9.

[AnneTheAgile]

Cross ref to blog post; http://www.serpentine.com/blog/2013/12/30/testing-a-utf-8-decoder-with-vigour/ (pet peeve, I wish github showed dates, not 'ago' timestamps... 2013-12-30 is not 2 months ago..)