Unicode handling

[headprogrammingczar]

The following line fails with the error "*** Exception: More data - lexer failed.". Note the single-quote that is actually codepoint 180, acute accent.

> parseLazyByteString JSP.string (BS.pack "\"asdfasdf bob´s burgers\"")
*** Exception: More data - lexer failed.
-- compare with
> parseLazyByteString JSP.string (BS.pack "\"asdfasdf bob's burgers\"")
["asdfasdf bob's burgers"]

This error is fixed by doing the following:

escapeUTF8 [] = []
escapeUTF8 (c:str) = case ord c of
  x | x < 128 -> c : (escapeUTF8 str)
    | otherwise -> '\\' : 'u' : (printf "%04x" x ++ escapeUTF8 str)

> parseLazyByteString JSP.string (BS.pack . escapeUTF8 $ "\"asdfasdf bob´s burgers\"")
["asdfasdf bob\180s burgers"]

I've briefly looked through the source, but with my limited familiarity I don't see any obvious place for the internal parsing to have fallen down on this input.

[ondrap]

You cannot use BS.pack for utf8 encoding. What BS.pack does is treat the string basically as Latin-1. Use Data.Text.Encoding.encodeUtf8 instead and it will work.

[headprogrammingczar]

Oh, that was silly of me. I had it right with the Aeson parser that had poor space behavior, but then must have refactored improperly.

[ondrap]

No, problem, this exact error happened to me too :)