-
Notifications
You must be signed in to change notification settings - Fork 474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mochiweb_html unicode breakage #54
Comments
The intention is that the input is already UTF-8, whether it is binary or iolist. This is consistent with what you would get by doing something like reading the request body from a mochiweb request
Note the usage of io:fwrite/1 to send the output "directly" to stdout instead of io:format/2 which does the wrong thing. io:format/2 does some really broken shit by default if you are using non-ascii characters.
|
@mattsta it's also worth noting here that you can use
|
I believe that unicode:characters_to_binary(S, utf8) is faster than mochiutf8:codepoints_to_bytes/1 (some functions in the unicode module are BIFs). Different error behavior though. |
Thanks for the clarification. I was passing in a fetched HTML body from httpc:request without first running through unicode/utf8 conversion. Everything works fine as long as I pre-convert:
What can we do to help people not bug you about this in the future? The type system isn't elaborate enough to reject non-utf8 outright. There's always the cheap:
Or maybe even try to fix it ourselves, though it could hide performance issues by double-evaluating on every run:
resolved: dumbmatt |
I'd accept a documentation patch that says the input must be UTF-8. I guess string() is a bit misleading, but I wrote all this stuff before Erlang had any attempt to support unicode (except for some functions hiding in xmerl). |
Example:
Going through
mochiweb_html
and replacing alliolist_to_binary/1
withunicode:characters_to_binary/1
almost works (along with replacing allbinary_to_list/1
withunicode:characters_to_list/1
). It causes the functions to return, but the output is garbled:Any ideas on how to help multi-byte characters survive a
mochiweb_html:parse/1
tomochiweb_html:to_html/1
cycle?The text was updated successfully, but these errors were encountered: