Issue with unicode parsing

[onionhammer]

When I connect to leguin.freenode.net, The MOTD is coming back with

Welcome to leguin.freenode.net in Umeå, Sweden, EU! Thanks to

where the å is 229, but is instead parsed as σ, resulting in:

Welcome to leguin.freenode.net in Umeσ, Sweden, EU! Thanks to

Relevant gist:

https://gist.github.com/onionhammer/7494336

And then crashing when I serialize the message, an assert fails:

Traceback (most recent call last)

marshal.nim(239) $$

marshal.nim(57) storeAny

marshal.nim(48) storeAny

marshal.nim(88) storeAny

unicode.nim(63) escapeJson

system.nim(2549) hiddenRaiseAssert

system.nim(1836) raiseAssert

Error: unhandled exception: ord(s[i + 1]) shr 6 == 0b00000000000000000000000000000010 [EAssertionFailed]

[Varriount]

I'd have to guess that this is because, although strings can store Unicode characters, string slices still go by ASCII slices - slices use char.

[dom96]

I can't reproduce it locally but this is the bug reponsible for crashing NimBot. What commit of the Nimrod compiler are you on?

[onionhammer]

I'm using the latest compiler from github, on windows 8.1.

You couldn't reproduce with the gist?

[onionhammer]

Actually.. now I cant reproduce the crash lol. It worked until I uploaded it to gist. I'll get the crash "working" again.

[onionhammer]

Okay, it looks like the act of copy/pasting/uploading to gist removed what was actually wrong with the text, try the output.dat zipped into this file:

https://dl.dropboxusercontent.com/u/417554/output.zip

[dom96]

The problem seems to be the fact that the file is not UTF-8 encoded:

» file output.dat
output.dat: ISO-8859 text, with no line terminators

[onionhammer]

Without those asserts on unicode.nim:63,64, it actually works as expected;

- Welcome to leguin.freenode.net in Umeå, Sweden, EU! Thanks to

"- Welcome to leguin.freenode.net in Ume\u5B20Sweden, EU! Thanks to"

[reactormonk]

Can you make it a warning?