UnicodeConversionError on Windows when trying to print unicode strings

[mspanc]

Environment

• Elixir & Erlang versions (elixir -v): Erlang/OTP 19 [erts-8.0] [64-bit] [smp:12:12] [async-threads:10], Elixir 1.3.4

• Operating system: Microsoft Windows [Version 10.0.14393]

Please note that I am using Polish version of windows. Probably default codepage in the shell is set to CP-1250 or something like this.

Current behavior

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. Wszelkie prawa zastrzeżone.

C:\Users\marcin>iex
Eshell V8.0  (abort with ^G)
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> "ł"
** (UnicodeConversionError) invalid encoding starting at <<136, 34, 10>>
    (elixir) lib/string.ex:1748: String.to_charlist/1

Expected behavior

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. Wszelkie prawa zastrzeżone.

C:\Users\marcin>iex
Eshell V8.0  (abort with ^G)
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> "ł"
"ł"

[josevalim]

ł is encoded as <<197, 130>>. The encoding in your terminal seems to be wrong.

Quoting the getting started guide:

Note: if you are running on Windows, there is a chance your terminal does not use UTF-8 by default. You can change the encoding of your current session by running chcp 65001 before entering iex (iex.bat).

http://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html

Is this your case?

[eksperimental]

shoulndn't we be setting this in the line that re-calls iex.bat?

call "%~dp0\elixir.bat" +iex --erl "-user Elixir.IEx.CLI" --no-halt %__ELIXIR_IEX_FLAGS% %*

[mspanc]

@josevalim it is not my case, however running chcp 65001 affects behaviour, no longer exception is raised but now it returns nil.

C:\Users\marcin>chcp 65001
Active code page: 65001

C:\Users\marcin>iex
Eshell V8.0  (abort with ^G)
Interactive Elixir (1.3.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> "ł"

nil
iex(2)> i "ł"

nil
iex(3)> i "a"
Term
  "a"
Data type
  BitString
Byte size
  1
Description
  This is a string: a UTF-8 encoded binary. It's printed surrounded by
  "double quotes" because all UTF-8 encoded codepoints in it are printable.
Raw representation
  <<97>>
Reference modules
  String, :binary
iex(4)>

Moreover, terminal hangs after I press Enter after typing "ł". "nil" shows up after I press Enter for the second time.

When I type "a" it works fine, I mean it prints "a" immediately after first Enter.

[josevalim]

I see. Does it work under iex --werl?

[mspanc]

Yes.

[josevalim]

Ok, I will get a Windows machine and try to reproduce the issue but I suspect it is not Elixir related. I am also not sure I can get my hands on a Windows 10. Does anyone else can reproduce the issue on Windows 10?

[mspanc]

I can set up a VM for you if you want.

[fishcakez]

** (UnicodeConversionError) invalid encoding starting at <<136, 34, 10>>

BTW 34 is " and 10 is \n.

I believe if you do the following we should see the list of bytes that elixir is receiving as the input:

:io.setopts([encoding: :latin1])
String.to_charlist("ł")

The trick is that latin1 is one byte per character, which means that " will end the string (as 34, 10 was input above). We then do the reverse of the incorrect conversion to get the actual bytes. I am unsure if this will help though but it should allow us to see the encoding being used.

[mspanc]

I think it is not only related to what characters elixir gets from the console because when I have perfectly valid utf-8 strings they are also printed as sequence of character codes instead of strings.

[fishcakez]

@mspanc when typing characters that can't be represented as latin1 they will be escaped on output (i.e. valid utf-8) and it should show what we get for "ł" if you just copy and paste.

[mspanc]

I meant that strings totally unrelated to the terminal input are not printed properly too

[fishcakez]

oh okay, for what its worth this is what I get locally with the above:

iex(1)> :io.setopts([encoding: :latin1])
:ok
iex(2)> String.to_charlist("\502")
[322]