Some non ascii are not parsed corectly

[silviucpp]

Try to parse the following file:

https://github.com/ua-parser/uap-core/blob/master/tests/test_device.yaml

if you do print user_agent_string for each row when you will hit:

SkyNet/1.5.1-0000(android:4.0.3;package:com.halfbrick.fruitninjafree;lang:zh_CN;app_version:null;channel:GF0S0N00000;device_brand:htccn_chs_cu;device_model:HTC T328w;resolution:480X800;udid:ffffffff-8d34-e60d-ffff-ffff92fede89;cpu_freq:1008000;google_account:null;phone_number:unknown;game_name:水果忍者;encoded:true;sdk_version:1.5.1;imei:353614053116514;location:unknown)

The Chinese chars are messed up. and list_to_binary fails as well

Silviu

[dumbbell]

Hi!

Sorry for taking all that time to get back to you...

This must be a problem when trying to print the result in an Erlang; I get the same problem:

$ cat /tmp/example.erl
"水果忍者".

$ erl -pa /path/to/yamerl/ebin
...
1> file:consult("/tmp/example.erl").
{ok,[[27700,26524,24525,32773]]}
2> unicode:characters_to_binary([27700,26524,24525,32773]).
<<230,176,180,230,158,156,229,191,141,232,128,133>>
3> io:format("~s~n", [unicode:characters_to_binary([27700,26524,24525,32773])]).
æ°´æ��å¿
ok

However, when using a normal program or a simple erl -eval, it behaves as expected:

$ cat /tmp/example.yaml 
'SkyNet/1.5.1-0000(android:4.0.3;package:com.halfbrick.fruitninjafree;lang:zh_CN;app_version:null;channel:GF0S0N00000;device_brand:htccn_chs_cu;device_model:HTC T328w;resolution:480X800;udid:ffffffff-8d34-e60d-ffff-ffff92fede89;cpu_freq:1008000;google_account:null;phone_number:unknown;game_name:水果忍者;encoded:true;sdk_version:1.5.1;imei:353614053116514;location:unknown)'

$ erl -noinput -A0 -pa /path/to/yamerl/ebin -eval 'application:start(yamerl), [String] = yamerl_constr:file("/tmp/bla.yaml"), Bin = unicode:characters_to_binary(String), io:format("~s~n", [Bin]), halt(0).'
SkyNet/1.5.1-0000(android:4.0.3;package:com.halfbrick.fruitninjafree;lang:zh_CN;app_version:null;channel:GF0S0N00000;device_brand:htccn_chs_cu;device_model:HTC T328w;resolution:480X800;udid:ffffffff-8d34-e60d-ffff-ffff92fede89;cpu_freq:1008000;google_account:null;phone_number:unknown;game_name:水果忍者;encoded:true;sdk_version:1.5.1;imei:353614053116514;location:unknown)

[silviucpp]

Pretty strange because I didn't tried to print the string when I discovered the problem. I found out that a pattern matching fails..

[dumbbell]

Could you please share a code snippet demonstrating that failing pattern matching?

[silviucpp]

My bad.. I had to use unicode:characters_to_binary instead list_to_binary

[dumbbell]

No problem, thank you for nailing it down!