PacketEncoder corrupts UTF-8 message

[evgeny-pasynkov]

Hi,

Using SocketIOClient.send("русский"), I receive on client string "Ñ�Ñ�Ñ�Ñ�кий"

I think that the problem is in PacketEncoder lines 272-274:

                            String str = b.toString(CharsetUtil.ISO_8859_1);
                            if (enc.canEncode(str)) {
                                buf.writeBytes(str.getBytes(CharsetUtil.UTF_8));
                            }

the string gets corrupted when converting it with ISO_8859_1 charset, and then it is sent corrupted

[Maypeur]

Hello,
I really don't know if it's a good solution but i used that to make it work in my case :

CharsetEncoder enc = CharsetUtil.UTF_8.newEncoder();
                            String str = b.toString(CharsetUtil.UTF_8);
                            if (enc.canEncode(str)) {
                                buf.writeBytes(str.getBytes(CharsetUtil.UTF_8));
                            } else {
                                buf.writeBytes(b);
                            }

[mrniko]

@evgeny-pasynkov hi!
Do use UTF-8 encoding for your sources?

[evgeny-pasynkov]

@mrniko What do you mean by "UTF-8 for sources"?

Is it default encoding for JVM process? If yes, then it isn't a good solution - I don't want my software to depend on server locale :)

Actually, converting my string to ISO_8859_1 damages it, so client cannot restore it further.

[evgeny-pasynkov]

@Maypeur your solution is tautology :) It is simply equivalent to "buf.writeBytes(b)"

[Maypeur]

@evgeny-pasynkov maybe ! I let it because there was a problem with websocket and accent !

[mrniko]

@evgeny-pasynkov could you purpose a better solution for this problem?

[evgeny-pasynkov]

@mrniko Could you point please to the mentioned websockets bug? Why not to simplify all this stuff to "buf.writeBytes(b)"?

[mrniko]

to avoid encoding problem

[evgeny-pasynkov]

What problems? In the code comment, you've mentioned the websockets bug. Which one?

BTW, socket.io had some UTF-8 encoding problems, they claimed to be fixed in socket.io 1.1.0
Check this topic: socketio/socket.io#1744

[Maypeur]

@evgeny-pasynkov it is for #137 , using new socket.io.client 1.1.0 and socket.io 1.7.3-SNAPSHOT make problem on UTF-8 characters, but with client 1.0.6 and 1.7.3-SNAPSHOT it's working, so now the bug is corrected in client maybe PacketEncoder need to be adapted. Only @mrniko can say what to do now !

[mrniko]

oh! i think to release next 1.7.4 version with this fix so it will be 1.1.0+ compatible only, ok?

[evgeny-pasynkov]

For me it is ok.

[Maypeur]

Since pre 1.1.0 versions have this major bug i think it must !

[mrniko]

please check

[evgeny-pasynkov]

The problem is fixed for my scenarios. Thank you!

[mrniko]

@Maypeur did you happy too?

[Maypeur]

UTF-8 is now correct, but i'm searching why sometimes the memory gow up and never decrease even if i got only 5 users !
Anyway thank you !!!

[mrniko]

@Maypeur take a look at memory dump