UTF-8 Encoding?

[amynguyen]

When using UTF-8 encoding to support other languages such as Korean, messages that are placed in the Kafka queue seem to have the incorrect offset. When consuming the UTF-8 message out of the queue for every UTF-8 encoded character 2 bytes are missing off the end. The message looks good in the Kafka logs but when they are consumed that are missing x bytes off the end.

[haio]

Could you please paste some code example here?

[amynguyen]

Sample producer:

var kafka = require('kafka-node'),
Producer = kafka.Producer,
client = new kafka.Client('xxxxxxxxxxx:2181','testing'),
producer = new Producer(client),
payloads = [
{ topic: 'Test_1.0', messages: '{"korean":"추ㅏㅗㅑㅐㄹㄷ"}', partition: 0 }
];
producer.on('ready', function () {
producer.send(payloads, function (err, data) {
console.log(data);
});
})

Results from running the consumer in kafka/bin directory:
{"korean":"추ㅏㅗ

[haio]

I can get the complete JSON from both Kafka-node consumer and the Kafka-console-consumer. Can you get the expected message using Kafka-node consumer?

[amynguyen]

I just update to a newer version of kafka-node and this seems to be fixed.
Sorry.

On Tue, Apr 1, 2014 at 9:15 AM, Zhong Yu notifications@github.com wrote:

I can get the complete JSON from both Kafka-node consumer and the
Kafka-console-consumer. Can you get the expected message using Kafka-node
consumer?

Reply to this email directly or view it on GitHubhttps://github.com//issues/35#issuecomment-39225092
.

[haio]

Good.