Unicode support

[emmettbutler]

We decided when implementing python3 that unicode support wasn't worth it, but I think it's time now. I'm putting this issue here to track progress on unicode support specifically for message contents in consumers and producers across both python 2 and 3. Supporting unicode for other things like topic names is less important, but there are a lot of cases where users want to send unicode in messages.

[yungchin]

It seems our hands are somewhat tied here: if we would accept and encode unicode payloads, afaict there's no field in the kafka message to convey what encoding was used? So on the one hand I also feel that it's terribly clunky that users currently have to encode to a bytes object before producing, and have to decode what they get from consume(), but on the other hand this just expresses the fact that the kafka protocol is ignorant about encodings.

As for topic and group names: it would be quite doable to start accepting a py3 str for those - all we need to check is that all the characters are in the allowed subset (it's a very small set) and we could throw a custom error if not.

[emmettbutler]

You're right about that - there's no way to get information about the message encoding if pykafka is consuming from a producer that's not pykafka. In that case I think there's not much to do. However, in the case where both sides are pykafka, what we can do is have the producer put some field in the message payload indicating the encoding and have the consumer check for that field and decode if necessary. The trick would be to figure out a way to indicate the encoding that would likely never be included in random messages in the wild. Maybe a pattern of 0x000x01's followed by the encoding string.

I actually like the py3 str idea for topic and group names. It's just more convenient for py3 users, and there isn't really a downside if we raise a custom exception for disallowed characters.

[johnistan]

The idea of padding a message payload gives me bad feelings. It reduces interoperability between different consumers in kafka cluster and strikes me as too magical. Another approach is to give hooks for users to register SerDe's. This is the approach samza takes and it gives maximum flexibility without touching the bytes in kafka.

Basically you could register a de/serializer for you producer/consumer.

def my_custom_serializer(msg):
    """"Transforms all msg fields into raw bytes"""
    msg.value = json.dumps(msg.value).encode()

def my_custom_deserializer(msg):
    """"Transforms msg fields into python objects"""
    msg.value = json.loads(msg.value.decode())


topic.get_producer(deserializer=my_custom_serializer)
topic.get_consumer(serializer=my_custom_deserializer)

I like the py3 string idea for topics and group names. Right now the byte casting in zookeeper creates some fud that could be cleanedup.

Here is screen shot from a zookeeper cluster:

[johnistan]

Now you could separate the value and key serde. something like

also you could use a lambda for simple stuff

p = topic.get_producer(value_serializer=lambda v: v.encode(), key_serializer=lambda k: k.encode())
p.produce('unicode fun!', 'also unicode key')

Here is the link to the samza docs:

https://samza.apache.org/learn/documentation/0.7.0/container/serialization.html

[ottomata]

+1 about not padding messages. It also gives me bad feelings. :)

On Wed, Jan 20, 2016 at 7:06 PM, John Dennison notifications@github.com
wrote:

The idea of padding a message payload gives me bad feelings. It reduces
interoperability between different consumers in kafka cluster and strikes
me as magical. Another approach is to give hooks for users to register
SerDe's. This is the approach samza takes and it gives maximum flexibility
without touching the bytes in kafka.

Basically you could register a de/serializer for you producer/consumer.

def my_custom_serializer(msg):
""""Transforms all msg fields into raw bytes"""
msg.value = json.dumps(msg.value).encode()

def my_custom_deserializer(msg):
""""Transforms msg fields into python objects"""
msg.value = json.loads(msg.value.decode())

topic.get_producer(deserializer=my_custom_serializer)
topic.get_consumer(serializer=my_custom_deserializer)

I like the py3 string idea for topics and group names. Right now the byte
casting in zookeeper creates some fud that could be cleanup.

Here is screen shot from a zookeeper cluster:

[image: image]
https://cloud.githubusercontent.com/assets/800669/12467391/1b908738-bf98-11e5-8c08-4e0706b1da5d.png

—
Reply to this email directly or view it on GitHub
#405 (comment).

[yungchin]

Regarding SerDes: what's also neat is that they would allow (optionally) integrating with http://docs.confluent.io/2.0.0/schema-registry/docs/intro.html

[steffann]

I just started using pyKafka and ran into the topic encoding "surprise". Some thoughts:

For topic and group names it shouldn't be a problem: all acceptable characters are in the ASCII range so converting between str and bytes is easy.

For sending messages it's also not that hard: we could default to using UTF-8 encoding to convert to bytes when a str is sent (on Py3, on Py2 we can do the same for unicode). That would probably work for most users, and users that want control can send bytes. Sending str is then just a convenience.

The receiving side is harder: we could try to decode UTF-8, but then some messages might accidentally be valid UTF-8 and the user won't be able to rely on what they are getting. Wanting to receive bytes and then suddenly getting a str because the bytes happen to be valid UTF-8...

And I think we should keep the API symmetrical. If we can't automatically determine the right thing to do on the receiving side we shouldn't do 'magic' stuff on the sending side. Otherwise users will be surprised when they send strings but receive bytes.

And then we end up at jofusa's suggestion :-)

[emmettbutler]

Thanks for the comment @steffann. This is a prime candidate for a community contribution, especially from anyone on this thread who is interested in starting a PR for this work.