Consider providing serialization-aware consumer API #6

Open
michaelklishin opened this Issue Apr 19, 2012 · 11 comments

Projects

None yet

4 participants

@michaelklishin

Many scenarios in message consumer applications are very common. For example, JSON or MsgPack used for serialization. We should investigate a convenient way to plug common serialization formats into consumer API. Then payload attribute can be a data structure instead of a bag of bytes.

@carlhoerberg

yes, and clojure data structure (de)serialization:

(defn serialize [data]
  (binding [*print-dup* true] (pr-str data)))

(defn deserialize [str]
  (with-in-str str (read))

suggested content-type: application/clojure

@carlhoerberg

https://github.com/mefesto/wabbitmq has generic serialization support, could be worth checking out..

@michaelklishin

We probably can take a page from Welle which in turn steals this idea from Sumo. I don't want this to replace langohr.consumers/subscribe, though, just be an useful option to choose from.

@carlhoerberg

maybe even include content-encoding, eg. automatic gzip (de)compression of payloads..

@mlimotte

I've written my own abstraction on top that uses a ring style to attach additional behaviors like this. So I have a wrap-decode-payload wrapper that support JSON or EDN based on content-type. Similarly, I have a wrap-encode-payload for going the other way and several other wrappers for other purposes. Gzip could be supported the same way. This pattern seems to work pretty well.

@michaelklishin

@mlimotte yes, I was contemplating doing just that.

@danielsz

Has the release of transit any incidence over this discussion?

@michaelklishin

Transit doesn't change anything for us. We can't support a single transport only.

@danielsz

OK. Thanks.

Just to be clear: is it up to everyone to build their abstractions on top of langohr for serialization/deserialization, or are there features/best practices/examples that can help when starting out?

@michaelklishin

Well, if you can deserialize a byte array in your consumer, you are all set today.

We can provide a protocol and a deserializing consumer function that will do it for you and call your handler with a Clojure data structure, that's pretty much it.

@danielsz

Thanks, Michael, I'll take the first suggestion for now.

Coming from Ruby, I'm always surprised when there's no native data structures straight off the box.
Not a big deal, but it's a philosophy thing that puzzles me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment