Revert "force message buffers to strings before decoding" #287

merged 1 commit into from Feb 12, 2014

2 participants


This reverts commit 7c47c36.


Anuj Mathur Revert "force message buffers to strings before decoding"
This reverts commit 7c47c36.

@anujm anujm merged commit 0d292ad into rackerlabs:master Feb 12, 2014

what was wrong with that?


For some reason the following piece of code was preventing the messages from being processed:

register('bufferjson', self.loads, anyjson.dumps,

    def loads(s):
        return anyjson.loads(BytesIO(s))

The messages were being read from the queue, but the callback(on_nova) was never being invoked. Did this work during your testing?


Strange - yes, this worked (and still works) for me.

I suspect decoding fails. That is precisely the reason I wrote #281 - no idea why this has not been merged yet. When decoding fails, without #281 messages are silently dropped.

In my case it was the otherway around: With libjson2 installed, decoding failed because libjson2 cannot handle buffers and the conversion code in anyjson did not work for some reason.

Could you try applying both #283 and #281 and see if any decoding errors are logged?


As suspected, it's an issue with decoding the json:

2014-02-12 18:29:19,911 - child_worker - ERROR - Decode Error: is not JSON serializable
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/kombu/", line 583, in receive_callback
decoded = None if on_m else message.decode()
File "/Library/Python/2.7/site-packages/kombu/", line 123, in decode
self.content_encoding, accept=self.accept)
File "/Library/Python/2.7/site-packages/kombu/", line 165, in loads
return decode(data)
File "/Library/Python/2.7/site-packages/anyjson/__init_
.py", line 141, in dumps
return implementation.dumps(value)
File "/Library/Python/2.7/site-packages/anyjson/", line 87, in dumps
return self.encode(data)
File "/Library/Python/2.7/site-packages/simplejson/__init_
.py", line 354, in dumps
return _default_encoder.encode(obj)
File "/Library/Python/2.7/site-packages/simplejson/", line 262, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/Library/Python/2.7/site-packages/simplejson/", line 340, in iterencode
return _iterencode(o, 0)
File "/Library/Python/2.7/site-packages/simplejson/", line 239, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: is not JSON serializable

Any ideas? The payload is definitely json serializable without #283.


Thanks for trying it out. I should have tested with more json libs. simplejson seems to behave differently.
I'll take a look later to come up with a better patch.

@bkw bkw referenced this pull request Feb 28, 2014

log decode errors, do not ack. #281

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment