Revert "force message buffers to strings before decoding" #287

Merged
merged 1 commit into from Feb 12, 2014

2 participants

@anujm

This reverts commit 7c47c36.

Conflicts:
worker/worker.py

Anuj Mathur Revert "force message buffers to strings before decoding"
This reverts commit 7c47c36.

Conflicts:
	worker/worker.py
e7195f5
@anujm anujm merged commit 0d292ad into rackerlabs:master Feb 12, 2014
@bkw

what was wrong with that?

@anujm

For some reason the following piece of code was preventing the messages from being processed:

register('bufferjson', self.loads, anyjson.dumps,
                 content_type='application/json',
                 content_encoding='binary')

    def loads(s):
        return anyjson.loads(BytesIO(s))

The messages were being read from the queue, but the callback(on_nova) was never being invoked. Did this work during your testing?

@bkw

Strange - yes, this worked (and still works) for me.

I suspect decoding fails. That is precisely the reason I wrote #281 - no idea why this has not been merged yet. When decoding fails, without #281 messages are silently dropped.

In my case it was the otherway around: With libjson2 installed, decoding failed because libjson2 cannot handle buffers and the conversion code in anyjson did not work for some reason.

Could you try applying both #283 and #281 and see if any decoding errors are logged?

@anujm

As suspected, it's an issue with decoding the json:

2014-02-12 18:29:19,911 - child_worker - ERROR - Decode Error: is not JSON serializable
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/kombu/messaging.py", line 583, in receive_callback
decoded = None if on_m else message.decode()
File "/Library/Python/2.7/site-packages/kombu/message.py", line 123, in decode
self.content_encoding, accept=self.accept)
File "/Library/Python/2.7/site-packages/kombu/serialization.py", line 165, in loads
return decode(data)
File "/Library/Python/2.7/site-packages/anyjson/__init_
.py", line 141, in dumps
return implementation.dumps(value)
File "/Library/Python/2.7/site-packages/anyjson/init.py", line 87, in dumps
return self.encode(data)
File "/Library/Python/2.7/site-packages/simplejson/__init_
.py", line 354, in dumps
return _default_encoder.encode(obj)
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 262, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 340, in iterencode
return _iterencode(o, 0)
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 239, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: is not JSON serializable

Any ideas? The payload is definitely json serializable without #283.

@bkw

Thanks for trying it out. I should have tested with more json libs. simplejson seems to behave differently.
I'll take a look later to come up with a better patch.

@bkw bkw referenced this pull request Feb 28, 2014
Merged

log decode errors, do not ack. #281

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment