log decode errors, do not ack. #281

Merged
merged 1 commit into from Mar 14, 2014

4 participants

@bkw
bkw commented Feb 6, 2014

In case of decoding errors (most likely caused be an installed json library that cannot deal with kombu's message buffers) log the error and do NOT ack the message. Otherwise we lose messages.

This is a precursor for another pr coming up, that fixes the decoding errors.

@SandyWalsh SandyWalsh and 1 other commented on an outdated diff Feb 6, 2014
worker/worker.py
@@ -135,6 +135,10 @@ def on_nova(self, body, message):
(e, json.loads(str(message.body))))
raise
+ def on_decode_error(self, message, exc):
+ _get_child_logger().exception("Decode Error: %s" % exc)
+ # no NOT call message.ack(), otherwise the message will be lost
@SandyWalsh
SandyWalsh added a line comment Feb 6, 2014

s/no Not/Do Not/

perhaps?

@bkw
bkw added a line comment Feb 6, 2014

fixed, thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@SandyWalsh

Where is this actually called?

@bkw
bkw commented Feb 6, 2014

in kombu serialization.

@bkw
bkw commented Feb 6, 2014

or if you mean where on_decode_error is called, it is also a hook from the kombu mixin: https://kombu.readthedocs.org/en/latest/reference/kombu.mixins.html#kombu-mixins

@bkw
bkw commented Feb 6, 2014

rebased to current master

@bkw
bkw commented Feb 6, 2014

rebased again

@thomasem
Rackspace Hosting member

Are we still pursuing the feature this PR is related to??

@bkw

Yes, but the other PR is only slightly related.
This one is useful on its own since it prohibits losing messages that could not be decoded for whatever reason. I strongly vote for merging this one, it helps diagnosing problems like #287.

@thomasem
Rackspace Hosting member

Going back through all that transpired, yeah, I agree. Hehe, that auto ack from the mixin base class in the on_decode_error is kind of scary.

@anujm

Can you rebase this from master, bkw?

@bkw

rebased again.

@anujm anujm merged commit 9d1a93a into rackerlabs:master Mar 14, 2014

1 check passed

Details default The Travis CI build passed
@anujm

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment