Queued messages should not get lost on exception during sending

[dirkschneemann]

When I run graphyte in interval mode to let it send messages to Graphite in a background thread, all messages stored until the next scheduled send operation are stored in a queue. Once the next scheduled send is reached, messages will be popped from the queue and batched for sending. However, if the send operation fails (e.g. due to a temporary network outage), this batch of messages will be lost because they will not be put back into the queue for the next try.

I think it would be useful to put messages back into the queue and try them again on the next scheduled interval, together with the ones which might have been added on top of them in the meantime.

If there are use cases where this behaviour is undesired, it could be made configurable in the constructor of the Sender class (retry_failed_messages=True or similar).

[benhoyt]

Thanks for the note. This is a worthwhile idea, but it could get tricky quickly, because I think you'd want to count how many retries had been performed for a particular batch so that you don't retry an arbitrary number of times if the network goes down (and putting them back on the queue would then potentially overflow the queue, preventing new messages from being sent). Then you have to worry about how many retries, backoff, etc.

I'd probably lean towards having your monitoring system monitor the error/exception logs in send_socket and alerting you on failure so you can fix the underlying issue.

However, if you did want to do this, I'd suggest subclassing Sender and overriding send_socket in your own code. You could either just retry a couple of times in a loop if there's an exception, or push the message back onto the queue to be picked up in the next interval.

[krzysbaranski]

Also library is swallowing almost all exceptions, so even in synchronous mode you cannot retry

[benhoyt]

@krzysbaranski I presume you're referring to the except Exception in send_socket? True, though that's definitely by design. The thinking is the caller shouldn't have to worry about errors; so graphyte logs them (assuming you're doing something with your logs).

If you want to do retries or more complex behavior, you can subclass Sender and override send_socket (which is used in both sync or async mode), adding your own exception handling, retries, etc. In your override, you'd call send_message -- which doesn't have a catch-all except clause -- to do the low-level sending operation.

I'm going to close this issue per my earlier message.