Improve Retry Logic

[kcajmagic]

Currently, when an insert fails with n number of records. The exact same query will be rejected.

To improve upon this I recommend having a retry channel where records get added upon to try again in a batch and as a last result do an insert one record at a time.

[kristinapathak]

Currently, we can retry n number of times. Per my understanding, this change would mean only being able to retry each record twice. This seems less flexible, but I do agree if there's a problem with one record in a batch it will eventually force us to drop all records in that batch.

Implementation suggestions:

• when we put a record into a retry channel, wrap it in a struct with the number of times it has been retried, so the retry workers can check and reject records as they find ones past the acceptable number of retries
• make the retry batch smaller, so instead of ultimately dropping 50 records maybe we drop 5.

I'm also worried individual/smaller batches of inserts after a failure will greatly slow down the database at a time when it may already be under load which is causing the failures.

[kcajmagic]

I like the idea of wrapping the object in an envelope to limit how many times a record can be retried.

I feel like the potential slow down is a necessary evil to have a higher success rate of getting events into the database.

[kristinapathak]

I'm good with some slowdown (with the retry batch size is a configurable value), but I don't think the last attempt should be inserting one record at a time. This could also make it harder and take longer to recover from an issue.

[kristinapathak]

This would involve moving batchInserter logic from svalinn into the database, combining the retryer and the batcher.

[kristinapathak]

After further discussion, we agreed on making the time to wait between retries exponential and to move the batchInserter from svalinn to codex. The retryer and batchInserter are still separate.