Add retry logic to storage writers

[black-adder]

Create a new struct (implements the storage writer interface) which takes care of retries with exponential backoff.

[vprithvi]

Aren't retries already configurable at the storage driver level? Why do we need this in application code?

[black-adder]

I'd rather have them in one place rather than exposing configs per storage mechanism (what if we use a driver that doesn't support it?).

[black-adder]

meh, I have no strong feelings about this. We can go with the config per driver for now given the configs already exist per driver.

[black-adder]

So we checked for both the elastic search and cassandra drivers that we use and both pseudo support retries as in they ask you to implement an interface that takes care of retries (and whatever other logic you need per retry) and then you pass in the retry struct. Given this, it makes sense for us to write one retry mechanism that we use for all storage writers.

[vprithvi]

I'm not particularly enthused with having our own retries because more information/access might be available at the storage level.
For e.g., One might write a Cassandra retrier that reduces data consistency requirements on every retry.

[yurishkuro]

do we have any data to support the requirement in the first place?

[black-adder]

elastic search is dropping a couple of spans every now and then, I estimate around a couple of hundred a day due to elasticsearch not being able to keep up. We can add more boxes, more shards, and retries.

Having typed this out, it makes sense to add more shards first before doing retries.