[Proposal] Configurable backoff strategies

[kishaningithub]

Currently the only back off strategy that is available is an exponential backoff strategy. From what i have experienced, the redshift data api adds a minimum of 2 second latency on top of the latency added by redshift. If the redshift data api's latency is high

Given the minimum is 200ms and a factor of 2 the delays will progress like 200 ms, 400 ms, 800 ms and so on

Time taken by redshift data api	Time taken by plugin to return response
2 seconds	3 Seconds (200ms + 400ms + 800ms + 1600ms)
3 seconds	3 Seconds (200ms + 400ms + 800ms + 1600ms)
4 seconds	~6 Seconds (200ms + 400ms + 800ms + 1600ms + 3200ms)
5 seconds	~6 Seconds(200ms + 400ms + 800ms + 1600ms + 3200ms)
6 seconds	~6 Seconds (200ms + 400ms + 800ms + 1600ms + 3200ms)
7 seconds	~12 Seconds (200ms + 400ms + 800ms + 1600ms + 3200ms + 6400ms)
8 seconds	~12 Seconds (200ms + 400ms + 800ms + 1600ms + 3200ms + 6400ms)
9 seconds	~12 Seconds (200ms + 400ms + 800ms + 1600ms + 3200ms + 6400ms)
10 seconds	~12 Seconds (200ms + 400ms + 800ms + 1600ms + 3200ms + 6400ms)

If you see above the plugin is currently adding latency overheads on top of what is added by redshift and redshift data api. I propose there be an option to eliminate this latency(Eg. using linear backoff) so that we can get the data in the dashboards as soon as it is available from the redshift data api for those who need it.

The linear back off strategy can query once every 200 ms till the data is available.

From an implementation point of view, i think it would suffice if the backoff factor is changed to 1 for linear backoff strategy and 2 for exponential back off strategy in this code

grafana-aws-sdk/pkg/sql/api/api.go

Line 65 in fb33415

Factor: 2,

[andresmgot]

Thanks for the suggestion @kishaningithub. I understand your point, for short running queries, having to wait for 2s is almost 50% of its total time.

In any case, I think it is still useful to use some kind of backoff for long running queries, this can help preventing reaching API limitations.

I am also not sure that we should expose this setup to all the users, it would clutter the UI and most of the users won't need it.

What we could do is reduce the factor so it's slowly increasing until it reaches its limit. I think we can try with a factor of 1.1 (e.g. a query taking 4 seconds, would return after 4.27s, saving a bunch of requests):

https://play.golang.com/p/uSLtWzwRUgS

What do you think?

[kishaningithub]

@andresmgot Your approach feels like a good way forward

[andresmgot]

I have just tagged a new version v0.10.2. Can you update the Redshift data source with this @kishaningithub ? Thanks!