use plugin.retryHydrate

[judell]

This is a query against the Steampipe public Slack that gets pretty far, then triggers a rate limit and will not complete.

with days as (
  select
    to_char(day, 'YYYY-MM-DD') as day
  from
    generate_series(date('2021-04-15'), date('2021-11-16'), '1 day') as day
),
queries as (
  select
    'in:#steampipe on:' || d.day as query
  from
    days d
),
data as (
  select
    user_name,
    user_id,
    to_char(timestamp, 'YYYY-MM-DD') as day,
    text
  from
    slack_search
  where
    query in ( (select query from queries) )
  order by 
    day 
)
select 
  day,
  count(*) as messages
from
  data 
group by
  day

Here is my first attempt to use plugin.retryHydrate: https://github.com/turbot/steampipe-plugin-slack/compare/use-retry-hydrate#diff-dd4fe9f3414fd76a85ed3ab2bc2e8896acd02d8e0687afc4599bc1bfec51059f

It seems to work, in that with this change the query will eventually complete (Time: 8m50.043227805s)

Again it gets fairly far before the rate limit kicks in. Then, there is a lot of chatter between e.g. the first line in the log extract below that returns 59 messages for a day, and the next line that returns 1 message for the following day.

If the API is going to make the plugin wait 3 seconds, would it be best to just take that wait, for example in shouldRetryError, with time.Sleep(3 * time.Second)?

I've tried that, and it also eventually completes (Time: 5m53.883121854s).

It appears that the explicit wait finishes sooner, but I'm not sure about the time comparison because the point at which rate limiting kicks in may vary from run to run.

My questions:

• What is retryHydrate's backoff strategy? I thought it would increase the backoff when the plugin encounters resistance, but I don't think I'm seeing evidence of that here.

• Is taking an explicit wait the right strategy here?

2021-11-16T15:53:32.858-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: returning msgs: =59
2021-11-16T15:53:32.869-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: retrying="\"\""
2021-11-16T15:53:33.009-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: api.SearchContext="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:33.009-0800 [ERROR] steampipe-plugin-slack.plugin: [ERROR] slack_search.listSearches: retry_error="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:33.110-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: retrying="\"\""
2021-11-16T15:53:33.239-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: api.SearchContext="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:33.239-0800 [ERROR] steampipe-plugin-slack.plugin: [ERROR] slack_search.listSearches: retry_error="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:33.439-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: retrying="\"\""
2021-11-16T15:53:33.578-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: api.SearchContext="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:33.578-0800 [ERROR] steampipe-plugin-slack.plugin: [ERROR] slack_search.listSearches: retry_error="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:33.879-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: retrying="\"\""
2021-11-16T15:53:34.015-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: api.SearchContext="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:34.015-0800 [ERROR] steampipe-plugin-slack.plugin: [ERROR] slack_search.listSearches: retry_error="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:34.516-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: retrying="\"\""
2021-11-16T15:53:34.657-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: api.SearchContext="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:34.657-0800 [ERROR] steampipe-plugin-slack.plugin: [ERROR] slack_search.listSearches: retry_error="slack rate limit exceeded, retry after 3s"
2021-11-16T15:53:35.457-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: retrying="\"\""
2021-11-16T15:53:35.719-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: api.SearchContext=<nil>
2021-11-16T15:53:35.719-0800 [WARN]  steampipe-plugin-slack.plugin: [WARN]  slack_search: returning msgs: =1

PS: This branch also increases the gulp size (params.Count = 100) from the default which is evidently 20.

[rajlearner17]

@judell very interesting finding, thanks for raising it, we will take a look at the existing backoff strategy used vs what we have been using in other plugins and initial appropriate change. Keep you posted on it.

[judell]

Thanks @rajlearner17. My sense of this evolving landscape is that, as a plugin writer:

1. You hope first that your API's golang SDK will handle everything

2. If not, the plugin SDK can help

In case of 2, would we expect that the plugin writer should generally expect to cooperate with retryHydrate, or rather expect to just let it do its thing?

[rajlearner17]

@judell we are considering this to check and work this week. We will keep posted.

[judell]

See also: turbot/steampipe-plugin-github#118

[judell]

@rajlearner17

PS: This branch also increases the gulp size (params.Count = 100) from the default which is evidently 20.

That a separate thing, I could've made a separate issue for it, should I?

[rajlearner17]

@judell noted. We will analyze when back in full capacity next week.

[judell]

Thanks @rajlearner17.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

'This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[github-actions]

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[github-actions]

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

[github-actions]

'This issue was closed because it has been stalled for 90 days with no activity.'

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 90 days with no activity.