-
Notifications
You must be signed in to change notification settings - Fork 499
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: PagerDuty Rate Limiting #7218
feat: PagerDuty Rate Limiting #7218
Conversation
In my tests (toy account with 1000 incidents), this fixed the rate-limit errors that prevented cloudquery from getting data. Fetching '*' took 5m13s unfortunately.
5ab5055
to
3b7b52c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code looks good @shimonp21. Have you considered using a retry with a backoff when the limit is reached (similar to here)? Otherwise with the approach in this PR we can't guarantee we use the max available quota
@erezrokah I considered it - but because the limit is per org (https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTUz-rate-limiting#what-are-our-limits), I strongly want to avoid "using all available quota" :). |
🤖 I have created a release *beep* *boop* --- ## [1.3.0](plugins-source-pagerduty-v1.2.2...plugins-source-pagerduty-v1.3.0) (2023-01-26) ### Features * Pagerduty `team_ids` config ([#7206](#7206)) ([29832cf](29832cf)) * Pagerduty `team_ids` config ([#7206](#7206)) ([883f6e6](883f6e6)) * PagerDuty Rate Limiting ([#7218](#7218)) ([09fb388](09fb388)) ### Bug Fixes * **deps:** Update module github.com/cloudquery/plugin-sdk to v1.29.0 ([#7121](#7121)) ([b7441c9](b7441c9)) * Fetch incidents with `dateRange="all"`, and add query examples to docs ([#7184](#7184)) ([0d84525](0d84525)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
In my tests (toy account with 1000 incidents), this fixed the rate-limit errors that prevented cloudquery from getting data. Fetching '*' took 5m13s unfortunately.
I used the
RateLimitHttpClient
instead of just callinglimiter.Wait
before every call, since some calls are paginated by the SDK (i.e. thefor
loop is in thepagerduty-go
sdk).mention: #6981