feat: Adding a batch updater to allow usage updates to be batched #1326
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mnorbury
force-pushed
the
feat/adding-batch-usage-updater
branch
3 times, most recently
from
ad97e41 to
0596175
Compare
This will allows a client to update the usage asynchronously. The updater will only call the API when a configurable number of rows have been updated or a timeout is reached. In addition the updater will flush any remaining rows before closing.
mnorbury
force-pushed
the
feat/adding-batch-usage-updater
branch
from
4cf66fb to
d4b85a3
Compare
mnorbury
force-pushed
the
feat/adding-batch-usage-updater
branch
from
c26cb27 to
22de9a3
Compare
mnorbury
requested review from
erezrokah,
hermanschaaf and
yevgenypats
as code owners
mnorbury
force-pushed
the
feat/adding-batch-usage-updater
branch
2 times, most recently
from
85c4412 to
4484abd
Compare
…o allow a minimum update interval to be set also
mnorbury
force-pushed
the
feat/adding-batch-usage-updater
branch
from
4484abd to
c2f5be4
Compare
Member
hermanschaaf
left a comment
hermanschaaf
left a comment
There was a problem hiding this comment.
Looks great! Left a couple of comments, nothing major
premium/usage.go
Outdated
| defaultMaxRetries = 5 | ||
| defaultMaxWaitTime = 60 * time.Second | ||
| defaultMinimumUpdateDuration = 10 * time.Second | ||
| defaultFlushDuration = 30 * time.Second |
Member
There was a problem hiding this comment.
Suggested change
| defaultFlushDuration = 30 * time.Second | |
| defaultMaxTimeBetweenFlushes = 30 * time.Second |
(nit)
premium/usage.go
Outdated
| defaultBatchLimit = 1000 | ||
| defaultMaxRetries = 5 | ||
| defaultMaxWaitTime = 60 * time.Second | ||
| defaultMinimumUpdateDuration = 10 * time.Second |
Member
There was a problem hiding this comment.
Suggested change
| defaultMinimumUpdateDuration = 10 * time.Second | |
| defaultMinTimeBetweenFlushes = 10 * time.Second |
(nit)
And we'd update the With... functions accordingly
premium/usage.go
Outdated
|
|
||
| // calculateRetryDuration calculates the duration to sleep relative to the query start time before retrying an update | ||
| func (u *BatchUpdater) calculateRetryDuration(statusCode int, headers http.Header, queryStartTime time.Time, retry int) (time.Duration, error) { | ||
| if statusCode == http.StatusTooManyRequests { |
Member
There was a problem hiding this comment.
429 is typically for when the client has made too many requests. We can keep handling this, but the server will typically return 503 if it's getting overloaded, so we should handle that as well https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503
Contributor
Author
There was a problem hiding this comment.
We should add 503 as a possible return here
In this case we want to respect the wait time the server has suggested, so just wait.
hermanschaaf
approved these changes
Oct 24, 2023
hermanschaaf
pushed a commit
that referenced
this pull request
Oct 30, 2023
🤖 I have created a release *beep* *boop* --- ## [4.17.0](v4.16.1...v4.17.0) (2023-10-30) ### Features * Add IsPaid flag to table definition ([#1327](#1327)) ([ffd14bf](ffd14bf)) * Add OnBeforeSend hook ([#1325](#1325)) ([023ebbc](023ebbc)) * Adding a batch updater to allow usage updates to be batched ([#1326](#1326)) ([0301ed7](0301ed7)) * Adding quota monitoring for premium plugins ([#1333](#1333)) ([b7a2ca5](b7a2ca5)) * Allow sync to be cancelled when in progress ([#1334](#1334)) ([6d7be0b](6d7be0b)) ### Bug Fixes * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to 50d3871 ([#1337](#1337)) ([f15a89d](f15a89d)) * **deps:** Update github.com/cloudquery/arrow/go/v14 digest to f46436f ([#1329](#1329)) ([ee24384](ee24384)) * **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.4.2 ([#1335](#1335)) ([2ecd2a1](2ecd2a1)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.13.0 ([#1332](#1332)) ([5553f85](5553f85)) * **deps:** Update module github.com/cloudquery/plugin-pb-go to v1.13.1 ([#1336](#1336)) ([b782ee7](b782ee7)) * **deps:** Update module google.golang.org/grpc to v1.58.3 [SECURITY] ([#1331](#1331)) ([43f60c2](43f60c2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This will allows a client to update the usage asynchronously. The updater will only call the API when a configurable number of rows have been updated or a timeout is reached. In addition the updater will flush any remaining rows before closing.
If an error is encountered then an exponential backoff up to a max number of retries or a maximum wait time (relative to the start time of the last query) will be followed.
In addition, if the server replies with a status of
429and includes aRetry-Afterheader, then the client will wait that number of seconds before retrying.It might be worth considering just using a retry library also e.g. https://github.com/avast/retry-go.