New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
enhancement(networking)!: Fibonacci instead of fixed backoff #1006
Conversation
Signed-off-by: Luke Steensen <luke.steensen@gmail.com>
I spoke to @lukesteensen about this offline and mentioned this AWS post on exponential backoffs + jitter. @LucioFranco @a-rodin do either of you have inisghts or experience with this? |
Also, this might be a good time to introduce #1022. I'd vote for a |
I don't have experience with Fibonacci backoff, but I'm thinking could it make sense to provide ability to set not only initial backoff, but ability to skip steps? My reasoning is that for exponential backoff we have two parameter to control, initial value and the exponent base. For Fibonacci numbers we can set only the initial value. But as an analog for the base it is possible to take not every number from the Fibonacci sequence, but instead only each |
Yeah, the value of supporting a bunch of different strategies is likely very low. The post you linked is specifically about the case of optimistic concurrency control and conflicting updates to a database. I don't really expect that requests from Vector to an external system will ever be considered conflicting and cause correlated failures in the same way. We could do a little simulation of our own, but our model (service being up/down/overloaded) isn't as interesting or precise.
I see the simpler configuration as a benefit of fibonacci over exponential. We could definitely explore an option like this if we see some demand from users, but I don't expect many people to need that level of control. |
Good idea 👍
what about
I think it makes sense to add that config option. Would we want to default like we do now to max u64? This overall looks great! |
Noting, we are waiting to merge this since it is a breaking change. We'll be bundling it with other configuration changes. |
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
82f804a
to
7dfc237
Compare
Ok @lukesteensen this has the rest of the changes if you can take a quick look. @binarylogic I was having trouble adding a new field to As a summary, the changes are as follows:
|
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
1b5cd73
to
a03a736
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
This is just a quick spike to start the discussion, not quite ready to be merged.
We currently use a fixed delay between retries, which is not ideal. Fibonacci backoff is a simple strategy that does pretty well in evaluations and is easy to implement with no added dependencies.
If we want to switch to this strategy, there are a few things we'll need to decide:
retry_backoff_secs
config item to specify that it's a base unit of a more complex strategy?max_backoff_secs
config item to every relevant sink?