Skip to content

Add a max reasonable timeout #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 23, 2020
Merged

Add a max reasonable timeout #318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f0cb44c
add a maximum timeout value to prevent extremely long timeouts
StephenButtolph Aug 23, 2020
a15dd9b
fixed timeout tests
StephenButtolph Aug 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion snow/networking/timeout/manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ func (m *Manager) Initialize(
return m.tm.Initialize(
time.Second,
500*time.Millisecond,
2,
10*time.Second,
1.1,
time.Millisecond,
namespace,
registerer,
Expand Down
8 changes: 8 additions & 0 deletions utils/timer/adaptive_timeout_manager.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ type AdaptiveTimeoutManager struct {
currentDurationMetric prometheus.Gauge

minimumDuration time.Duration
maximumDuration time.Duration
increaseRatio float64
decreaseValue time.Duration

Expand All @@ -66,6 +67,7 @@ type AdaptiveTimeoutManager struct {
func (tm *AdaptiveTimeoutManager) Initialize(
initialDuration time.Duration,
minimumDuration time.Duration,
maximumDuration time.Duration,
increaseRatio float64,
decreaseValue time.Duration,
namespace string,
Expand All @@ -77,6 +79,7 @@ func (tm *AdaptiveTimeoutManager) Initialize(
Help: "Duration of current network timeouts in nanoseconds",
})
tm.minimumDuration = minimumDuration
tm.maximumDuration = maximumDuration
tm.increaseRatio = increaseRatio
tm.decreaseValue = decreaseValue
tm.currentDuration = initialDuration
Expand Down Expand Up @@ -164,6 +167,11 @@ func (tm *AdaptiveTimeoutManager) remove(id ids.ID, currentTime time.Time) {
// If the current timeout duration is less than or equal to the
// timeout that was triggered, double the duration.
tm.currentDuration = time.Duration(float64(tm.currentDuration) * tm.increaseRatio)

if tm.currentDuration > tm.maximumDuration {
// Make sure that we never get stuck in a bad situation
tm.currentDuration = tm.maximumDuration
}
}
} else {
// This request is being removed because it finished successfully.
Expand Down
1 change: 1 addition & 0 deletions utils/timer/adaptive_timeout_manager_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ func TestAdaptiveTimeoutManager(t *testing.T) {
tm.Initialize(
time.Millisecond, // initialDuration
time.Millisecond, // minimumDuration
time.Hour, // maximumDuration
2, // increaseRatio
time.Microsecond, // decreaseValue
"gecko", // namespace
Expand Down