Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

change aws encryptedCheck to exponential backoff #78601

Merged
merged 4 commits into from Jul 9, 2019
Merged

change aws encryptedCheck to exponential backoff #78601

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e691f7d
change aws encryptedCheck to exponential backoff
May 31, 2019
98dd133
rename and change aws backoff vars
Jun 8, 2019
ee5d623
fix volumecreate comments
Jun 8, 2019
2eeb863
add sleep 5 before exponential backoff in waitUntilVolumeAvailable
Jun 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
17 changes: 11 additions & 6 deletions staging/src/k8s.io/legacy-cloud-providers/aws/aws.go
View file
Open in desktop
Expand Up @@ -225,12 +225,13 @@ const (
createTagFactor = 2.0
createTagSteps = 9

// encryptedCheck* is configuration of poll for created volume to check
// volumeCreate* is configuration of exponential backoff for created volume to check
// it has not been silently removed by AWS.
// On a random AWS account (shared among several developers) it took 4s on
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it has not been silently removed by AWS

Seems like this is no longer accurate

// average.
encryptedCheckInterval = 1 * time.Second
encryptedCheckTimeout = 30 * time.Second
// average, 8s max.
volumeCreateInitialDelay = 5 * time.Second
volumeCreateBackoffFactor = 1.2
volumeCreateBackoffSteps = 10

// Number of node names that can be added to a filter. The AWS limit is 200
// but we are using a lower limit on purpose
Expand Down Expand Up @@ -2419,8 +2420,12 @@ func (c *Cloud) waitUntilVolumeAvailable(volumeName KubernetesVolumeID) error {
// Unreachable code
return err
}

err = wait.Poll(encryptedCheckInterval, encryptedCheckTimeout, func() (done bool, err error) {
backoff := wait.Backoff{
Duration: volumeCreateInitialDelay,
Factor: volumeCreateBackoffFactor,
Steps: volumeCreateBackoffSteps,
}
err = wait.ExponentialBackoff(backoff, func() (done bool, err error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might even be better to not poll volume immediately after creation at all because volume is never available immediately after creation. But I see that ExponentialBackoff does not offer a way to do that.

Yeah, that could save us a DescribeVolume request, which might be helpful for clusters hitting the AWS quota limit.

What if we use PollUntil instead? PollUntil waits the interval before firing the function.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with adding time.Sleep(5 * time.Second) before proceeding with exponential backoff. It has same effect as anything else and works with exponential backoff..

vol, err := disk.describeVolume()
if err != nil {
return true, err
Expand Down