Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

node e2e tests - retry fkaky failures caused by gcloud authentication #19375

Merged
merged 2 commits into from
Jan 13, 2016
Merged

node e2e tests - retry fkaky failures caused by gcloud authentication #19375

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aebc1bc
node e2e tests - retry fkaky failures caused by gcloud authentication
pwittrock Jan 7, 2016
4397bde
Errof -> Warningf
pwittrock Jan 7, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 10 additions & 0 deletions test/e2e_node/gcloud/gcloud.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,16 @@ func (gc *gCloudClientImpl) Run(
go func() {
// Create the tmp directory
out, err := gc.Command("mkdir", "-p", tDir)

// Work around for gcloud flakiness - TODO: debug why gcloud sometimes cannot find credentials for some hosts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ugly. Sigh :(
Should we instead add ssh tokens and use ssh instead of gcloud?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, maybe we will need to resort to ssh tokens, though the k8s e2e tests use gcloud and do not seem to encounter this. I am hoping we may be able to debug the issue and continue to use gcoud.

// If there was an error about credentials, retry making the directory 6 times to see if it can be resolved
// This is to help debug if the credential issues are persistent for a given host on a given run, or transient
// And if downstream gcloud commands are also impacted
for i := 0; i < 6 && err != nil && strings.Contains(string(out), "does not have any valid credentials"); i++ {
glog.Warningf("mkdir failed on host %s due to credential issues, retrying in 5 seconds %v %s", gc.host, err, out)
time.Sleep(5 * time.Second)
out, err = gc.Command("mkdir", "-p", tDir)
}
if err != nil {
glog.Errorf("mkdir failed %v %s", err, out)
h.Output <- RunResult{out, err, fmt.Sprintf("mkdir -p %s", tDir)}
Expand Down