[Failing Test] ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts

[thejoycekung]

Which jobs are failing:
ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts

Which test(s) are failing:
Overall

Since when has it been failing:
At least October 5

Testgrid link:
https://testgrid.k8s.io/sig-release-master-informing#capg-conformance-v1alpha3-k8s-master

Reason for failure:
A few different error messages ... I'm not sure which one is the root cause for failure so I'll include all of them.

# Get kubeconfig and store it locally.
kubectl get secrets test1-kubeconfig -o json | jq -r .data.value | base64 --decode > ./kubeconfig
timeout 15m bash -c "while ! kubectl --kubeconfig=./kubeconfig get nodes | grep master; do sleep 1; done"
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding

There's also a long string of error messages from Gazelle, like:

gazelle: finding module path for import titanic.biz/bar: exit status 1: go: finding module for package titanic.biz/bar
cannot find module providing package titanic.biz/bar: module titanic.biz/bar: reading https://proxy.golang.org/titanic.biz/bar/@v/list: 410 Gone
	server response: not found: titanic.biz/bar@latest: unrecognized import path "titanic.biz/bar": parsing titanic.biz/bar: XML syntax error on line 1: expected attribute name in element

or

gazelle: finding module path for import lib: exit status 1: package lib is not in GOROOT (/bazel-scratch/.cache/bazel/_bazel_root/cae228f2a89ef5ee47c2085e441a3561/external/go_sdk/src/lib)
gazelle: finding module path for import nosuchpkg: exit status 1: package nosuchpkg is not in GOROOT (/bazel-scratch/.cache/bazel/_bazel_root/cae228f2a89ef5ee47c2085e441a3561/external/go_sdk/src/nosuchpkg)

or

gazelle: finding module path for import golang.org/x/tools/internal/lsp/circular/double/one: exit status 1: go: finding module for package golang.org/x/tools/internal/lsp/circular/double/one
module golang.org/x/tools@latest found (v0.0.0-20201019175715-b894a3290fff), but does not contain package golang.org/x/tools/internal/lsp/circular/double/one

Also looks like it's having trouble copying out the results:

INFO: Build completed successfully, 3323 total actions
INFO: Build completed successfully, 3323 total actions
mkdir -p /home/prow/go/src/k8s.io/kubernetes/_output/bin/
+ cp /home/prow/go/src/k8s.io/kubernetes/bazel-bin/test/e2e/e2e.test /home/prow/go/src/k8s.io/kubernetes/_output/bin/e2e.test
cp: cannot stat '/home/prow/go/src/k8s.io/kubernetes/bazel-bin/test/e2e/e2e.test': No such file or directory

Also looks like it's having trouble finding kubeconfig:

+ kubectl --kubeconfig=/home/prow/go/src/k8s.io/kubernetes/kubeconfig version
error: stat /home/prow/go/src/k8s.io/kubernetes/kubeconfig: no such file or directory

And finally gcloud is having troubles with access configs?

+ gcloud compute --project k8s-jkns-gce-slow-1-4 instances add-access-config --zone us-east4-a test1-md-0-p7bzr
ERROR: (gcloud.compute.instances.add-access-config) Could not fetch resource:
 - At most one access config currently supported.

There are some errors during cleanup/teardown but I think those are checking to make sure the resource is properly removed, e.g.:

+ timeout 600 kubectl delete cluster test1
cluster.cluster.x-k8s.io "test1" deleted
+ timeout 600 kubectl wait --for=delete cluster/test1
Error from server (NotFound): clusters.cluster.x-k8s.io "test1" not found
+ true
+ make kind-reset
make: *** No rule to make target 'kind-reset'.  Stop.
+ true
++ go env GOPATH
+ cd /home/prow/go/src/k8s.io/kubernetes
+ rm -f _output/bin/e2e.test
+ gcloud compute forwarding-rules delete --project k8s-jkns-gce-slow-1-4 --global test1-apiserver --quiet
ERROR: (gcloud.compute.forwarding-rules.delete) Could not fetch resource:
 - The resource 'projects/k8s-jkns-gce-slow-1-4/global/forwardingRules/test1-apiserver' was not found
+ true

Anything else we need to know:
Example spyglass link: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts/1318486842399526912

/sig cluster-lifecycle
/cc @kubernetes/ci-signal

[neolit123]

i've notified the #cluster-api channel on k8s slack about this.
thanks for the report @thejoycekung

[cpanato]

/assign
/triage accepted

[cpanato]

much better https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-cluster-api-provider-gcp#capg-conformance-v1alpha3-k8s-master have a fail, but i will observe in the following days

[neolit123]

Thanks, Carlos!

[thejoycekung]

@cpanato Thanks for your help! I think the fail is unrelated, that test is failing/flaking across several other jobs so I logged another issue for it.

[cpanato]

I opened a PR for a fix proposal for that test: #95831