ci-kubernetes-e2e-kind-rootless is failing due to Required 'compute.networks.create' permission for 'projects/k8s-prow-builds/global/networks/kt2-kindinv-prow-def2ae21'

[AkihiroSuda]

ci-kubernetes-e2e-kind-rootless creates a GCP network using the preset service account, but it is failing due to Required 'compute.networks.create' permission for 'projects/k8s-prow-builds/global/networks/kt2-kindinv-prow-def2ae21'

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kind-rootless/1730168423461163008

+ exec kubetest2 kindinv --gcp-project=k8s-prow-builds --gcp-zone=us-west1-b --instance-image=ubuntu-os-cloud/ubuntu-2204-lts --instance-type=n2-standard-4 --kind-rootless --user=rootless --build --up --down --test=ginkgo -- '--focus-regex=\[NodeConformance\]' '--skip-regex=\[Environment:NotInUserNS\]|\[Slow\]' --parallel=8
I1130 10:17:49.102709    1411 app.go:61] The files in RunDir shall not be part of Artifacts
I1130 10:17:49.102839    1411 app.go:62] pass rundir-in-artifacts flag True for RunDir to be part of Artifacts
I1130 10:17:49.102863    1411 app.go:64] RunDir for this run: "/home/prow/go/src/k8s.io/kubernetes/_rundir/def2ae21-08a0-4132-bdd6-16f52da994aa"
I1130 10:17:49.105643    1411 app.go:130] ID for this run: "def2ae21-08a0-4132-bdd6-16f52da994aa"
W1130 10:17:49.105771    1411 deployer.go:515] stat /home/prow/go/src/k8s.io/kubernetes/_output/bin/e2e.test: no such file or directory (Hint: `make WHAT=test/e2e/e2e.test -C $(go env GOPATH)/src/k8s.io/kubernetes`)
W1130 10:17:49.105802    1411 deployer.go:518] stat /home/prow/go/src/k8s.io/kubernetes/_output/bin/ginkgo: no such file or directory (Hint: `make ginkgo -C $(go env GOPATH)/src/k8s.io/kubernetes`)
W1130 10:17:49.105815    1411 deployer.go:521] stat /home/prow/go/src/k8s.io/kubernetes/_output/bin/kubectl: no such file or directory
I1130 10:17:49.106067    1411 deployer.go:236] Executing: ["gcloud" "--project=k8s-prow-builds" "compute" "networks" "create" "kt2-kindinv-prow-def2ae21"]
ERROR: (gcloud.compute.networks.create) Could not fetch resource:
 - Required 'compute.networks.create' permission for 'projects/k8s-prow-builds/global/networks/kt2-kindinv-prow-def2ae21'

E1130 10:17:50.983643    1411 deployer.go:349] failed to run ["gcloud" "--project=k8s-prow-builds" "compute" "networks" "create" "kt2-kindinv-prow-def2ae21"]: exit status 1
I1130 10:17:50.983787    1411 deployer.go:236] Executing: ["gcloud" "--project=k8s-prow-builds" "compute" "firewall-rules" "create" "kt2-kindinv-prow-def2ae21" "--network=kt2-kindinv-prow-def2ae21" "--allow=tcp:22"]
Creating firewall...
failed.
ERROR: (gcloud.compute.firewall-rules.create) Could not fetch resource:
 - Required 'compute.firewalls.create' permission for 'projects/k8s-prow-builds/global/firewalls/kt2-kindinv-prow-def2ae21'
...

YAML:

test-infra/config/jobs/kubernetes/sig-testing/kubernetes-kind.yaml

Lines 585 to 648 in ea5d50f

	- interval: 24h
	cluster: k8s-infra-prow-build
	name: ci-kubernetes-e2e-kind-rootless
	annotations:
	testgrid-dashboards: sig-testing-kind
	testgrid-tab-name: kind-rootless
	description: Kubernetes in Rootless Docker (in GCE VM)
	# GitHub ID: @AkihiroSuda
	testgrid-alert-email: suda.kyoto@gmail.com
	testgrid-num-columns-recent: '6'
	labels:
	preset-service-account: "true"
	preset-k8s-ssh: "true"
	decorate: true
	decoration_config:
	timeout: 90m
	extra_refs:
	- org: kubernetes
	repo: kubernetes
	base_ref: master
	path_alias: k8s.io/kubernetes
	spec:
	containers:
	- image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20231122-5f461e0995-master
	command:
	- runner.sh
	args:
	- /bin/bash
	- -c
	- |
	set -eux
	# kindinv: Kubernetes in (Rootless) Docker in (GCE) VM
	# See https://github.com/rootless-containers/kubetest2-kindinv
	#
	# GCE VM is used for setting up cgroup v2 and systemd.
	# (k8s-infra-prow-build lacks cgroup v2, and the kubekins-e2e container lacks systemd)
	(cd ; GO111MODULE=on go install github.com/rootless-containers/kubetest2-kindinv@master)
	mkdir -p -m 0700 ~/.ssh
	cp -f "${GCE_SSH_PRIVATE_KEY_FILE}" ~/.ssh/google_compute_engine
	cp -f "${GCE_SSH_PUBLIC_KEY_FILE}" ~/.ssh/google_compute_engine.pub
	exec kubetest2 kindinv \
	--gcp-project=k8s-prow-builds \
	--gcp-zone=us-west1-b \
	--instance-image=ubuntu-os-cloud/ubuntu-2204-lts \
	--instance-type=n2-standard-4 \
	--kind-rootless \
	--user=rootless \
	--build \
	--up \
	--down \
	--test=ginkgo \
	-- \
	--focus-regex='\[NodeConformance\]' \
	--skip-regex='\[Environment:NotInUserNS\]|\[Slow\]' \
	--parallel=8
	resources:
	limits:
	memory: 2Gi
	cpu: 2
	requests:
	memory: 2Gi
	cpu: 2
	rerun_auth_config:
	allow_anyone: true

Deployer code:
https://github.com/rootless-containers/kubetest2-kindinv/blob/8e92c94cc24b141f53800057a42ff8509cc49d92/deployer/deployer.go#L325-L342

[AkihiroSuda]

/sig testing

[BenTheElder]

This is WAI, the test needs to request an e2e test project instead of trying to create resources in the project hosting the CI cluster. It does not have permission to create resources in this project and shouldn't.

This way we can scan and delete all resources in the test project when the run is over (via https://github.com/kubernetes-sigs/boskos)

[AkihiroSuda]

PR to use Boskos:

• ci-kubernetes-e2e-kind-rootless: use Boskos instead of raw GCP project #31577

[AkihiroSuda]

The GCE instance is now running and SSHable, but the script is failing with some git error that didn't happen on my own environment

I0116 05:27:23.775985    2329 deployer.go:295] Executing: ["ssh" "-o" "StrictHostKeyChecking=no" "-o" "User=rootless" "kt2-kindinv-prow-f506a09c.us-west1-b.k8s-infra-e2e-boskos-031" "--" "/bin/sh -euxc \"cd /home/rootless/go/src/k8s.io/kubernetes && git config --add receive.denyCurrentBranch warn\""]
+ cd /home/rootless/go/src/k8s.io/kubernetes
+ git config --add receive.denyCurrentBranch warn
I0116 05:27:25.165686    2329 deployer.go:295] Executing: ["git" "push" "--progress" "-f" "ssh://rootless@kt2-kindinv-prow-f506a09c.us-west1-b.k8s-infra-e2e-boskos-031:/home/rootless/go/src/k8s.io/kubernetes"]
fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use
    git push --set-upstream ssh://rootless@kt2-kindinv-prow-f506a09c.us-west1-b.k8s-infra-e2e-boskos-031:/home/rootless/go/src/k8s.io/kubernetes master
To have this happen automatically for branches without a tracking
upstream, see 'push.autoSetupRemote' in 'git help config'.
Error: failed to run ["git" "push" "--progress" "-f" "ssh://rootless@kt2-kindinv-prow-f506a09c.us-west1-b.k8s-infra-e2e-boskos-031:/home/rootless/go/src/k8s.io/kubernetes"]: exit status 128

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kind-rootless/1747126392576806912

[AkihiroSuda]

Fixed in:

• run git with --set-upstream rootless-containers/kubetest2-kindinv#8

Now tests are passing

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-kind-rootless/1747146398966484992

...
Ran 161 of 7407 Specs in 489.046 seconds
SUCCESS! -- 161 Passed | 0 Failed | 0 Pending | 7246 Skipped

[pacoxu]

https://testgrid.k8s.io/sig-testing-kind#kind-rootless is green now.
🎉