ci-kubernetes-e2e-kind-rootless: `apt-get` on GCE host is flaky

[AkihiroSuda]

What should be cleaned up or changed:
Flaky apt-get on GCE should be analyzed and fixed:

+ apt-get install -y make uidmap
Reading package lists...
Building dependency tree...
Reading state information...
Package make is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
 E: Package 'make' has no installation candidate
Error: failed to run ["ssh" "-o" "StrictHostKeyChecking=no" "-o" "User=rootless" "kt2-kindinv-prow-d860d175.us-central1-b.k8s-infra-e2e-boskos-025" "--" "sudo" "~/kubetest2-kindinv-provision.sh"]: exit status 100

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kind-rootless/1937710777624432640

Provide any links for context:
Flaky since May 19
https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kind-rootless

test-infra/config/jobs/kubernetes/sig-testing/kubernetes-kind-ci.yaml

Lines 258 to 321 in d2cd9b0

	- interval: 24h
	cluster: k8s-infra-prow-build
	name: ci-kubernetes-e2e-kind-rootless
	annotations:
	testgrid-dashboards: sig-testing-kind
	testgrid-tab-name: kind-rootless
	description: Kubernetes in Rootless Docker (in GCE VM)
	# GitHub ID: @AkihiroSuda
	testgrid-alert-email: suda.kyoto@gmail.com
	testgrid-num-columns-recent: '6'
	labels:
	preset-service-account: "true"
	preset-k8s-ssh: "true"
	decorate: true
	decoration_config:
	timeout: 90m
	extra_refs:
	- org: kubernetes
	repo: kubernetes
	base_ref: master
	path_alias: k8s.io/kubernetes
	spec:
	containers:
	- image: gcr.io/k8s-staging-test-infra/kubekins-e2e:v20250613-876fb90a97-master
	command:
	- runner.sh
	args:
	- /bin/bash
	- -c
	- |
	set -eux
	# kindinv: Kubernetes in (Rootless) Docker in (GCE) VM
	# See https://github.com/rootless-containers/kubetest2-kindinv
	#
	# GCE VM is used for setting up cgroup v2 and systemd.
	# (k8s-infra-prow-build lacks cgroup v2, and the kubekins-e2e container lacks systemd)
	(cd ; GO111MODULE=on go install github.com/rootless-containers/kubetest2-kindinv@master)
	mkdir -p -m 0700 ~/.ssh
	cp -f "${GCE_SSH_PRIVATE_KEY_FILE}" ~/.ssh/google_compute_engine
	cp -f "${GCE_SSH_PUBLIC_KEY_FILE}" ~/.ssh/google_compute_engine.pub
	exec kubetest2 kindinv \
	--boskos-location=http://boskos.test-pods.svc.cluster.local \
	--gcp-zone=us-central1-b \
	--instance-image=ubuntu-os-cloud/ubuntu-2204-lts \
	--instance-type=n2-standard-4 \
	--kind-rootless \
	--user=rootless \
	--build \
	--up \
	--down \
	--test=ginkgo \
	-- \
	--focus-regex='\[NodeConformance\]' \
	--skip-regex='\[Environment:NotInUserNS\]|\[Slow\]' \
	--parallel=8
	resources:
	limits:
	memory: 2Gi
	cpu: 2
	requests:
	memory: 2Gi
	cpu: 2
	rerun_auth_config:
	allow_anyone: true

apt-get is invoked here: https://github.com/rootless-containers/kubetest2-kindinv/blob/0ec080069be26fa4b9f874f34f72d053f0a010e6/deployer/kubetest2-kindinv-provision.sh#L12

[BenTheElder]

Worth noting that CI runs on cgroup v2 these days so you may not need a separate remote VM anymore.

Apt-get is running in the externally created VM right? Not the CI pod.

For the CI pod we have some known options to reduce DNS flakiness (configuring pod DNS ...), but not for an arbitrary VM.

/sig node testing

[AkihiroSuda]

Worth noting that CI runs on cgroup v2 these days so you may not need a separate remote VM anymore.

We still need some VM, or VM-like containers (LXD/Incus), to provide systemd.

IIUC, we still have no support for nested virt in the CI?

Apt-get is running in the externally created VM right? Not the CI pod.

Right.
https://github.com/rootless-containers/kubetest2-kindinv/blob/0ec080069be26fa4b9f874f34f72d053f0a010e6/deployer/kubetest2-kindinv-provision.sh#L12