Add e2e tests for rootless control-plane.

[vinayakankugoyal]

edit: neolit123

xref #2473
xref kubernetes/enhancements#2568

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[neolit123]

this looks fine as a start.

as we discussed we can do the following workflow

• create cluster nodes
• write rootless test scripts (examples in discovery.yaml)
• init / join
• test rootless (examples in discovery.yaml)
• do kubeadm and k8s e2e suite tests
• upgrade
  we can try - --upgrade-version={{ .vars.kubernetesVersion }} there
• test rootless again

with the new locally committed "rootless" changes in config.yaml does:
./hack/update-workflows.sh work?

[vinayakankugoyal]

as we discussed we can do the following workflow

• create cluster nodes
• write rootless test scripts (examples in discovery.yaml)
• init / join
• test rootless (examples in discovery.yaml)
• do kubeadm and k8s e2e suite tests
• upgrade
  we can try - --upgrade-version={{ .vars.kubernetesVersion }} there
• test rootless again

Done!

with the new locally committed "rootless" changes in config.yaml does:
./hack/update-workflows.sh work?

Yeah, I think I was missing the template file in testinfra and after adding that it worked.

[neolit123]

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubeadm/2511/pull-kubeadm-kinder-upgrade-latest/1407070836409503744

can not mix '--config' with arguments [feature-gates]

ok, so my mistake, but we actually do not allow mixing --feature-gates and --config, so the option must be passed via the ClusterConfiguration.

it must be done here:

kubeadm/kinder/pkg/cluster/manager/actions/kubeadm-init.go

Line 64 in bcc3065

if err := KubeadmInitConfig(c, kubeadmConfigVersion, copyCertsMode, cp1); err != nil {

and all the way to here:

kubeadm/kinder/pkg/cluster/manager/actions/kubeadm-config.go

Line 179 in 47c0def

func getKubeadmConfig(c *status.Cluster, n *status.Node, data kubeadm.ConfigData, options kubeadmConfigOptions) (string, error) {

then the feature gates can be templated in the v1beta2 and v1beta3 config:

kubeadm/kinder/pkg/kubeadm/config.go

Lines 38 to 40 in 47c0def

	templateSource = configTemplateBetaV2
	case "v1beta3":
	templateSource = configTemplateBetaV3

by converting feature gates Foo=true,Bar=true to:

featureGates:
- Foo: true
- Bar: true

under ClusterConfiguration.

these calls no longer need the featuregates explicitly:

kubeadm/kinder/pkg/cluster/manager/actions/kubeadm-init.go

Lines 75 to 77 in bcc3065

	err = kubeadmInitWithPhases(cp1, copyCertsMode, patchesDir, ignorePreflightErrors, featureGates, vLevel)
	} else {
	err = kubeadmInit(cp1, copyCertsMode, patchesDir, ignorePreflightErrors, featureGates, vLevel)

because the values would be prepared in config.

[vinayakankugoyal]

@neolit123 - Thanks! I updated it roughly based on your guidelines, but instead of multiple feature-gates I kept the scope to a single feature-gate for ease of implementation.

[neolit123]

it is a bit limiting in terms of UX.

that said:

• i don't see having a single test with kinder that tests multiple FGs.
• we don't have that many FGs in kubeadm anyway, currently.

so supporting a single feature gate for the time being seems fine to me.

[neolit123]

[preflight] Some fatal errors occurred:
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.22.0-alpha.3.379_a3f24e84594654: output: Error response from daemon: manifest for k8s.gcr.io/kube-apiserver:v1.22.0-alpha.3.379_a3f24e84594654 not found
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.22.0-alpha.3.379_a3f24e84594654: output: Error response from daemon: manifest for k8s.gcr.io/kube-controller-manager:v1.22.0-alpha.3.379_a3f24e84594654 not found
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.22.0-alpha.3.379_a3f24e84594654: output: Error response from daemon: manifest for k8s.gcr.io/kube-scheduler:v1.22.0-alpha.3.379_a3f24e84594654 not found
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.22.0-alpha.3.379_a3f24e84594654: output: Error response from daemon: manifest for k8s.gcr.io/kube-proxy:v1.22.0-alpha.3.379_a3f24e84594654 not found
, error: exit status 1

looks unrelated:
kubernetes/kubernetes#103101

[vinayakankugoyal]

/retest

[neolit123]

the test will not pass until we fix the related issue.
i can send a PR and ping you for LGTM.

[neolit123]

this looks fine to me.
could you please send the kubernetes/test-infra PR as well?

running ./hack/update-workflows.sh would write the new file in the test-infra clone that you have.

[vinayakankugoyal]

Ill do that once we merge this to avoid having to keep both in sync.

[vinayakankugoyal]

/retest

[vinayakankugoyal]

/retest

[neolit123]

/lgtm
/approve

thanks, for enduring this review.
now need the test-infra PR and see if it actually works with Prow.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neolit123, vinayakankugoyal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• kinder/OWNERS [neolit123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment