Enable EKS upgrade test #3229

pydctw · 2022-02-17T22:52:53Z

What type of PR is this?

What this PR does / why we need it:
Enable EKS upgrade test, updated k8s version to EKS supported versions and setup AWSClusterStaticIdentity in a central location for use by multiple EKS tests.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3228

Special notes for your reviewer:

Checklist:

squashed commits
includes documentation
adds unit tests
adds or updates e2e tests

Release note:

NONE

k8s-ci-robot · 2022-02-17T22:53:00Z

@pydctw: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pydctw · 2022-02-17T22:53:10Z

/test?

k8s-ci-robot · 2022-02-17T22:53:12Z

@pydctw: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-aws-build
/test pull-cluster-api-provider-aws-test
/test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-blocking
/test pull-cluster-api-provider-aws-e2e-conformance
/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
/test pull-cluster-api-provider-aws-e2e-eks

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-aws-build
pull-cluster-api-provider-aws-test
pull-cluster-api-provider-aws-verify

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pydctw · 2022-02-17T22:53:34Z

/test pull-cluster-api-provider-aws-e2e-eks

pydctw · 2022-02-18T00:42:50Z

/hold

pydctw · 2022-02-18T00:59:27Z

The upgrade test actually passed.

• [SLOW TEST:2638.881 seconds]
EKS Cluster upgrade test
/home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/suites/managed/upgrade_test.go:37
  [managed] [upgrade] should create a cluster and upgrade the kubernetes version
  /home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/shared/common.go:179

pydctw · 2022-02-18T01:04:58Z

The failure is in the eks cluster test - it failed while trying to create a secret for AWSClusterStaticIdentity.

secrets \"e2e-account-creds\" already exists"

Are we supposed to set up AWSClusterStaticIdentity once and use for both upgrade and regular eks tests? I thought they should be set up per cluster but will do more investigation.

k8s-triage-robot · 2022-02-18T03:07:43Z

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

richardcase · 2022-02-18T08:06:51Z

Are we supposed to set up AWSClusterStaticIdentity once and use for both upgrade and regular eks tests? I thought they should be set up per cluster but will do more investigation.

They should be done once in the bootstrap cluster. So we probably need to move:

ginkgo.By("setting up AWS static credentials")
shared.SetupStaticCredentials(ctx, namespace, e2eCtx)

To the shared test setup. So somewhere in Node1BeforeSuite when the bootstrap cluster has been created.

We will also need to move the credential cleanup:

ginkgo.By("Deleting AWS static credentials")
shared.CleanupStaticCredentials(ctx, namespace, e2eCtx)

to the shared test teardown. So somewhere in Node1AfterSuite.

pydctw · 2022-02-23T22:04:31Z

/test pull-cluster-api-provider-aws-e2e-eks

pydctw · 2022-02-23T22:15:43Z

@richardcase, thanks for the idea. I have moved SetupStaticCredentials and CleanupStaticCredentials functions to suite.go so that it can be set up once and used for both tests.

Note that I refactored AWSClusterStaticIdentity functions as part of this.

Not to specify namespace for the object as it is a cluster-scoped object.
Allow all namespaces (AllowedNamespaces) so that it can be used for each test run with a newly created namespace.

pydctw · 2022-02-28T16:17:35Z

/hold cancel

test/e2e/shared/suite.go

test/e2e/shared/identity.go

richardcase · 2022-03-01T16:10:51Z

e2e test passed and from the logs we can see the upgrade test is running. So:

/lgtm
/approve

k8s-ci-robot · 2022-03-01T16:11:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pydctw · 2022-03-01T16:26:41Z

I0301 16:22:49.605924   22251 logr.go:252]  "msg"="Wait completed, proceeding to shutdown the manager"  
panic: Failed to stop envtest: timeout waiting for process kube-apiserver to stop

Looks likes a flake with envtest. Rerunning the test

/retest

sedefsavas · 2022-03-01T16:32:56Z

/hold

until above comment is addressed.

pydctw · 2022-03-01T16:34:49Z

pull-cluster-api-provider-aws-test passed but it's failing at pull-cluster-api-provider-aws-verify.

Looks like the CI env is slow or sth today. Failed at verify, which has never failed for the PR before.

Init container clonerefs not ready: (state: waiting, reason: "PodInitializing", message: "") Init container initupload not ready: (state: waiting, reason: "PodInitializing", message: "") Init container place-entrypoint not ready: (state: waiting, reason: "PodInitializing", message: "")

/retest

Update k8s version to EKS supported versions. Set up AWSClusterStaticIdentity for the managed test suite. A few refactoring of the code. - Not to specify namespace in AWSClusterStaticIdentity as it is a cluster-scoped object. - Delete DeleteClusterSpec helper func that could be replaced with CAPI framework function.

pydctw · 2022-03-01T17:27:28Z

/test pull-cluster-api-provider-aws-e2e-eks

pydctw · 2022-03-01T19:25:42Z

/test pull-cluster-api-provider-aws-verify

pydctw · 2022-03-01T19:43:01Z

Having an issue with the env today. pull-cluster-api-provider-aws-verify continues to fail. Will try again later.

Init container clonerefs not ready: (state: waiting, reason: "PodInitializing", message: "") Init container initupload not ready: (state: waiting, reason: "PodInitializing", message: "") Init container place-entrypoint not ready: (state: waiting, reason: "PodInitializing", message: "")

sedefsavas · 2022-03-01T20:14:13Z

/lgtm

pydctw · 2022-03-01T23:46:26Z

/test pull-cluster-api-provider-aws-verify

pydctw · 2022-03-01T23:56:36Z

All tests passed.

/hold cancel

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. needs-priority needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 17, 2022

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Feb 17, 2022

k8s-ci-robot requested review from dlipovetsky and dthorsen February 17, 2022 22:53

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 18, 2022

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 18, 2022

pydctw force-pushed the eks-upgrade-test branch from c2ba25f to bb0e874 Compare February 23, 2022 22:04

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 23, 2022

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 28, 2022

sedefsavas reviewed Mar 1, 2022

View reviewed changes

test/e2e/shared/suite.go Outdated Show resolved Hide resolved

richardcase reviewed Mar 1, 2022

View reviewed changes

test/e2e/shared/identity.go Show resolved Hide resolved

k8s-ci-robot assigned richardcase Mar 1, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 1, 2022

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 1, 2022

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 1, 2022

pydctw force-pushed the eks-upgrade-test branch from bb0e874 to 42918e2 Compare March 1, 2022 17:27

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 1, 2022

k8s-ci-robot assigned sedefsavas Mar 1, 2022

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 1, 2022

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 1, 2022

k8s-ci-robot merged commit 9a74bcc into kubernetes-sigs:main Mar 2, 2022

k8s-ci-robot added this to the v1.x milestone Mar 2, 2022

pydctw mentioned this pull request Apr 7, 2022

[e2e] Fix the EKS tests so we can enable the full suite #2239

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable EKS upgrade test #3229

Enable EKS upgrade test #3229

pydctw commented Feb 17, 2022 •

edited

k8s-ci-robot commented Feb 17, 2022

pydctw commented Feb 17, 2022 •

edited

k8s-ci-robot commented Feb 17, 2022

pydctw commented Feb 17, 2022

pydctw commented Feb 18, 2022

pydctw commented Feb 18, 2022

pydctw commented Feb 18, 2022

k8s-triage-robot commented Feb 18, 2022

richardcase commented Feb 18, 2022

pydctw commented Feb 23, 2022

pydctw commented Feb 23, 2022 •

edited

pydctw commented Feb 28, 2022

richardcase commented Mar 1, 2022

k8s-ci-robot commented Mar 1, 2022

pydctw commented Mar 1, 2022

sedefsavas commented Mar 1, 2022

pydctw commented Mar 1, 2022 •

edited

pydctw commented Mar 1, 2022

pydctw commented Mar 1, 2022

pydctw commented Mar 1, 2022

sedefsavas commented Mar 1, 2022

pydctw commented Mar 1, 2022

pydctw commented Mar 1, 2022

Enable EKS upgrade test #3229

Enable EKS upgrade test #3229

Conversation

pydctw commented Feb 17, 2022 • edited

k8s-ci-robot commented Feb 17, 2022

pydctw commented Feb 17, 2022 • edited

k8s-ci-robot commented Feb 17, 2022

pydctw commented Feb 17, 2022

pydctw commented Feb 18, 2022

pydctw commented Feb 18, 2022

pydctw commented Feb 18, 2022

k8s-triage-robot commented Feb 18, 2022

richardcase commented Feb 18, 2022

pydctw commented Feb 23, 2022

pydctw commented Feb 23, 2022 • edited

pydctw commented Feb 28, 2022

richardcase commented Mar 1, 2022

k8s-ci-robot commented Mar 1, 2022

pydctw commented Mar 1, 2022

sedefsavas commented Mar 1, 2022

pydctw commented Mar 1, 2022 • edited

pydctw commented Mar 1, 2022

pydctw commented Mar 1, 2022

pydctw commented Mar 1, 2022

sedefsavas commented Mar 1, 2022

pydctw commented Mar 1, 2022

pydctw commented Mar 1, 2022

pydctw commented Feb 17, 2022 •

edited

pydctw commented Feb 17, 2022 •

edited

pydctw commented Feb 23, 2022 •

edited

pydctw commented Mar 1, 2022 •

edited