Create a cross-provider dashboard for E2E tests

[fabriziopandini]

In order to get a better signal about CAPI and the growing ecosystem, we should create and new dashboard reporting results for a common set of tests executed on different providers.

As the initial scope for this effort IMO we should consider the following tests:

• Create an HA workload cluster - TBD if this should be a separated test or if it is included in the following
• Kubernetes conformance
• Kubernetes upgrade
• Provider upgrade (clusterctl upgrade)
  Repeated for CAPD, CAPA, CAPV, CAPZ.

Happy to consider a scope review/expansion according to the capacity we can redirect to this task.

/kind feature

[fabriziopandini]

/area testing
/milestone v0.3.10

[vincepri]

/priority important-soon
/milestone v0.4.0

Moving this to v1alpha4, given that we're closing v0.3.x soon

[fabriziopandini]

As per https://kubernetes.slack.com/archives/C8TSNPY4T/p1605628753355000,
Trying to figure out how to match this issue, with other E2E efforts (e.g. testing different K8s version, testing with K8s latest):

Prow constraints:

• Prow dashboard reports at job level (e.g. https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api)
• List of tests executed by each job is accessible only on the detail page (e.g. https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main)

The problem has three dimensions:

• Version of the management cluster, which can be simplified to the recommended K8s version only as per Document policies around Kubernetes and dependencies  #3783 (comment) (currently v1.19.x)
• Version of the workload cluster, AFAIK from v1.13 up to ci/latest, which can be simplified to the current 3 supported versions (as per https://kubernetes.io/docs/setup/release/version-skew-policy/) + ci/latest
• Version of CAPI, ideally from v0.3.0 to v0.4.0, which can be simplified to v.0.X.Y and v0.x.Y-1 + ci/latest for each Cluster API branch

The resulting job list is:

• 1 Periodic test for CAPI controller’s behaviors for CAPI main branch + 1 for the 0.3 branch (e.g KCP adoption, drain, self-hosting etc.)
• 4 Periodic tests for workload cluster conformance & upgrades with CAPI main branch + 4 with the 0.3 branch
  • upgrade v1.17 --> v1.18, conformance v1.18
  • upgrade v1.18 --> v1.19, conformance v1.19
  • upgrade v1.19 --> v1.20, conformance v1.20
  • upgrade v1.20 --> ci/latest, conformance ci/latest
• 2 Periodic tests for clusterctl upgrades within 0.3 branch
  • v0.3.10 --> v0.3.11
  • v0.3.11 --> latest/v0.3 branch
• 1 Periodic tests for clusterctl upgrades from v0.3 to v0.4
  • latest/v0.3 branch --> v0.4.0
• (as soon as possible) 1 Periodic tests for clusterctl upgrades within v0.4 branch

Considering the above Jobs, the following will go in the new cross provider - release informing dashboard for CAPI:

• 1 + 1 upgrade v1.19 --> v1.20, conformance v1.20 (CAPD Kubernetes upgrade, CAPD Kubernetes conformance)
• 1 + 1 CAPI controller’s behaviors (CAPI commons)
• 1 v0.3.11 --> latest/v0.3 branch (clusterctl upgrade v1alpha3)
• 1 latest/v0.3 branch --> v0.4.0 (clusterctl upgrade v1alpha3-v1alpha4)

Of the above tests, we do aspect the providers to implement only:

• 1 upgrade v1.19 --> v1.20, conformance v1.20", with CAPI main branch + 1 with 0.3 branch (CAP* Kubernetes upgrade, CAP* Kubernetes conformance)

The above provider's jobs will show up in the cross provider - release informing dashboard for CAPI; might be we should have 1 dashboard for main branch and one for the 0.3 branch.

@vincepri @CecileRobertMichon @randomvariable @yastij @detiber does this makes sense?

[CecileRobertMichon]

1 upgrade v1.19 --> v1.20, conformance v1.20", with CAPI main branch + 1 with 0.3 branch (CAP* Kubernetes upgrade, CAP* Kubernetes conformance)

just to clarify, when we say k8s upgrade, are we talking about KCP + MD + MP or just KCP (assuming it's the former)?

[fabriziopandini]

just to clarify, when we say k8s upgrade, are we talking about KCP + MD + MP or just KCP (assuming it's the former)?

Yes, testing K8s upgrades in all of its possible nuances

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fabriziopandini]

/lifecycle frozen

[vincepri]

Are we still interested in pursuing this issue?

/milestone Next

[fabriziopandini]

/close
We can reconsider this later on when we start discussion about conformance

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

/close
We can reconsider this later on when we start discussion about conformance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.