🌱 Fix self-hosted flakes in E2E tests

[fabriziopandini]

What this PR does / why we need it:
This PR aims to fix self-hosted flakes by ensuring that mhc is not run during this test and by adding two delay in order to avoid doing move too aggressively.

In order to make this happen the entire management of cluster templates was re-architected so:

• cluster templates for tests are now generated from a set composable bases
• MHC-remediation has a new dedicated template
• all the other templates are without MHC
• also, in order to simplify things, this PR drops the distinction between CI and DEV test config

On top of that, this PR includes, a set of nits for improving test logs (mostly uppercase)

Which issue(s) this PR fixes:
Fixes #3589

[fabriziopandini]

/milestone v0.3.10

[fabriziopandini]

/retest

[fabriziopandini]

/hold
investigating an error on ci-e2e.sh that shows up un CI only, while locally everything works just fine 🤔

[vincepri]

From the PR's description, seems we're now splitting manifests in order to separate MHC components? Would this be something that we would expect users to do as well?

During move, we should be able to pause controllers and look at the state of the cluster if it's safe to move, is there anything stopping us to do so?

Can we split this PR in multiple ones? In particular:

also, in order to simplify things, this PR drops the distinction between CI and DEV test config
On top of that, this PR includes, a set of nits for improving test logs (mostly uppercase)

These changes seems outside of the PR's scope

[sedefsavas]

Thanks for the fix!

This PR is also addressing this issue: #3461
I am fine with adding it in the title and the issues fixed by this PR or making this separate PRs.

[sedefsavas]

@fabriziopandini Locally, works for me as well.

/test pull-cluster-api-e2e

[sedefsavas]

In test/e2e/Makefile

cd $(TOOLS_DIR); go build -tags=tools -o $(BIN_DIR)/kustomize sigs.k8s.io/kustomize/kustomize/v3

should be

cd $(TOOLS_DIR) && go build -tags=tools -o $(BIN_DIR)/kustomize sigs.k8s.io/kustomize/kustomize/v3

[fabriziopandini]

@vincepri

From the PR's description, seems we're now splitting manifests in order to separate MHC components? Would this be something that we would expect users to do as well?

No. The current MHC object in E2E tests is specifically configured to always trigger remediation after 30s the node is started, and I removed it from the other e2e tests in order to avoid flakes due to remediation kicking in case of slow execution

During move, we should be able to pause controllers and look at the state of the cluster if it's safe to move, is there anything stopping us to do so?

This is not how the move logics works. Currently we are pausing reconciliation on clusters included in the Move scope, but the controller will continue to run.

Can we split this PR in multiple ones?

Done! rif
#3649
#3650
#3651

These changes seems outside of the PR's scope

Some of them are nits, but refactoring templates for MHC is potentially related to self-hosted flakes, see the note above about MHC and #3589 (comment).
However, no problem to move to another PR

[fabriziopandini]

/hold cancel
/test pull-cluster-api-e2e-full

[vincepri]

No. The current MHC object in E2E tests is specifically configured to always trigger remediation after 30s the node is started, and I removed it from the other e2e tests in order to avoid flakes due to remediation kicking in case of slow execution

I assume this change has moved to the other PR, I'll take a look there before commenting

This is not how the move logics works. Currently we are pausing reconciliation on clusters included in the Move scope, but the controller will continue to run.

We should probably consider stopping all controllers before move, for safety purposes and to avoid any running slow worker to perform actions while move is running.

[vincepri]

/test pull-cluster-api-e2e-full

[randomvariable]

/lgtm

pending e2e pass

[vincepri]

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment