build: do not create kind clusters in parallel

[stefansedich]

Noticing the test flakes and saw the discussion in kubernetes-sigs/kind#1288 changing to 2 parallel jobs to see if that has any effect on stability of the e2e suite in CI, @hiddeco any objection to trying this out?

I also factored the e2e tests out into a CircleCI command which IMO cleans things up a little but required an update to the 2.1 build (which should not break anything else).

[stefansedich]

Looks like 6 When rollback.maxRetries is set to 1, upgrade is only retried once is giving me grief but looking at previous builds looks like that test has been known to do that.

[hiddeco]

@stefansedich looks like changing this to 2 clusters has made the flakiness of the test persistent, while this is great to debug what goes wrong, it doesn't help to reduce the flakiness of the cluster bootstrapping.

[hiddeco]

For the (public) record: persistent failure happens due to Tiller not being properly cleaned up in the teardown, as helm reset --force deletes the instance but keeps the releases.

Two options:

1. Remove all releases before nuking Tiller
2. Install Tiller into the same namespace as the operator is in, this would require a bit more wiring but would give the advantage of testing the Tiller configuration flags while also making the teardown itself easier as the namespace can just be nuked.

[stefansedich]

Have gone for option #2, pushed and 🤞 for a solid CI experience.

[stefansedich]

I guess this isn't really needed as we nuke the namespace but it does not hurt.

[hiddeco]

This function is not aware of how it is utilized and should simply undo what its install companion creates, so this is perfect.

[hiddeco]

This function is not aware of how it is utilized and should simply undo what its install companion creates, so this is perfect.

[hiddeco]

Can you add a note here about the --force flag, and the fact that this only resets the Tiller instance and not the release storage.