capo: Failure to add kubeapi server members to LB

[garloff]

In my testing with cluster-API, I have seen several times that control nodes that were spun up successfully by capo did not show up as backend members in the loadbalancer's pool that is in front of the kubernetes API.
Manually adding them quickly would help the cluster creation to succeed. (The health check does test and mark the members ONLINE and the further cluster roll-out succeeds.)
If that is not done, the control-plane machines go into a failed state and the cluster needs manual intervention to recover :-(
This is with capi-1.1.5 and capo-0.6.2.

This needs investigation:

• Why is the member not registered? Does capo try and fail (e.g. on an PENDING_UPDATE LB state)?
• Why is this not reconciled? A totally healthy LB that lacks a member can easily be added one, no difficult considerations here ...

[garloff]

Adding @batistein, @janiskemper, @curx, @flyersa just in case they have seen this happening before and want to add color.

One additional side-note: This seems to be a new phenomenon -- I have not observed this behavior with capi-1.0.5+capo-0.5.3 which we had in R2.

[garloff]

Well, the test environment has this "LB and VMs sometimes takes 5 additional minutes" right now, capi/capo do not seem to be very robust against this.

[batistein]

This is unfortunately only due to capo... we should definitely put resources into fixing the known issues in capo.