OpenStack: HA proxy on service vm switches back and forth from/to bootstrap and production cluster

[jomeier]

Hi,

What happened?

On the api VM with HAProxy and CoreDNS after around 10-15 Minutes HAProxy starts continuously to restart.

It looks like the haproxy-watcher triggers a switch over from the bootstrap node to the production cluster too early.

In my case the production control plane wasn't ready. The API server hasn't started leading to a broken install.

If I turn off the haproxy-watcher, I get almost through the installation procedure.

What you expected to happen?

HAProxy should only switch to production cluster if the control plane is really stable.

Greetings,

Josef

Version

$ openshift-install version
./openshift-install unreleased-master-1112-g7a54a6b120a0044cf9c07b14331760af8e3c216b-dirty
built from commit 7a54a6b120a0044cf9c07b14331760af8e3c216b
release image registry.svc.ci.openshift.org/origin/release:4.2

Platform (aws|libvirt|openstack):

OpenStack (Stein), 4GByte RAM, 2 VCPUs, 60 GByte Disk

[jomeier]

[core@test-cluster-njrtc-api ~]$ sudo podman run --rm -ti --net=host -v /etc/haproxy:/usr/local/etc/haproxy:ro docker.io/library/haproxy:1.7
<7>haproxy-systemd-wrapper: executing /usr/local/sbin/haproxy -p /run/haproxy.pid -db -f /usr/local/etc/haproxy/haproxy.cfg -Ds 
[WARNING] 165/051754 (9) : config : missing timeouts for proxy 'test-cluster-njrtc-api-masters'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 165/051754 (9) : config : missing timeouts for proxy 'test-cluster-njrtc-api-workers'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.

[jomeier]

Maybe it has something to do with the haproxy-watcher service. Maybe the watcher script gets some nonsense back from the unhealthy cluster which triggers stuff (new haproxy config, restart of the haproxy service, ...) which is not wanted at that point in time?

[mandre]

Yes, I also noticed the haproxy-watcher service does weird stuff that could cause back and forth between the bootstrap and the production control plane. In my experience though, the LB setup will eventually converge to the production control plane once it is stable.
FYI we're in the process of removing the API VM (since it's single point of failure), and move the LB service to the master nodes. This part is going to be re-worked.

[bjhuangr]

@mandre guess DNS service will also moved to master nodes once API VM removed?

[mandre]

@mandre guess DNS service will also moved to master nodes once API VM removed?

That's correct, we'll be running haproxy, keepalived and coredns as static pods on the master nodes.

[mandre]

/label platform/openstack

[mandre]

/close

With #1959 we've removed the service VM that was causing the back and forth.

[openshift-ci-robot]

@mandre: Closing this issue.

In response to this:

/close

With #1959 we've removed the service VM that was causing the back and forth.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.