Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad ingress object results in unstable HAProxy #135

Closed
tamalsaha opened this issue Jun 1, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@tamalsaha
Copy link
Member

commented Jun 1, 2017

Scenario 1: Ingress points to some services that don't exist.

Scenario 2: Ingress points to some services that points to pods that have bad docker image tag or docker registry facing outage. As a result the pods don't start and are in ImagePullBackoff state.

This seems to be causing the configmap to be reloading frequently. This eventually causes HAProxy to not serve any traffic.

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 1, 2017

daemon.info: Jun  1 21:18:25 reloader: Starting HAProxy configuration watcher and reloader ...
daemon.info: Jun  1 21:18:25 reloader: exec /kloader --config-map=voyager-haproxy --mount-location=/etc/haproxy --boot-cmd=/etc/sv/reloader/reload --v=4
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692403     664 logs.go:19] FLAG: --alsologtostderr="false"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692531     664 logs.go:19] FLAG: --boot-cmd="/etc/sv/reloader/reload"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692543     664 logs.go:19] FLAG: --config-map="voyager-haproxy"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692551     664 logs.go:19] FLAG: --k8s-config=""
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692558     664 logs.go:19] FLAG: --k8s-master=""
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692568     664 logs.go:19] FLAG: --log_backtrace_at=":0"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692576     664 logs.go:19] FLAG: --log_dir=""
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692582     664 logs.go:19] FLAG: --logtostderr="true"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692592     664 logs.go:19] FLAG: --mount-location="/etc/haproxy"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692599     664 logs.go:19] FLAG: --stderrthreshold="2"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692607     664 logs.go:19] FLAG: --v="4"
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.692615     664 logs.go:19] FLAG: --vmodule=""
daemon.err: Jun  1 21:18:25 reloader: W0601 21:18:25.692623     664 client_config.go:481] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
daemon.err: Jun  1 21:18:25 reloader: I0601 21:18:25.693766     664 main.go:32] Running ConfigMap Mounter for configMap voyager-haproxy
daemon.err: Jun  1 21:18:25 reloader: F0601 21:18:55.695319     664 mount.go:63] Failed to get ConfigMap, Cause Get https://10.0.0.1:443/api/v1/namespaces/default/configmaps/voyager-haproxy: dial tcp 10.0.0.1:443: i/o timeout
daemon.info: Jun  1 21:18:55 reloader: Starting HAProxy configuration watcher and reloader ...
daemon.info: Jun  1 21:18:55 reloader: exec /kloader --config-map=voyager-haproxy --mount-location=/etc/haproxy --boot-cmd=/etc/sv/reloader/reload --v=4
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793148     790 logs.go:19] FLAG: --alsologtostderr="false"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793260     790 logs.go:19] FLAG: --boot-cmd="/etc/sv/reloader/reload"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793272     790 logs.go:19] FLAG: --config-map="voyager-haproxy"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793280     790 logs.go:19] FLAG: --k8s-config=""
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793286     790 logs.go:19] FLAG: --k8s-master=""
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793297     790 logs.go:19] FLAG: --log_backtrace_at=":0"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793304     790 logs.go:19] FLAG: --log_dir=""
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793308     790 logs.go:19] FLAG: --logtostderr="true"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793313     790 logs.go:19] FLAG: --mount-location="/etc/haproxy"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793317     790 logs.go:19] FLAG: --stderrthreshold="2"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.793321     790 logs.go:19] FLAG: --v="4"
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.794846     790 logs.go:19] FLAG: --vmodule=""
daemon.err: Jun  1 21:18:55 reloader: W0601 21:18:55.794860     790 client_config.go:481] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
daemon.err: Jun  1 21:18:55 reloader: I0601 21:18:55.795206     790 main.go:32] Running ConfigMap Mounter for configMap voyager-haproxy
@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 2, 2017

Add 2 sub commands for kloader.

kloader check to check api server connection & kloader args. if this exits with 1, don't even start runit.

kloader watch is the process that runs as a runit process (current behavior).

@tamalsaha tamalsaha closed this Jun 6, 2017

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Jun 6, 2017

Fixed in #140

@tamalsaha tamalsaha added this to the 1.5.6 milestone Jun 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.