Updating controller default to always use the LB - endpoint.

[timothysc]

Currently we default controllers to connect to their masters on localhost:

 --public-master='https://localhost:8443': The master address for use by public clients, if different (host, host:port, or URL). Defaults to same as --master.

In an HA loaded cluster this is actually the worst possible situation in terms of cpu and network load, and can cause overall system instability on a fail-over.

xref:
kubernetes/kubernetes#22669
openshift/origin#7845

The controllers should always default to hit the load-balanced endpoint if the parameter is available.

/cc @jeremyeder @rrati

[detiber]

@sdodson We will need to separate the configs for the controllers and the api servers because of this... currently because we share the config (mainly the loopback kubeconfig) we can't set the loopback context for just the api service (which we want to connect locally for bootstrapping) and not the controllers service (which should connect to the lb address).

Also, we may want to wait on the api service being available at the lb address before starting the controllers service.

[timothysc]

Quoting @sdodson

"Aside from the scalability issues raised elsewhere, the components fail independently, so API server could be dead on localhost but available on other hosts and via the load balancer, wouldn't we want to take advantage of the other api hosts availability?"

[detiber]

So, my current thoughts on this....

We can create a controllers subdirectory under /etc/origin/master and hardlink all of the relevant master config files, with the exception of the loopbackKubeconfig, which can ensure is set to the original default context.

[timothysc]

Couldn't we just over-ride the unit file for controllers is the lb argument is present? Does the unit file command line override usurp config file, I would hope so.

[detiber]

On Fri, Mar 11, 2016 at 10:29 AM, Timothy St. Clair <
notifications@github.com> wrote:

Couldn't we just over-ride the unit file for controllers is the lb
argument is present? Does the unit file command line override usurp config
file, I would hope so.

In general, I believe that is the case, but there doesn't seem to be a way
to do so for the masterClients.openshiftLoopbackKubeConfig, which is the
value we would need to override. I don't think that using the --kubernetes
flag would be sufficient for all of the controllers
(masterClients.externalKubernetesKubeConfig)

—
Reply to this email directly or view it on GitHub
#1563 (comment)
.

Jason DeTiberus

[detiber]

@deads2k @liggitt What are the chances of exposing the ability to set masterClients.openshiftLoopbackKubeConfig from the command line for either the api server or controllers server? Otherwise, it looks like we are going to have to replicate config instead of being able to rely on the centralized master config file like we are currently doing now.

We are currently overriding the listen option on both api/controllers services and the --master option on the api service. All other values are coming from the common config/kubeconfigs.

It looks like the only option I see is the --kubeconfig option, which seems to map to masterClients.externalKubernetesKubeConfig.

[deads2k]

@detiber I really don't want to end up in cases where I half run from config and half run from command line arguments.

@liggitt I see StringSource and env var (less than perfect since we'd have two different envs for the same config on the same system) or new value in the masterConfig for "loopback through load balancer (for controllers)" or some such.

[detiber]

@deads2k we are already doing that with the --master and --listen options, but I'm all for unifying whatever approach is best.

ENV vars wouldn't be a problem, since each is running under systemd with different env files (which is where the current command line options are listed)