Scaling clusters: 1000's of services in env vars

[lavalamp]

Clearly you ought to be able to make more services in a k8s cluster than it is reasonable to pass to pods in env vars.

Possible solutions:

• Segment by namespace
• Require predeclarations

@bgrant0607 I know you hate env vars; do you have a preferred solution for this?

[thockin]

If we move env-var generation down to kubelet, this is less awful :)

[bgrant0607]

My preferred solution is to eliminate the env vars. :-) That would also eliminate start-up-order dependencies (assuming clients could tolerate DNS lookup failures). I think even Docker buys that links aren't an acceptable long-term approach.

Barring that, segmentation by namespace seems like it should work if people use namespaces as intended.

We can reconsider predeclaration if neither of those approaches work.

Didn't @erictune already move the env-var generation to kubelet?

Note that we'll have scaling problems with iptables rules, too.

[thockin]

I tried to measure the impact of O(thousands) of iptables rules, but could
not find a decent way to measure it. It did not affect latency or
throughput in any obvious way.

On Thu, Jan 8, 2015 at 5:05 PM, bgrant0607 notifications@github.com wrote:

My preferred solution is to eliminate the env vars. :-) That would also
eliminate start-up-order dependencies (assuming clients could tolerate DNS
lookup failures). I think even Docker buys that links aren't an acceptable
long-term approach.

Barring that, segmentation by namespace seems like it should work if
people use namespaces as intended.

We can reconsider predeclaration if neither of those approaches work.

Didn't @erictune https://github.com/erictune already move the env-var
generation to kubelet?

Note that we'll have scaling problems with iptables rules, too.

Reply to this email directly or view it on GitHub
#3345 (comment)
.

[brendandburns]

Now that we have DNS, I think it's reasonable to consider deprecating the
environment variables.

On Thu, Jan 8, 2015 at 7:58 PM, Tim Hockin notifications@github.com wrote:

I tried to measure the impact of O(thousands) of iptables rules, but could
not find a decent way to measure it. It did not affect latency or
throughput in any obvious way.

On Thu, Jan 8, 2015 at 5:05 PM, bgrant0607 notifications@github.com
wrote:

My preferred solution is to eliminate the env vars. :-) That would also
eliminate start-up-order dependencies (assuming clients could tolerate
DNS
lookup failures). I think even Docker buys that links aren't an
acceptable
long-term approach.

Barring that, segmentation by namespace seems like it should work if
people use namespaces as intended.

We can reconsider predeclaration if neither of those approaches work.

Didn't @erictune https://github.com/erictune already move the env-var
generation to kubelet?

Note that we'll have scaling problems with iptables rules, too.

Reply to this email directly or view it on GitHub
<
#3345 (comment)

.

—
Reply to this email directly or view it on GitHub
#3345 (comment)
.

[bgrant0607]

Docker's networking proposal (Docker issue 9983 -- not linked to avoid noise on that issue) confirms that links will be deprecated in favor or "network-based discovery".

[derekwaynecarr]

@pmorie - fyi

[pmorie]

@lavalamp @bgrant0607 @thockin @derekwaynecarr I think predeclaration should be possible but you should also be able to opt out of it and get metadata / firewall rules for all visible services in your namespace

[lavalamp]

Also note that before we get rid of env vars, we have to solve a bootstrapping problem-- env vars are how SkyDNS knows where to find k8s master.

[bgrant0607]

Bootstrapping could be resolved by the downward API #386, or by creating a magic link-local address for the master.

[pmorie]

@lavalamp regarding the skydns use case, I think there's an orthogonal use-case to envs that exists, which is being able to state that a pod shouldn't start unless some services it depends on are ready.

[erictune]

Services could later go down. Clients need to handle that. So there is
nothing but a false sense of correctness in offering a way for pod start to
block on a service ready
On Jan 12, 2015 2:55 PM, "Paul Morie" notifications@github.com wrote:

@lavalamp https://github.com/lavalamp regarding the skydns use case, I
think there's an orthogonal use-case to envs that exists, which is being
able to state that a pod shouldn't start unless some services it depends on
are ready.

—
Reply to this email directly or view it on GitHub
#3345 (comment)
.

[stp-ip]

I agree unavailable dependencies need to be handled on the client.
To not make an easy workaround, which ends in things as "but it worked as I defined service1 starts before service2", I would not provide dependency declarations/support yet. There are more concerning issues at hand and furthermore I'm not even sure, if dependencies should be handled by kubernetes.

[pmorie]

@erictune @stp-ip Services can go down, and you won't even know if your pod started before an IP/port was allocated to an interesting service :)

[bgrant0607]

We definitely need to segment by namespace.

We've also basically settled on predeclaration for people who want env. vars.: #1768.