Proposal: Public IPs per Pod #446

slaskawi · 2017-03-13T08:34:44Z

Hey!

Please have a look at a design proposal for solving kubernetes/kubernetes#28660

It is very likely that the newly created interface for External IPs would also be affected by kubernetes/enhancements#88. However, because of an early stage of #88, it has not been included in this proposal.

Thanks,
Sebastian Łaskawiec

slaskawi · 2017-03-16T06:39:53Z

cc @bprashanth @enisoc @foxish @janetkuo @Kargakis @kow3ns @smarterclayton

justinsb · 2017-04-03T01:08:23Z

cc @kubernetes/sig-network-misc

thockin

I'm still not in favor of this, mostly because I don't understand how it will be implemented.

thockin · 2017-04-09T23:23:00Z

contributors/design-proposals/public-identities-for-pods.md

+* Last edit: [2017-03-13](#history)
+* Status: design
+
+Approvers:


please put me in this list

Sure, no problem.

thockin · 2017-04-09T23:26:37Z

contributors/design-proposals/public-identities-for-pods.md

+)
+```
+
+A new interface in `cloud.go` will be created to allow creating an external IP addresses.


CloudProvider is a mostly frozen API and is being disassembled. We should not rely on it being ubiquitous. It's fine as a model, though.

Thanks for letting me know.

Let me also explain myself a little bit. This is my first proposal to Kubernetes and don't have much hands-on experience with its code base (I work on Data Grid solutions on daily basis). Please forgive me any obvious mistakes that you might see. I'm also open for any suggestions to improve this proposal.

So as I mentioned in a comments a little bit higher, I wanted to give a chance to cloud vendors to implement a kind of External IP Allocator. This might be a much cheaper solution opposed to abusing Load Balancers. Unfortunately that would require adding some bits into cloud.go. Of course I might be totally wrong here and there might be a different mechanism for adding such capabilities. In that case, could you please advice me how can I achieve this?

The problem is that, at least for some clouds, an external IP == a load-balancer. We already have the primitives you need to get an IP assigned, without making a new API.

@kow3ns mentioned in the comment below about Elastic IP (AWS) and Static External IPs (GCP). Is it possible to use those things somehow?

thockin · 2017-04-09T23:32:00Z

contributors/design-proposals/public-identities-for-pods.md

+	// Implementations must treat the *v1.Service and *v1.Node
+	// parameters as read-only and not modify them.
+	// Parameter 'clusterName' is the name of the cluster as presented to kube-controller-manager
+	EnsureExternalIP(clusterName string, service *v1.Service, nodes []*v1.Node) (*v1.ExternalIPStatus, error)


I don't understand this API. What is the service argument for? Who writes a status back to Service? What is the nodes argument for? How do I know which backend pod this IP is supposed to be for?

I am having a hard time imagining the implementation of this..

I think I made a mistake here. I wanted to pass a clusterName, a Service (which governs a clustered app deployed in Kubernetes, such as Data Grid) and a list of Pod (not Nodes, I'm sorry). I'm pretty sure the cloud vendor could easily extract a list of Pods from governing Service, so let me remove the last argument.

The reason for returning a status here is that some cloud vendors might not implement this api and we will return an error here. Then we could fall back to abusing Load Balancers for the instance.

thockin · 2017-04-09T23:33:30Z

contributors/design-proposals/public-identities-for-pods.md

+Note that there is no `UpdateExternalIP` function in `ExternalIP` interface. This means that a single IP address will not
+be reassigned into another Pod. It will be explicitly deleted (by `EnsureExternalIPDeleted`).
+
+Since `ExternalIP` interface might not be implemented and supported by all Cloud Vendors, we allow creating a Load Balancer


Is it implementable in ANY provider? GCP's ForwardingRule is close, but it is a load-balancer underneath. ELB is a non-transparent LB, which depends on nodePorts (which are not addressed in this doc at all).

I would kindly ask for your advice here. I imagine this could be implemented very easily by a pool of externally available IP addresses and a simple allocator. But my (very naive) vision might not be implementable. What do you think?

It doesn't map to any cloud I know. I get what you're trying to express but it is more complicated than you imagine. It will required either iptables rules, routing rules, or node ports (or combinations thereof) - which is exactly what Service provides.

thockin · 2017-04-09T23:33:40Z

contributors/design-proposals/public-identities-for-pods.md

+be reassigned into another Pod. It will be explicitly deleted (by `EnsureExternalIPDeleted`).
+
+Since `ExternalIP` interface might not be implemented and supported by all Cloud Vendors, we allow creating a Load Balancer
+per Pod. This should however be considered as a last-resort approach. In that case, a user will need to add


Why is it a last resort?

My guess would be $$$. LBs are not cheap.

I don't know what the first-resort is - I'm looking for even one case where this is implemntable in a way that doesn't exactly equal Services.

caseydavenport · 2017-04-04T00:11:03Z

contributors/design-proposals/public-identities-for-pods.md

+* [Behavior](#behavior)
+* [Upgrading](#upgrading)
+* [Implementation](#implementation)
+* [Alternatives](#alternatives)


I don't see this section below :)

Sorry, I think I messed something up when editing the file. Let me add it.

caseydavenport · 2017-04-04T00:12:06Z

contributors/design-proposals/public-identities-for-pods.md

+  * Data Grids (with L7 routing, such as [Infinispan](http://infinispan.org/docs/dev/user_guide/user_guide.html#using_hot_rod_server))
+  * Media Servers
+
+* The secondary goal is to create an API similar to `LoadBalancer` to manage external IP addresses.


I'm not so sure this is a "goal" so much as a potential implementation of public IP addresses for Pods, right?

Yes and no. I wanted to propose a similar API to Load Balancers and give cloud vendors a chance to allocate an IP address available externally. I imagine some of them (cloud vendors) might not want to abuse Load Balancers (they might be expensive and @smarterclayton mentioned that checking their health might be tricky).

philips · 2017-04-10T22:10:01Z

I want to echo @thockin's sentiment: I don't get this.

It would be great if a section could be written about the alternatives that could be considered for this proposal and why they are untenable for your use case.

slaskawi · 2017-04-26T13:28:06Z

@philips Thanks for the feedback. The alternatives section has been added.

slaskawi · 2017-04-26T13:34:45Z

I hope I address most of the concerns in my comments. I also added alternatives section which was missing.

Just to let you know, I also started to work on a proof-of-concept which allocates a Load Balancer per Pod without touching Kubernetes codebase. A very dirty code might be found here. The demo creates an additional binary proxy between Load Balancer and target Pod (I required it for some testing and will be removing it soon).

psycotica0-shopify · 2017-05-02T21:39:23Z

I have also written a controller that creates a LoadBalancer service for each pod, scoped by adding an ordinal tag to each pod.
It works, but it'd be cool if the system had something built in for it.

My use case is Kafka, which is really great with statefulsets, but users of the system must be able to individually address brokers. Can't just throw them all behind a simple LB.

kow3ns · 2017-05-03T00:49:05Z

@slaskawi are you proposing to provision elastic IPs (AWS) or ephemeral/static external IPs (GCP) as opposed to LBs on a per Pod basis and to forward traffic from the external address to the Pod?

psycotica0-shopify · 2017-05-03T15:56:52Z

Not to hijack the proposal, but if there are concerns that this might be similar to Services, what if there was a field on StatefulSet called serviceTemplate, similar to volumeClaimTemplates where you can put a service that will be created and scoped to each pod individually.

Then, if you want to use an LB, you can put that as the type in the spec.
But, if some cloud provides a service type that represents a route from ExternalIP directly to a pod somehow, then you can put that there instead?

Service spec already allows specifying a static IP from a pool you've already got allocated. (Though indexing from the generated service into that might be a little tricky compared to how things already work... I don't know if there are patterns for this)

slaskawi · 2017-05-09T06:51:37Z

@kow3ns Yes, my initial though was to create a new interface for cloud provider which would allow them to provision an Elastic IPs or Static External IPs. But it seems there might be problems to implement it on cloud provider side. See #446 (comment) and #446 (comment)

@psycotica0-shopify I actually think those serviceTemplates could really do the job here. Plus, as you mentioned, they would look consistent to volumeClaimTemplatess. @thockin convinced me that we should be really looking more into load balancer-based solution rather than trying to expose an External IP (please correct me if I got it wrong). So adding those two together seems an idea to move this design forward.

thockin · 2017-05-09T16:09:21Z

A GCP External IP *is* a load-balancer. There's really no difference. I am not deeply familiar with how AWS elastic IPs work, but to get a VM-oriented IP into a container requires some work in the node beyond just assigning an IP. That's part of why I don't like this - we already do that work, it's called Service. If you want to explore it, I can't stop you. I can say that I don't think we want a generic construct of external IP for pods. This is why Kubernetes has (and is adding more) extension points. You should be able to write a controller and per-node agent to manage this all.

…

On Mon, May 8, 2017 at 11:56 PM, Sebastian Łaskawiec < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In contributors/design-proposals/public-identities-for-pods.md <#446 (comment)>: > +A new type of Service needs to be created in order to indicate that an external IP address will be exposed per Pod: + +```go +const ( + //Existing Service types + ServiceTypeClusterIP ServiceType = "ClusterIP" + ServiceTypeNodePort ServiceType = "NodePort" + ServiceTypeLoadBalancer ServiceType = "LoadBalancer" + ServiceTypeExternalName ServiceType = "ExternalName" + + //new Service type - expose IP per Pod + ServiceTypeIPPerPod ServiceType = "IPPerPod" +) +``` + +A new interface in `cloud.go` will be created to allow creating an external IP addresses. @kow3ns <https://github.com/kow3ns> mentioned in the comment below about Elastic IP (AWS) and Static External IPs (GCP). Is it possible to use those things somehow? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#446 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVIw0vR8MW02fqQQLKNWz0ZBr3uO6ks5r4A4sgaJpZM4Ma99P> .

slaskawi · 2017-05-22T11:12:05Z

I think I collected enough feedback to summarize this discussion and design. Thank you all for lots and lots of interesting comments.

So the bottom line is that an External IP is a Load Balancer for most of the cloud vendors. So there is no need to duplicate this sort of functionality.

Just before I close this Pull Request, I would like to ask one last question. @thockin Could you please tell me if there are any plans for implementing something similar to serviceTemplates that @psycotica0-shopify proposed? If not I would be happy to investigate this approach.

thockin · 2017-05-22T19:37:48Z

I don't think there are PLANS per se, but it's not the first time it has come up, so maybe worth exploring with sig-apps (StatefulSet)

slaskawi · 2017-05-23T11:55:17Z

@thockin Ok, great! Thanks for all the guidance!

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 13, 2017

slaskawi mentioned this pull request Mar 13, 2017

public network identities for pets kubernetes/kubernetes#28660

Closed

smarterclayton changed the title ~~Public identities per Pods design~~ Public network IPs per Pods design Apr 3, 2017

smarterclayton changed the title ~~Public network IPs per Pods design~~ Proposal: Public network IPs per Pod Apr 3, 2017

smarterclayton changed the title ~~Proposal: Public network IPs per Pod~~ Proposal: Public IPs per Pod Apr 3, 2017

kow3ns mentioned this pull request Apr 6, 2017

StatefulSet pods lack name label kubernetes/kubernetes#44103

Closed

thockin reviewed Apr 9, 2017

View reviewed changes

caseydavenport reviewed Apr 10, 2017

View reviewed changes

Public identities per Pods design

484f5c5

slaskawi force-pushed the public_network_identities_for_pods branch from 143f27e to 484f5c5 Compare April 26, 2017 13:27

slaskawi mentioned this pull request May 8, 2017

Ingress and TCP kubernetes/kubernetes#23291

Closed

thockin closed this May 22, 2017

danehans pushed a commit to danehans/community that referenced this pull request Jul 18, 2023

Fix dead link (kubernetes#446)

4d53bf7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal: Public IPs per Pod #446

Proposal: Public IPs per Pod #446

slaskawi commented Mar 13, 2017

slaskawi commented Mar 16, 2017

justinsb commented Apr 3, 2017

thockin left a comment

thockin Apr 9, 2017

slaskawi Apr 26, 2017

thockin Apr 9, 2017

slaskawi Apr 26, 2017

thockin May 2, 2017

slaskawi May 9, 2017

thockin Apr 9, 2017

slaskawi Apr 26, 2017

thockin Apr 9, 2017

slaskawi Apr 26, 2017

thockin May 2, 2017

thockin Apr 9, 2017

kow3ns Apr 12, 2017

slaskawi Apr 26, 2017

thockin May 2, 2017

caseydavenport Apr 4, 2017

slaskawi Apr 26, 2017

caseydavenport Apr 4, 2017

slaskawi Apr 26, 2017

philips commented Apr 10, 2017

slaskawi commented Apr 26, 2017

slaskawi commented Apr 26, 2017

psycotica0-shopify commented May 2, 2017

kow3ns commented May 3, 2017

psycotica0-shopify commented May 3, 2017

slaskawi commented May 9, 2017

thockin commented May 9, 2017 via email

slaskawi commented May 22, 2017

thockin commented May 22, 2017

slaskawi commented May 23, 2017

Proposal: Public IPs per Pod #446

Proposal: Public IPs per Pod #446

Conversation

slaskawi commented Mar 13, 2017

slaskawi commented Mar 16, 2017

justinsb commented Apr 3, 2017

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philips commented Apr 10, 2017

slaskawi commented Apr 26, 2017

slaskawi commented Apr 26, 2017

psycotica0-shopify commented May 2, 2017

kow3ns commented May 3, 2017

psycotica0-shopify commented May 3, 2017

slaskawi commented May 9, 2017

thockin commented May 9, 2017 via email

slaskawi commented May 22, 2017

thockin commented May 22, 2017

slaskawi commented May 23, 2017