Repository navigation
Allow Service NodePorts on low ports #9995
Description
To capture some of the discussion.
Gripe: Service NodePorts can only be in a "special" port range (apiserver flag). This is annoying for people who want to expose a service on port 80 (for example) but instead have to use something like 30986.
Rationale for this behavior:
- We don't want service node ports to tromp on real ports used by the node (e.g. 22).
- We don't want service node ports to tromp on pod host ports.
- We don't want to randomly allocate someone port 80 or 443 or 22.
Proposed compromise: Allow NodePorts on low ports with a bunch of caveats.
To address rationale points (1) and (2): Make kube-proxy open (and hold open) ports that it is using. This will prevent it from using port 22 (for example) for a service node port.
To address rationale point (3): Use the flag-configured range for random allocations, but allow users to request any port they want.
Caveats and analysis:
-
Error reporting is not good. We do not know about what non-container stuff is using ports on the host and we do not have an easy way for the API server to allocate ports between pods and services. Not easy to resolve. The implication of this is that port-in-use errors will only be detected by kube-proxy when it tries to open the port on the node. Sending events for this is possible, but not great (one event from every node) and no other node-level error gets a kube-proxy event. Net result: user asked for a service on node port 22, and it just doesn't work and they have no idea why.
-
Doing dual-ranges (use the flag range for allocations but allow the whole range to be used) is non-trivial and has to be plumbed through more code than I am comfortable with at this point. The implication of not doing this is that sometimes people will get allocated a port that happens to be 22 and can never work. Combined with caveat (1) this is really unpleasant. We could do some REALLY hacky things like just retry the random allocation if it is not in the flagged range. This avoids plumbing the dual-range logic down, but is embarassingly ugly. I am ashamed to have suggested it.
-
Holding ports open is pretty easy and we should do this anyway. The error reporting properties are still not good.
Summary: I am unconvinced that this is worthwhile to rush into v1. We still have the option of baking a container that receives traffic on a HostPort and redirects it to a service. I have a demo of using socat to do this (simulating a pod manually):
root@kubernetes-minion-nnx6:/home/thockin# docker run -d --dns=10.0.0.10 --dns-search=default.svc.cluster.local -p 8080:93 thockin/hostport-to-service 93 hostnames2:80
1dcc1e94c30834290ae243ac298c6699b2a3348fc014b4b77ae34c13ead44854
root@kubernetes-minion-nnx6:/home/thockin# curl localhost:8080
hostnames2-gxt4f
Activity
thockin commentedon Jun 18, 2015
Priority to be decided - some argue that it is P0.
thockin commentedon Jun 18, 2015
I hope my summary wasn't TOO slanted. There are legitimate issues that
this sort of change would fix.
On Jun 17, 2015 6:16 PM, "Quinton Hoole" notifications@github.com wrote:
thockin commentedon Jun 18, 2015
That is one of the legitimate use cases, though no user is claiming this
one is a hard requirement. The biggest issue is simply friction and "kick
the tires" operation.
On Jun 17, 2015 7:23 PM, "Quinton Hoole" notifications@github.com wrote:
eparis commentedon Jun 18, 2015
A great example would be the L7 ingress router used by openshift. It needs to run on 80 and 443. Quite reasonable and not at all a 'kick-the-tires' kind of thing.
While hostPort might be possible it does mean that I can't use rc's any more, as I have to start pinning the ingress router container to specific nodes. With publicIPs this was solved, as externally people could be told to use publicIP[0]:443 and everything would be ok as the ingress router could still be managed by the cluster.
(actually for some cert checking reasons we've had to bastardize the ingress router, but it worked really well on 80)
thockin commentedon Jun 19, 2015
@eparis I don't think I understand your model. You run L7-as-a-service and you tell people to connect to $node_ip on :80 and :443, rather than a load balancer? And you're confident that no user of the cluster will ever ask for a Service with port 80 or 443 and ask for NodePort of the same port?
If so, you could change the apiserver's --service-node-port-range to "1-32767" and install your L7 before anyone "randomly" claims :80 or :443.
To reiterate, we WANT to enable this, but there's not time to do it properly before 1.0 - if you're comfortable with the caveats above, you can fix this today. My argument is that it should not default to "ridiculously brittle" mode and that the changes to make it not-brittle are too much churn for this point in the release cycle.
eparis commentedon Jun 19, 2015
--service-node-port-range=1-32767 suffers extremely from accidentally stomping on port 22, not someone screwed up, the SYSTEM screwed up....
eparis commentedon Jun 19, 2015
@thockin some people do have an external load balancer to terminate ssl and load balance to pods, which operates with access to the overlay network. Those people have better scalability for sure. But we provide a simple 'load balancer', running on ${some_node_ips} :80 :443 (nodePort would mean $all_node_ips).
We have no controller for nodePort yet, but yes, the plan is to outright reject a service if a 'normal' user specified anything other than 0.
To be honest, it does make me wonder what we do today if someone asks for hostPort=22. I think we just outright reject hostPort, but I don't know for sure...
22 remaining items
rektide commentedon Sep 26, 2020
This seems to be the only resistance I've heard to this & I don't understand the complaint. It wasn't elaborated or explained.
This issue seems like a straightforward way to host a range of services easily that currently are a huge pain to deal with (like http) while maintaining current functionality for most. This should be re-opened. I don't understand why hosting HTTP or DNS is so very very difficult for small operators right now.
paulmiu commentedon Oct 3, 2020
@liggitt @wojtek-t @dims @justaugustus (Sorry for mentioning you all, I just randomly picked still active contributors.)
If you're not responsible for this part of k8s, but know who is, please include that person in this conversation.
Thanks.
liggitt commentedon Oct 3, 2020
@thockin can reopen it if desired and has good context on why it would be a good or bad idea to do so. Adding this topic to the next sig-network agenda would be a good way to discuss it
paulmiu commentedon Oct 3, 2020
@liggitt Thanks for your quick response!
Can you include it in the sig-network agenda (or if I am able to do that too, how can I add this topic to it)?
liggitt commentedon Oct 3, 2020
The time and agenda info is at https://github.com/kubernetes/community/tree/master/sig-network
Joining the kubernetes-dev or kubernetes-sig-network mailing lists should get you access to those
paulmiu commentedon Oct 3, 2020
Thanks, I'll do that.
Edit:
It's now on the agenda for the next SIG Network Meeting. I've also sent an email to the mailing list.
denizdogan commentedon Nov 17, 2020
Was it ever discussed by the SIG? I'm assuming no, based on the note from October 15 that says:
fuomag9 commentedon Mar 2, 2021
Is there any update on this? I find it very illogical not to be able to open ports different from port 22!
theAkito commentedon Mar 2, 2021
I think it is important to stay fair to all parties in a controversial discussion, which is why I would like you to clarify what you are talking about. This has nothing to do with Port 22 (default SSH) port, as in the whole point of the special NodePort range is that ports such as the one for SSH do not get allocated.
All that said, it is best practice to change the default SSH port to something other than 22, anyway.
fuomag9 commentedon Mar 2, 2021
I probably bad worded it, I meant it in the "don't conflict with existing stuff" but still allow a lower number in the range of ports to be allocated by NodePort
BenTheElder commentedon Mar 2, 2021
I just want to point out that this is actually up to your cluster operator, the node port range has long been configurable
with the
--service-node-port-rangeoption for kube-apiserver. https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/Also this old comment:
Unless I've completely misunderstood the intent here ... You can absolutely do that. It still has to be within the configured range though. Set the
nodePortfield on the ports entry.https://kubernetes.io/docs/concepts/services-networking/service/#nodeport
SIG Network still owns this. Any community member has the ability to reopen, but personally I would still defer to the network special interest group. I've sent a poke to them.
aojea commentedon Mar 3, 2021
This was answered in the mailing list too
https://groups.google.com/g/kubernetes-sig-network/c/Ihq_1AslEMQ/m/oFiOxOJcAQAJ
I think that this will be reopened as a bug if
--service-node-port-rangecan not select a different nodeport range or, if it can, those doesn't workwzrdtales commentedon Mar 3, 2021
@BenTheElder also wondering why this pops up in my mails again and again your interpretation is right, that is what this issue was about. changing the node port range is supported. I guess people that hit on that problem find in google instead of the documentation this issue first.
thockin commentedon Mar 5, 2021
If you set the service-node-port-range to include 22, then that means 22 might be randomly allocated to any Service, even if you have SSH running and want to keep it running. BAD.
One option discussed a long time back was to allow low ports to be EXPLICITLY requested but not allocated. But then what governs who may request low ports? And how do you block off the ports that you don't want anyone to use?
TL;DR: this seems like a lot of work and design for something that isn't really that common. You can use a pod hostPort to claim a low port, but only on a single node. That's better, but has many of the same problems - governance, collisions, exclusions.
NodePorts were designed to build LBs on top of. They are not supposed to be very friendly to humans because humans are not the target audience,