Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Services with same port, different protocol display wrongly in kubectl and have wrong merge key #39188

Closed
thockin opened this issue Dec 23, 2016 · 75 comments
Labels
area/api Indicates an issue on api area. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.

Comments

@thockin
Copy link
Member

thockin commented Dec 23, 2016

User reported:

I am running a service with both TCP and UDP:

spec:
  type: NodePort
  ports:
    - protocol: UDP
      port: 30420
      nodePort: 30420
    - protocol: TCP
      port: 30420
      nodePort: 30420

but kubectl describe service shows only UDP.

Type:			NodePort
IP:			10.0.13.152
Port:			<unset>	30420/UDP
NodePort:		<unset>	30420/UDP
Endpoints:		10.244.4.49:30420

When I change the order then it shows only TCP.

This looks like using the wrong mergeKey, and indeed the code backs that up:

2508 type ServiceSpec struct {
2509 // The list of ports that are exposed by this service.
2510 // More info: http://kubernetes.io/docs/user-guide/services#virtual-ips-and-service-proxies
2511 Ports []ServicePort json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"port" protobuf:"bytes,1,rep,name=ports"

The key should probably be "name", though that can be empty if there is a single port only - is that a problem? @ymqytw ?

@mengqiy
Copy link
Member

mengqiy commented Dec 28, 2016

@thockin The validation code doesn't allow me to create two ports with names unspecified.

The Service "my-service" is invalid: 
* spec.ports[0].name: Required value
* spec.ports[1].name: Required value

What version did the user use?

@thockin
Copy link
Member Author

thockin commented Dec 28, 2016 via email

@mengqiy
Copy link
Member

mengqiy commented Dec 28, 2016

Still, the port name seems like the more correct merge key

Agree.
If we want to use name as mergeKey, we should add some validation code to make sure the names are unique. And we need to take care of an empty name if there is a single port only.

@thockin This is a breaking change. So we should make this change in 1.6, right?

cc: @pwittrock

@thockin
Copy link
Member Author

thockin commented Dec 28, 2016 via email

@mengqiy
Copy link
Member

mengqiy commented Dec 28, 2016

Can "" be a valid value for the case of single port?

Don't know yet. I need to look into it.

I guess changing merge key is technically a breaking change, but is there
any real impact?

Yes, like #36024(kubectl is broken, even with minor version skew)

@thockin
Copy link
Member Author

thockin commented Dec 28, 2016 via email

@mengqiy
Copy link
Member

mengqiy commented Dec 28, 2016

Can "" be a valid value for the case of single port?

The mergeKey must present in the go struct to calculate patch. But object round tripping will drop empty field, which is name in this case. So simply change the mergeKeywill break the no-name single-port case.

for: "../svc.yaml": map: map[protocol:UDP port:30420 targetPort:0 nodePort:30432] does not contain declared merge key: name

We should to change the mergeKey, otherwise kubectl apply will do something wrong.
e.g. Creating a service using kubectl apply

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort
  ports:
    - protocol: UDP
      port: 30420
      name: updport
      nodePort: 30420
    - protocol: TCP
      port: 30420
      name: tcpport
      nodePort: 30420
  selector:
    app: MyApp

Make change to tcpport and apply the change

    - protocol: TCP
      port: 30420
      name: tcpport
      nodePort: 30000 # Change this nodePort

kubectl get the service back

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"kind":"Service","apiVersion":"v1","metadata":{"name":"my-service","creationTimestamp":null},"spec":{"ports":[{"name":"updport","protocol":"UDP","port":30420,"targetPort":0,"nodePort":30420},{"name":"tcpport","protocol":"TCP","port":30420,"targetPort":0,"nodePort":30000}],"selector":{"app":"MyApp"},"type":"NodePort"},"status":{"loadBalancer":{}}}
  name: my-service
...
spec:
  clusterIP: 10.0.0.249
  ports:
  - name: updport
    nodePort: 30000 # Wrong change
    port: 30420
    protocol: UDP
    targetPort: 30420
  - name: tcpport
    nodePort: 30420
    port: 30420
    protocol: TCP
    targetPort: 30420
  selector:
    app: MyApp
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

The change goes to the wrong place.

@thockin
Copy link
Member Author

thockin commented Dec 28, 2016 via email

@mengqiy
Copy link
Member

mengqiy commented Dec 29, 2016

Can we have an empty mergekey automatically become a random string?

Yes, it's possible but it will make the logic complicated. Because apply need original config(in annotation), current config(on API server) and modified config(in user's file) to do 3-way diff.

I'm not sure if @pwittrock @bgrant0607 will like this. Need their comments before going further.

@bgrant0607
Copy link
Member

"Random" string -- no.

We have other cases where a single field isn't sufficient as a merge key, such as lists of object references. I think treating all fields specified by the user's config as the key works in all cases I've thought of. I'd probably make that a new patch strategy. It would probably even be backward compatible.

@pwittrock
Copy link
Member

@bgrant0607 Are you thinking this would support only the primitive fields as part of the unique/merge key? Or would this strategy only be valid for types that contain only primitive fields?

Based on your description, the merge key for this strategy would be dynamic and defined by the client.

Have we considered supporting merge keys that are multiple values, but otherwise treated the same as single-value merge keys (in this case it could be <port,protocol> or <port,destinationport,protocol>)?

@bgrant0607
Copy link
Member

@pwittrock No, I was thinking we'd use all fields specified by the user as the key, including nested maps. Multiple keys would also work and may be less error-prone.

@thockin
Copy link
Member Author

thockin commented Jan 12, 2017 via email

@pwittrock
Copy link
Member

pwittrock commented Jan 12, 2017

@thockin I was thinking something similar yesterday. Something like:

If len(patch list) == 1 && merge key is missing from patch list entry && len(dest list) == 1 && merge key is missing from dest list entry
-> Then merge into only entry in destination list

My biggest hesitation here was that it adds some complexity to the already poorly understood merge semantics. There would probably be some weird edge cases like what happens when we add the merge key to the existing entry. e.g.

If len(patch list) == 1 && merge key exists in the list entry && len(dest list) == 1 && merge key is missing from dest list entry
-> Then merge into only entry in destination list, setting the merge key

@pwittrock
Copy link
Member

@ymqytw Can you make sure this gets resolved in 1.7?

@pwittrock pwittrock added this to the v1.7 milestone Mar 7, 2017
@mengqiy
Copy link
Member

mengqiy commented Mar 7, 2017

@pwittrock If using the approach in #39188 (comment), then yes.
But if we want support combined merge key, which contains multiple keys, then I think we need a proposal first.

@pwittrock
Copy link
Member

Lets sync up with @thockin to determine the minimal requirements for correctness. I don't want to introduce a partial solution since it will create a maintenance burden. Once the merge-key audit is complete, lets come up with a proposal.

I think it is ok if we write a design proposal. I will make sure it gets appropriately reviewed within ~weeks.

@bgrant0607 bgrant0607 added the sig/cli Categorizes an issue or PR as relevant to SIG CLI. label Mar 9, 2017
@thockin
Copy link
Member Author

thockin commented Mar 17, 2017

I've run into a lot of issues with ports being PATCHed wrongly through 1.6. Do we believe these issues are fixed now? I can't seem to repro them, but I wasn't dilligent in cataloging them all...

@mengqiy
Copy link
Member

mengqiy commented Mar 17, 2017

I guess the pain you have when PATCHing the ports is caused by the bug of using the wrong json pkg in the api server (#42488). It has been fixed by #42489.

issues with ports:

@jamiehannaford
Copy link
Contributor

Has this been fixed? I'm seeing both ports on describe:

› kubectl describe svc nginx
Name:			nginx
Namespace:		default
Labels:			<none>
Annotations:		<none>
Selector:		app=nginx
Type:			NodePort
IP:			10.0.0.133
Port:			udp	30420/UDP
NodePort:		udp	30420/UDP
Endpoints:		172.17.0.4:30420
Port:			tcp	30420/TCP
NodePort:		tcp	30420/TCP
Endpoints:		172.17.0.4:30420
Session Affinity:	None
Events:			<none>

@mengqiy
Copy link
Member

mengqiy commented Apr 20, 2017

@jamiehannaford Not yet. describe is not affected by this issue.
Proposal in kubernetes/community#476. Review welcome.

@pwittrock pwittrock modified the milestones: v1.8, v1.7 Jun 2, 2017
@mateuszdrab
Copy link

mateuszdrab commented Oct 29, 2022

This bug still exists and made it impossible for me to use Consul's agents - ports are exposed (with different name) on both TCP and UDP only one was working. Again, applying a config or a applying a patch didn't fix it - I had to destroy the daemonset and recreate it, only then was the hostPort for both UDP and TCP preserved (at least in 1.23.1)

God! I scratched my head so many times and never thought this could actually be a real bug... one assumes basics like this would never be broken for so long 😅
Explains a lot why my syslog forwarder is not working on TCP and UDP - hope this gets resolved soon. The issue's been open for years

@fgleixner
Copy link

Amazing. How can one vote up for this bug?

@allyn-bottorff
Copy link

This person's blog describes the issue and a workaround: https://ben-lab.github.io/kubernetes-UDP-TCP-bug-same-port/

Seems like this works fine as long as the service is created initially with both ports, it's only adding a matching port after the fact which triggers the bug. This was also my experience setting up a TCP and UDP DNS server.

anitgandhi added a commit to digitalocean/digitalocean-cloud-controller-manager that referenced this issue Dec 29, 2022
This was originally added in #409 due to an upstream bug (kubernetes/kubernetes#39188). It turns out, however, that we and many others misinterpreted the scope of that bug - it really only applies for updating an existing service with a new port definition. It does not apply for creating a new service.

Our customers are affected by this, and this validation is overly restrictive compared to errors the Load Balancers API actually returns during its validation. As such, the validation we do here can be relaxed.
anitgandhi added a commit to digitalocean/digitalocean-cloud-controller-manager that referenced this issue Jan 3, 2023
This was originally added in #409 due to an upstream bug (kubernetes/kubernetes#39188). It turns out, however, that we and many others misinterpreted the scope of that bug - it really only applies for updating an existing service with a new port definition. It does not apply for creating a new service.

Our customers are affected by this, and this validation is overly restrictive compared to errors the Load Balancers API actually returns during its validation. As such, the validation we do here can be relaxed.
@bluepuma77
Copy link

Amazing. How can one vote up for this bug?

Yeah, amazing that this bug is with us since 2016. I believe to express your support you can upvote the initial post.

@thockin
Copy link
Member Author

thockin commented Jan 16, 2023

We should re-title this (or close and open new one) describing the problem.

kubectl describe works now and has for a while.

$ k describe svc kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
...
Port:              dns-udp  53/UDP
TargetPort:        53/UDP
Endpoints:         10.64.1.46:53,10.64.3.55:53
Port:              dns  53/TCP
TargetPort:        53/TCP
Endpoints:         10.64.1.46:53,10.64.3.55:53
Port:              metrics  9153/TCP
TargetPort:        9153/TCP

Client-side apply (and edit) is broken because the merge-key is wrong (just .port). But kubectl replace works, and kubectl apply --server-side works (key is port+protocol). Nobody has been able to find a way that makes client-side apply able to be fixed "safely".

So can someone tell me what the current problem is, other than client-side apply being broken?

@mateuszdrab
Copy link

Looks like server-side apply is the future, so I think we may have to stop expecting it to get fixed in client-side but I too wonder why such thing is left unresolved for so many years...
I guess we belong to the small niche of people who actually expose dual protocol services on the same port?

@erwbgy
Copy link

erwbgy commented Jan 16, 2023

@thockin That client-side kubectl apply silently fails to do the expected updates is the problem.

The Server Side Apply Is Great And You Should Be Using It blog post says: "Soon we (SIG API Machinery) hope to make this the default and remove the “client side” apply completely!". I expect that this issue can finally be closed once that happens.

In the meantime documenting issues like this in the Comparison with Client Side Apply section of the documentation might help.

@thockin
Copy link
Member Author

thockin commented Jan 16, 2023

@mateuszdrab - lots of people use dual-protocol (~everyone - DNS). It works fine EXCEPT FOR patch/apply/edit. Fixing it has not been possible because that could be an API break for things that actually work today. I've spent a bit of time looking into it more this weekend, and I can't actually tell if it is POSSIBLE to specify a compound key. I think it is not.

@erwbgy - that doc is about the feature of apply, not about specific bugs. Still, I don't know if this hole is documented anywhere

@mateuszdrab
Copy link

mateuszdrab commented Jan 16, 2023

@mateuszdrab - lots of people use dual-protocol (~everyone - DNS). It works fine EXCEPT FOR patch/apply/edit. Fixing it has not been possible because that could be an API break for things that actually work today. I've spent a bit of time looking into it more this weekend, and I can't actually tell if it is POSSIBLE to specify a compound key. I think it is not.

@erwbgy - that doc is about the feature of apply, not about specific bugs. Still, I don't know if this hole is documented anywhere

DNS is exactly what I use and struggled with this issue so many times... Until I found this issue 😂🤦🏼‍♂️

@thockin
Copy link
Member Author

thockin commented Jan 16, 2023 via email

@mateuszdrab
Copy link

I'm confused about "so many times" - kubectl create works and kubectl apply should work when there's not an existing resource (I think? Now
that I type it out, I am not so sure). Help me understand?

On Mon, Jan 16, 2023 at 11:00 AM Mateusz Drab @.***>
wrote:

@mateuszdrab https://github.com/mateuszdrab - lots of people use
dual-protocol (~everyone - DNS). It works fine EXCEPT FOR patch/apply/edit.
Fixing it has not been possible because that could be an API break for
things that actually work today. I've spent a bit of time looking into it
more this weekend, and I can't actually tell if it is POSSIBLE to specify a
compound key. I think it is not.

@erwbgy https://github.com/erwbgy - that doc is about the feature of
apply, not about specific bugs. Still, I don't know if this hole is
documented anywhere

DNS is exactly what I use and struggled with this issue so many times...
Until I found this issue


Reply to this email directly, view it on GitHub
#39188 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKWAVHRNJ5ZPNGCOFKWRDTWSWLGPANCNFSM4C2SJQKQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

I use flux which was applying the manifest server side, but when I was making changes to my deployment I'd apply things manually via kubectl and didn't notice things were broken for some time.
I never investigated it deeply then, so it was just one of those mysterious things until one day when I realized it's only one of the two ports that work... that's how I got here...

@thockin
Copy link
Member Author

thockin commented Jan 16, 2023

More information in #103544 and #104486

@thockin
Copy link
Member Author

thockin commented Jan 16, 2023

I'm accumulating some details in #105610

@nerzhul
Copy link

nerzhul commented Sep 5, 2023

i just fall in the same issue. I was trying to understand why my DNS queries goes to port 53 instead of 5353 and only in TCP. The service was doing only TCP 53->53. Editing TCP port on the service edited the UDP port (if not the same than UDP port), very weird.
kubectl replace worked fine, but kubectl edit not. Be warned.

VannTen added a commit to VannTen/kubespray that referenced this issue May 3, 2024
Kubectl client-side apply breaks coredns on upgrade because the old and
the new version are not resolved correctly (see
kubernetes/kubernetes#39188 (comment))
TL;DR: the merge key for the port array is only the port number, not
port+protocol.
Server-side apply solves this issue, but our custom kube module is not
server-side apply ready.

While we could improve our custom kube module, it will be a lesser
maintenance load going forward to progressively drop it and switch to
kubernetes.core.k8s, which has more features and is maintained by
upstream.

Do that now for coredns manifests.
Add the python k8s client on the control plane (no need for it
elsewhere), and the python-venv on distributions which need it to create
a virtualenv.
VannTen added a commit to VannTen/kubespray that referenced this issue May 13, 2024
Kubectl client-side apply breaks coredns on upgrade because the old and
the new version are not resolved correctly (see
kubernetes/kubernetes#39188 (comment))
TL;DR: the merge key for the port array is only the port number, not
port+protocol.
Server-side apply solves this issue, but our custom kube module is not
server-side apply ready.

While we could improve our custom kube module, it will be a lesser
maintenance load going forward to progressively drop it and switch to
kubernetes.core.k8s, which has more features and is maintained by
upstream.

Do that now for coredns manifests.
Add the python k8s client on the control plane (no need for it
elsewhere), and the python-venv on distributions which need it to create
a virtualenv.
sreber84 added a commit to sreber84/argo-cd that referenced this issue May 17, 2024
rh-pre-commit.version: 2.2.0
rh-pre-commit.check-secrets: ENABLED
@itseramin
Copy link

I also had to delete and redeploy our entire NGINX ingress controller deployment for it work, not only the desired application's service and deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api Indicates an issue on api area. kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Development

No branches or pull requests