Services with same port, different protocol display wrongly in kubectl and have wrong merge key #39188

thockin · 2016-12-23T04:49:49Z

User reported:

I am running a service with both TCP and UDP:

spec:
  type: NodePort
  ports:
    - protocol: UDP
      port: 30420
      nodePort: 30420
    - protocol: TCP
      port: 30420
      nodePort: 30420

but kubectl describe service shows only UDP.

Type:			NodePort
IP:			10.0.13.152
Port:			<unset>	30420/UDP
NodePort:		<unset>	30420/UDP
Endpoints:		10.244.4.49:30420

When I change the order then it shows only TCP.

This looks like using the wrong mergeKey, and indeed the code backs that up:

2508 type ServiceSpec struct {
2509 // The list of ports that are exposed by this service.
2510 // More info: http://kubernetes.io/docs/user-guide/services#virtual-ips-and-service-proxies
2511 Ports []ServicePort json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"port" protobuf:"bytes,1,rep,name=ports"

The key should probably be "name", though that can be empty if there is a single port only - is that a problem? @ymqytw ?

mengqiy · 2016-12-28T00:11:32Z

@thockin The validation code doesn't allow me to create two ports with names unspecified.

The Service "my-service" is invalid: 
* spec.ports[0].name: Required value
* spec.ports[1].name: Required value

What version did the user use?

thockin · 2016-12-28T00:18:12Z

It looks like the user deleted the comment where they claimed this, so probably realized they made a mistake (sorry for not seeing that). Still, the port name seems like the more correct merge key

…

On Tue, Dec 27, 2016 at 4:11 PM, Mengqi Yu ***@***.***> wrote: @thockin <https://github.com/thockin> The validation code <https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go#L2481> doesn't allow me to create two ports with names unspecified. The Service "my-service" is invalid: * spec.ports[0].name: Required value * spec.ports[1].name: Required value What version did the user use? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVH8_TwXVE7sudwh4kZl230-wkIPgks5rMalGgaJpZM4LUkwV> .

mengqiy · 2016-12-28T00:40:12Z

Still, the port name seems like the more correct merge key

Agree.
If we want to use name as mergeKey, we should add some validation code to make sure the names are unique. And we need to take care of an empty name if there is a single port only.

@thockin This is a breaking change. So we should make this change in 1.6, right?

cc: @pwittrock

thockin · 2016-12-28T00:45:15Z

Names are unique. We can't require that name not be empty for the single-port case, and we already demand it in the multi-port case. Can "" be a valid value for the case of single port? I guess changing merge key is technically a breaking change, but is there any real impact?

…

On Tue, Dec 27, 2016 at 4:40 PM, Mengqi Yu ***@***.***> wrote: Still, the port name seems like the more correct merge key Agree. If we want to use name as mergeKey, we should add some validation code to make sure the names are unique. And we need to take care of an empty name if there is a single port only. @thockin <https://github.com/thockin> This is a breaking change. So we should make this change in 1.6, right? cc: @pwittrock <https://github.com/pwittrock> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVFEiLbxVHCgP4QtVUV4t9PML2OC2ks5rMa_7gaJpZM4LUkwV> .

mengqiy · 2016-12-28T01:00:18Z

Can "" be a valid value for the case of single port?

Don't know yet. I need to look into it.

I guess changing merge key is technically a breaking change, but is there
any real impact?

Yes, like #36024(kubectl is broken, even with minor version skew)

thockin · 2016-12-28T01:03:27Z

yuck

…

On Tue, Dec 27, 2016 at 5:00 PM, Mengqi Yu ***@***.***> wrote: Can "" be a valid value for the case of single port? Don't know yet. I need to look into it. I guess changing merge key is technically a breaking change, but is there any real impact? Yes, like #36024 <#36024>( kubectl is broken, even with minor version skew) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVI2Vj195GZZg7oyCZwmm4SaqQaOZks5rMbSzgaJpZM4LUkwV> .

mengqiy · 2016-12-28T22:20:18Z

Can "" be a valid value for the case of single port?

The mergeKey must present in the go struct to calculate patch. But object round tripping will drop empty field, which is name in this case. So simply change the mergeKeywill break the no-name single-port case.

for: "../svc.yaml": map: map[protocol:UDP port:30420 targetPort:0 nodePort:30432] does not contain declared merge key: name

We should to change the mergeKey, otherwise kubectl apply will do something wrong.
e.g. Creating a service using kubectl apply

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort
  ports:
    - protocol: UDP
      port: 30420
      name: updport
      nodePort: 30420
    - protocol: TCP
      port: 30420
      name: tcpport
      nodePort: 30420
  selector:
    app: MyApp

Make change to tcpport and apply the change

    - protocol: TCP
      port: 30420
      name: tcpport
      nodePort: 30000 # Change this nodePort

kubectl get the service back

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"kind":"Service","apiVersion":"v1","metadata":{"name":"my-service","creationTimestamp":null},"spec":{"ports":[{"name":"updport","protocol":"UDP","port":30420,"targetPort":0,"nodePort":30420},{"name":"tcpport","protocol":"TCP","port":30420,"targetPort":0,"nodePort":30000}],"selector":{"app":"MyApp"},"type":"NodePort"},"status":{"loadBalancer":{}}}
  name: my-service
...
spec:
  clusterIP: 10.0.0.249
  ports:
  - name: updport
    nodePort: 30000 # Wrong change
    port: 30420
    protocol: UDP
    targetPort: 30420
  - name: tcpport
    nodePort: 30420
    port: 30420
    protocol: TCP
    targetPort: 30420
  selector:
    app: MyApp
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

The change goes to the wrong place.

thockin · 2016-12-28T23:11:55Z

I agree we should change it, but how can we do that if name is optional? Can we have an empty mergekey automatically become a random string?

…

On Wed, Dec 28, 2016 at 2:20 PM, Mengqi Yu ***@***.***> wrote: Can "" be a valid value for the case of single port? The mergeKey must present in the go struct to calculate patch. But object round tripping will drop empty field, which is name in this case. So simply change the mergeKeywill break the no-name single-port case. We should to change the mergeKey, otherwise kubectl apply will do something wrong. e.g. Creating a service using kubectl apply apiVersion: v1kind: Servicemetadata: name: my-servicespec: type: NodePort ports: - protocol: UDP port: 30420 name: updport nodePort: 30420 - protocol: TCP port: 30420 name: tcpport nodePort: 30420 selector: app: MyApp Make change to tcpport and apply the change - protocol: TCP port: 30420 name: tcpport nodePort: 30000 # Change this nodePort kubectl get the service back apiVersion: v1kind: Servicemetadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"kind":"Service","apiVersion":"v1","metadata":{"name":"my-service","creationTimestamp":null},"spec":{"ports":[{"name":"updport","protocol":"UDP","port":30420,"targetPort":0,"nodePort":30420},{"name":"tcpport","protocol":"TCP","port":30420,"targetPort":0,"nodePort":30000}],"selector":{"app":"MyApp"},"type":"NodePort"},"status":{"loadBalancer":{}}} name: my-service...spec: clusterIP: 10.0.0.249 ports: - name: updport nodePort: 30000 # Wrong change port: 30420 protocol: UDP targetPort: 30420 - name: tcpport nodePort: 30420 port: 30420 protocol: TCP targetPort: 30420 selector: app: MyApp sessionAffinity: None type: NodePortstatus: loadBalancer: {} The change goes to the wrong place. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVB00_aCZ2pF7mjyy3y2f810VjtMaks5rMuDGgaJpZM4LUkwV> .

mengqiy · 2016-12-29T00:09:33Z

Can we have an empty mergekey automatically become a random string?

Yes, it's possible but it will make the logic complicated. Because apply need original config(in annotation), current config(on API server) and modified config(in user's file) to do 3-way diff.

I'm not sure if @pwittrock @bgrant0607 will like this. Need their comments before going further.

bgrant0607 · 2017-01-10T01:36:55Z

"Random" string -- no.

We have other cases where a single field isn't sufficient as a merge key, such as lists of object references. I think treating all fields specified by the user's config as the key works in all cases I've thought of. I'd probably make that a new patch strategy. It would probably even be backward compatible.

pwittrock · 2017-01-12T00:07:41Z

@bgrant0607 Are you thinking this would support only the primitive fields as part of the unique/merge key? Or would this strategy only be valid for types that contain only primitive fields?

Based on your description, the merge key for this strategy would be dynamic and defined by the client.

Have we considered supporting merge keys that are multiple values, but otherwise treated the same as single-value merge keys (in this case it could be <port,protocol> or <port,destinationport,protocol>)?

bgrant0607 · 2017-01-12T03:10:13Z

@pwittrock No, I was thinking we'd use all fields specified by the user as the key, including nested maps. Multiple keys would also work and may be less error-prone.

thockin · 2017-01-12T05:03:40Z

For this particular case "" is a valid name in only in the case of a single entry, and that is enforced by validation, so if "" was allowed it would just work?

…

On Wed, Jan 11, 2017 at 7:10 PM, Brian Grant ***@***.***> wrote: @pwittrock <https://github.com/pwittrock> No, I was thinking we'd use all fields specified by the user as the key, including nested maps. Multiple keys would also work and may be less error-prone. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVDqLA1mRB-bEcODC1TQDuFNPXi_7ks5rRZmmgaJpZM4LUkwV> .

pwittrock · 2017-01-12T17:07:41Z

@thockin I was thinking something similar yesterday. Something like:

If len(patch list) == 1 && merge key is missing from patch list entry && len(dest list) == 1 && merge key is missing from dest list entry
-> Then merge into only entry in destination list

My biggest hesitation here was that it adds some complexity to the already poorly understood merge semantics. There would probably be some weird edge cases like what happens when we add the merge key to the existing entry. e.g.

If len(patch list) == 1 && merge key exists in the list entry && len(dest list) == 1 && merge key is missing from dest list entry
-> Then merge into only entry in destination list, setting the merge key

pwittrock · 2017-03-07T16:09:21Z

@ymqytw Can you make sure this gets resolved in 1.7?

mengqiy · 2017-03-07T16:44:24Z

@pwittrock If using the approach in #39188 (comment), then yes.
But if we want support combined merge key, which contains multiple keys, then I think we need a proposal first.

pwittrock · 2017-03-07T17:09:42Z

Lets sync up with @thockin to determine the minimal requirements for correctness. I don't want to introduce a partial solution since it will create a maintenance burden. Once the merge-key audit is complete, lets come up with a proposal.

I think it is ok if we write a design proposal. I will make sure it gets appropriately reviewed within ~weeks.

thockin · 2017-03-17T20:22:34Z

I've run into a lot of issues with ports being PATCHed wrongly through 1.6. Do we believe these issues are fixed now? I can't seem to repro them, but I wasn't dilligent in cataloging them all...

mengqiy · 2017-03-17T23:43:46Z

I guess the pain you have when PATCHing the ports is caused by the bug of using the wrong json pkg in the api server (#42488). It has been fixed by #42489.

issues with ports:

#42488 number conversion issue caused by using the wrong json pkg.
(this issue) mergeKey for ports (port #) is not unique. The ports will still be PATCHed wrongly if the user does something like #39188 (comment)

jamiehannaford · 2017-04-20T10:39:05Z

Has this been fixed? I'm seeing both ports on describe:

› kubectl describe svc nginx
Name:			nginx
Namespace:		default
Labels:			<none>
Annotations:		<none>
Selector:		app=nginx
Type:			NodePort
IP:			10.0.0.133
Port:			udp	30420/UDP
NodePort:		udp	30420/UDP
Endpoints:		172.17.0.4:30420
Port:			tcp	30420/TCP
NodePort:		tcp	30420/TCP
Endpoints:		172.17.0.4:30420
Session Affinity:	None
Events:			<none>

mengqiy · 2017-04-20T17:44:05Z

@jamiehannaford Not yet. describe is not affected by this issue.
Proposal in kubernetes/community#476. Review welcome.

bgrant0607 · 2018-01-23T05:01:05Z

/remove-lifecycle stale

yue9944882 · 2018-02-08T01:38:22Z

Related #59482

bgrant0607 · 2018-02-08T02:08:01Z

Yes #59482 has the same root cause.

praseodym · 2018-04-28T22:27:28Z

I think containerPorts in Deployments have the same problem. I can't seem to specify the same port for two different protocols.

fabianvf · 2018-06-20T15:09:37Z

This issue doesn't seem related to the CLI, the issue is in the API server. Should we change the sig to api-machinery or something like that?

fejta-bot · 2018-09-18T15:27:46Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

george-angel · 2018-09-18T15:50:16Z

/remove-lifecycle stale

fejta-bot · 2018-12-17T16:23:04Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

george-angel · 2018-12-17T16:29:18Z

/remove-lifecycle stale


        Changes made before discovering k8s bug kubernetes/kubernetes#39188.

skwokie · 2019-08-13T00:17:26Z

What happened:
I think I've reproduced this; please feel free to let me know if my expectation is wrong.
I've made a deployment with the following ports declaration:

ports:
  - name: https
    containerPort: 443
  - name: cport10000tcp
    containerPort: 10000
    protocol: TCP
  - name: cport10000udp
    containerPort: 10000
    protocol: UDP
  - name: cport10001udp
    containerPort: 10001
    protocol: UDP

When I describe the pod, it has skipped mentioning the port (10000) being opened for UDP.

Pod Template:
  Labels:  app=svr
           env=dev
  Containers:
   svr:
    Image:       dev-svr:0.92
    Ports:       443/TCP, 10000/TCP, 10001/UDP
    Host Ports:  0/TCP, 0/TCP, 0/UDP

Environment:

Kubernetes version (use kubectl version):
$ kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration:
Mac
OS (e.g: cat /etc/os-release):
macOS Version 10.14.6
Kernel (e.g. uname -a):
$ uname -a Darwin skwok-mbp.local 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64
Install tools:
Docker Desktop version 19.03.1
Network plugin and version (if this is a network-related bug):
Others:

davidalpert · 2019-08-26T22:33:15Z

I can repeat that behavior, plus PATCH fails on a service or deployment described with the same port listed twice (even when each port has a unique name or protocol attribute) as it appears that the matchKey only considers port

berlincount · 2019-09-26T12:04:17Z

Running into the same issue as well.

thockin added area/api area/kubectl labels Dec 23, 2016

thockin referenced this issue Dec 23, 2016

Allow services which use same port, different protocol to use the same nodePort for both #30253

Merged

mengqiy referenced this issue Mar 2, 2017

Unable to use `kubectl edit` to change service type from NodePort to ClusterIP #42282

Closed

pwittrock assigned mengqiy Mar 7, 2017

pwittrock added this to the v1.7 milestone Mar 7, 2017

bgrant0607 added the sig/cli label Mar 9, 2017

pwittrock modified the milestones: v1.8, v1.7 Jun 2, 2017

k8s-ci-robot added the lifecycle/stale label Jan 4, 2018

k8s-ci-robot removed the lifecycle/stale label Jan 23, 2018

yue9944882 referenced this issue Feb 8, 2018

kubectl apply will not add protocols to services that already exist #59482

Closed

mengqiy referenced this issue Apr 24, 2018

Strategic merge patch sort causes confusion #62830

Closed

ctrox referenced this issue May 25, 2018

k8s_raw StatefulSet confuses same containerPort for TCP and UDP #40654

Closed

fabianvf referenced this issue May 25, 2018

Patching a StatefulSet's containerPorts causes values to be clobbered #64321

Closed

liggitt referenced this issue Jul 5, 2018

Possible to bypass Service validation via "apply", loses declarative spec and locks down the object forever #491

Closed

k8s-ci-robot added the lifecycle/stale label Sep 18, 2018

k8s-ci-robot removed the lifecycle/stale label Sep 18, 2018

mengqiy referenced this issue Nov 27, 2018

Multiple protocols are clobbered when rendering ports #558

Closed

k8s-ci-robot added the lifecycle/stale label Dec 17, 2018

k8s-ci-robot removed the lifecycle/stale label Dec 17, 2018

bgrant0607 added the lifecycle/frozen label Jan 18, 2019

bgrant0607 unassigned mengqiy Jan 18, 2019

This was referenced Mar 25, 2019

Unable to make TCP DNS queries on one cluster, not others, until DestinationRule recreated istio/istio#11658

Closed

bug fix: tcp/udp port may conflict istio/istio#12742

Closed

pepedocs added a commit to pepedocs/pantheon-k8s that referenced this issue May 28, 2019

Changes made before discovering k8s bug kubernetes/kubernetes#39188.

e77657e

beorn7 referenced this issue Oct 27, 2019

Cannot define a TCP port and a UDP port when both use the same port number #99

Closed

kubernetes/kubernetes

Join GitHub today

Services with same port, different protocol display wrongly in kubectl and have wrong merge key #39188

Comments

thockin commented Dec 23, 2016

This comment has been minimized.

mengqiy commented Dec 28, 2016

This comment has been minimized.

thockin commented Dec 28, 2016

This comment has been minimized.

mengqiy commented Dec 28, 2016

This comment has been minimized.

thockin commented Dec 28, 2016

This comment has been minimized.

mengqiy commented Dec 28, 2016

This comment has been minimized.

thockin commented Dec 28, 2016

This comment has been minimized.

mengqiy commented Dec 28, 2016 • edited

This comment has been minimized.

thockin commented Dec 28, 2016

This comment has been minimized.

mengqiy commented Dec 29, 2016

This comment has been minimized.

bgrant0607 commented Jan 10, 2017

This comment has been minimized.

pwittrock commented Jan 12, 2017

This comment has been minimized.

bgrant0607 commented Jan 12, 2017

This comment has been minimized.

thockin commented Jan 12, 2017

This comment has been minimized.

pwittrock commented Jan 12, 2017 • edited

This comment has been minimized.

pwittrock commented Mar 7, 2017

This comment has been minimized.

mengqiy commented Mar 7, 2017

This comment has been minimized.

pwittrock commented Mar 7, 2017

This comment has been minimized.

thockin commented Mar 17, 2017

This comment has been minimized.

mengqiy commented Mar 17, 2017

This comment has been minimized.

jamiehannaford commented Apr 20, 2017

This comment has been minimized.

mengqiy commented Apr 20, 2017

This comment has been minimized.

bgrant0607 commented Jan 23, 2018

This comment has been minimized.

yue9944882 commented Feb 8, 2018

This comment has been minimized.

bgrant0607 commented Feb 8, 2018

This comment has been minimized.

praseodym commented Apr 28, 2018

This comment has been minimized.

fabianvf commented Jun 20, 2018

This comment has been minimized.

fejta-bot commented Sep 18, 2018

This comment has been minimized.

george-angel commented Sep 18, 2018

This comment has been minimized.

fejta-bot commented Dec 17, 2018

This comment has been minimized.

george-angel commented Dec 17, 2018

This comment has been minimized.

skwokie commented Aug 13, 2019

This comment has been minimized.

davidalpert commented Aug 26, 2019

This comment has been minimized.

berlincount commented Sep 26, 2019

mengqiy commented Dec 28, 2016 •

edited

pwittrock commented Jan 12, 2017 •

edited