New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Services with same port, different protocol display wrongly in kubectl and have wrong merge key #39188
Comments
@thockin The validation code doesn't allow me to create two ports with names unspecified.
What version did the user use? |
It looks like the user deleted the comment where they claimed this, so
probably realized they made a mistake (sorry for not seeing that). Still,
the port name seems like the more correct merge key
…On Tue, Dec 27, 2016 at 4:11 PM, Mengqi Yu ***@***.***> wrote:
@thockin <https://github.com/thockin> The validation code
<https://github.com/kubernetes/kubernetes/blob/master/pkg/api/validation/validation.go#L2481>
doesn't allow me to create two ports with names unspecified.
The Service "my-service" is invalid:
* spec.ports[0].name: Required value
* spec.ports[1].name: Required value
What version did the user use?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#39188 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVH8_TwXVE7sudwh4kZl230-wkIPgks5rMalGgaJpZM4LUkwV>
.
|
Agree. @thockin This is a breaking change. So we should make this change in 1.6, right? cc: @pwittrock |
Names are unique. We can't require that name not be empty for the
single-port case, and we already demand it in the multi-port case. Can ""
be a valid value for the case of single port?
I guess changing merge key is technically a breaking change, but is there
any real impact?
…On Tue, Dec 27, 2016 at 4:40 PM, Mengqi Yu ***@***.***> wrote:
Still, the port name seems like the more correct merge key
Agree.
If we want to use name as mergeKey, we should add some validation code to
make sure the names are unique. And we need to take care of an empty name
if there is a single port only.
@thockin <https://github.com/thockin> This is a breaking change. So we
should make this change in 1.6, right?
cc: @pwittrock <https://github.com/pwittrock>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#39188 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVFEiLbxVHCgP4QtVUV4t9PML2OC2ks5rMa_7gaJpZM4LUkwV>
.
|
Don't know yet. I need to look into it.
Yes, like #36024( |
yuck
…On Tue, Dec 27, 2016 at 5:00 PM, Mengqi Yu ***@***.***> wrote:
Can "" be a valid value for the case of single port?
Don't know yet. I need to look into it.
I guess changing merge key is technically a breaking change, but is there
any real impact?
Yes, like #36024 <#36024>(
kubectl is broken, even with minor version skew)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#39188 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVI2Vj195GZZg7oyCZwmm4SaqQaOZks5rMbSzgaJpZM4LUkwV>
.
|
The mergeKey must present in the go struct to calculate patch. But object round tripping will drop empty field, which is
We should to change the apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
type: NodePort
ports:
- protocol: UDP
port: 30420
name: updport
nodePort: 30420
- protocol: TCP
port: 30420
name: tcpport
nodePort: 30420
selector:
app: MyApp Make change to - protocol: TCP
port: 30420
name: tcpport
nodePort: 30000 # Change this nodePort
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"kind":"Service","apiVersion":"v1","metadata":{"name":"my-service","creationTimestamp":null},"spec":{"ports":[{"name":"updport","protocol":"UDP","port":30420,"targetPort":0,"nodePort":30420},{"name":"tcpport","protocol":"TCP","port":30420,"targetPort":0,"nodePort":30000}],"selector":{"app":"MyApp"},"type":"NodePort"},"status":{"loadBalancer":{}}}
name: my-service
...
spec:
clusterIP: 10.0.0.249
ports:
- name: updport
nodePort: 30000 # Wrong change
port: 30420
protocol: UDP
targetPort: 30420
- name: tcpport
nodePort: 30420
port: 30420
protocol: TCP
targetPort: 30420
selector:
app: MyApp
sessionAffinity: None
type: NodePort
status:
loadBalancer: {} The change goes to the wrong place. |
I agree we should change it, but how can we do that if name is optional?
Can we have an empty mergekey automatically become a random string?
…On Wed, Dec 28, 2016 at 2:20 PM, Mengqi Yu ***@***.***> wrote:
Can "" be a valid value for the case of single port?
The mergeKey must present in the go struct to calculate patch. But object
round tripping will drop empty field, which is name in this case. So
simply change the mergeKeywill break the no-name single-port case.
We should to change the mergeKey, otherwise kubectl apply will do
something wrong.
e.g. Creating a service using kubectl apply
apiVersion: v1kind: Servicemetadata:
name: my-servicespec:
type: NodePort
ports:
- protocol: UDP
port: 30420
name: updport
nodePort: 30420
- protocol: TCP
port: 30420
name: tcpport
nodePort: 30420
selector:
app: MyApp
Make change to tcpport and apply the change
- protocol: TCP
port: 30420
name: tcpport
nodePort: 30000 # Change this nodePort
kubectl get the service back
apiVersion: v1kind: Servicemetadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: | {"kind":"Service","apiVersion":"v1","metadata":{"name":"my-service","creationTimestamp":null},"spec":{"ports":[{"name":"updport","protocol":"UDP","port":30420,"targetPort":0,"nodePort":30420},{"name":"tcpport","protocol":"TCP","port":30420,"targetPort":0,"nodePort":30000}],"selector":{"app":"MyApp"},"type":"NodePort"},"status":{"loadBalancer":{}}} name: my-service...spec:
clusterIP: 10.0.0.249
ports:
- name: updport
nodePort: 30000 # Wrong change
port: 30420
protocol: UDP
targetPort: 30420
- name: tcpport
nodePort: 30420
port: 30420
protocol: TCP
targetPort: 30420
selector:
app: MyApp
sessionAffinity: None
type: NodePortstatus:
loadBalancer: {}
The change goes to the wrong place.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#39188 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVB00_aCZ2pF7mjyy3y2f810VjtMaks5rMuDGgaJpZM4LUkwV>
.
|
Yes, it's possible but it will make the logic complicated. Because I'm not sure if @pwittrock @bgrant0607 will like this. Need their comments before going further. |
"Random" string -- no. We have other cases where a single field isn't sufficient as a merge key, such as lists of object references. I think treating all fields specified by the user's config as the key works in all cases I've thought of. I'd probably make that a new patch strategy. It would probably even be backward compatible. |
@bgrant0607 Are you thinking this would support only the primitive fields as part of the unique/merge key? Or would this strategy only be valid for types that contain only primitive fields? Based on your description, the merge key for this strategy would be dynamic and defined by the client. Have we considered supporting merge keys that are multiple values, but otherwise treated the same as single-value merge keys (in this case it could be <port,protocol> or <port,destinationport,protocol>)? |
@pwittrock No, I was thinking we'd use all fields specified by the user as the key, including nested maps. Multiple keys would also work and may be less error-prone. |
For this particular case "" is a valid name in only in the case of a single
entry, and that is enforced by validation, so if "" was allowed it would
just work?
…On Wed, Jan 11, 2017 at 7:10 PM, Brian Grant ***@***.***> wrote:
@pwittrock <https://github.com/pwittrock> No, I was thinking we'd use all
fields specified by the user as the key, including nested maps. Multiple
keys would also work and may be less error-prone.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#39188 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFVgVDqLA1mRB-bEcODC1TQDuFNPXi_7ks5rRZmmgaJpZM4LUkwV>
.
|
@thockin I was thinking something similar yesterday. Something like: If len(patch list) == 1 && merge key is missing from patch list entry && len(dest list) == 1 && merge key is missing from dest list entry My biggest hesitation here was that it adds some complexity to the already poorly understood merge semantics. There would probably be some weird edge cases like what happens when we add the merge key to the existing entry. e.g. If len(patch list) == 1 && merge key exists in the list entry && len(dest list) == 1 && merge key is missing from dest list entry |
@ymqytw Can you make sure this gets resolved in 1.7? |
@pwittrock If using the approach in #39188 (comment), then yes. |
Lets sync up with @thockin to determine the minimal requirements for correctness. I don't want to introduce a partial solution since it will create a maintenance burden. Once the merge-key audit is complete, lets come up with a proposal. I think it is ok if we write a design proposal. I will make sure it gets appropriately reviewed within ~weeks. |
I've run into a lot of issues with ports being PATCHed wrongly through 1.6. Do we believe these issues are fixed now? I can't seem to repro them, but I wasn't dilligent in cataloging them all... |
I guess the pain you have when PATCHing the ports is caused by the bug of using the wrong json pkg in the api server (#42488). It has been fixed by #42489. issues with ports:
|
Has this been fixed? I'm seeing both ports on describe: › kubectl describe svc nginx
Name: nginx
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=nginx
Type: NodePort
IP: 10.0.0.133
Port: udp 30420/UDP
NodePort: udp 30420/UDP
Endpoints: 172.17.0.4:30420
Port: tcp 30420/TCP
NodePort: tcp 30420/TCP
Endpoints: 172.17.0.4:30420
Session Affinity: None
Events: <none> |
@jamiehannaford Not yet. |
Yes, I think that would be part of option 1. You still have existing services out there with a single port without a name though. I'm not sure how you can still allow those to be updated. |
I am having same problem creating a ClusterIP service with same port for TCP and UDP... and reading through this thread, it sounds like it's not fixed after all this time. But, I'm puzzled by kube-dns under kube-system namespace. It's also a ClusterIP, but it seems to have same port for TCP and UDP:
I looked at the service definition yaml (https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/deployments/kube-dns.yaml#L15) and it's very simple. I have the same syntax for my service.yaml, but it only shows TCP... well, whichever is on top. Anybody know what's different about kube-dns that makes this work? |
@brianpursley You forgot 4. Use server-side apply and the new listMapKey that allow for multiple merge keys. |
There's an aspect here that adds some complication. The discussion has been centered around definitions that are the same port but vary by protocol. However, the current behavior accepts yaml that has definitions that vary only by name. It won't function as one might expect, but the yaml is accepted. Conceptually, there's an argument in support of multiple names for a single port: Ports have port aliases - I can refer to port 25 as either smtp or as mail. |
This seems fixed in 1.20 (maybe before but I have 20 at hand :):
Are there other aspects of this that are still pending? |
@thockin You're able to patch both there and get the expected results? Here is 1.20. This is where you'd expect a change, right? |
With regards to port name aliasing (which I think is related to the second part of that issue's name, "wrong merge key"), the semantics going forward for this list is that the associative key is SSA will not work with this, and field management will fail on objects that don't follow the rule of unique associative keys. It's currently accepted by validation, but ssa code can't accept it. We should probably deprecate lists with duplicate port/protocol pairs in the future and eventually forbid them through validation (using some ratcheting of some sort). |
Kubernetes Version: v1.20.0+bafe72f coredns.yaml
We now have a service that is working for UDP! We realize that we forgot a critical item for DNS, TCP! -- DOH!
Add to ports section:
Only one of the port 53 services will be realized on the service.
newcoredns.yaml
This issue only occurs when trying to patch a previously configured service. Workaround - delete and re-create service <-- Not optimal, particularly if you are doing something that IPs matter - cluster admin at least can specify IP for a service as long as it hasn't been stomped -- (IPs sometimes matter for things like dns inside a cluster, which we are doing for reasons that are fully valid OK :-P ) |
No. Not option (3) please. There are use cases for this - like DNS and supporting both H2-over-TLS and H3-over-QUIC and so on. |
Was this actually fixed with SSA? How? The API and docs still say the I think the best way to fix this issue is to add compound key support:
This way legacy clients still function in the current (broken) way. And new clients can take advantage of the compound key. |
We fixed it more or less how you describe it, but only in SSA. The compound key is set through the |
Hi @apelisse I'm curious to know if the change you mentioned above to leverage that compound key needs to be pulled into the Helm CLI as well so folks utilizing Helm for deployment won't run into this issue. I was following along since one of our customers hit this when running
|
I'm not familiar enough with |
cc @seans3, might be related to our discussion today :-) |
This bug still exists and made it impossible for me to use Consul's agents - ports are exposed (with different name) on both TCP and UDP only one was working. Again, applying a config or a applying a patch didn't fix it - I had to destroy the daemonset and recreate it, only then was the hostPort for both UDP and TCP preserved (at least in 1.23.1) |
Hello @apelisse, can you please describe more how to specify |
This is a property of the type rather than a property that you would set when you apply, so the author has to consider that when they create their CRD. |
User reported:
I am running a service with both TCP and UDP:
but kubectl describe service shows only UDP.
When I change the order then it shows only TCP.
This looks like using the wrong mergeKey, and indeed the code backs that up:
2508 type ServiceSpec struct {
2509 // The list of ports that are exposed by this service.
2510 // More info: http://kubernetes.io/docs/user-guide/services#virtual-ips-and-service-proxies
2511 Ports []ServicePort
json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"port" protobuf:"bytes,1,rep,name=ports"
The key should probably be "name", though that can be empty if there is a single port only - is that a problem? @ymqytw ?
The text was updated successfully, but these errors were encountered: