Unstable connection when using "-tcp" suffix in targetPort with Helm chart

[a1expol]

Title: Unstable Redis Connection When Using "-tcp" Suffix in targetPort with Helm chart

What happened:

While deploying Redis through Helm chart and specifying the targetPort with a "-tcp" suffix (e.g. "6379-tcp"), I'm experiencing unstable connections to Redis. The connections frequently drop and are lost.

What you expected to happen:

I expect a reliably stable connection to Redis even when using the "-tcp" suffix. I suspect that this instability may be related to how the ingress configuration or Helm chart is handling this suffix.

NGINX Ingress controller version: 1.8.0

Kubernetes version (run kubectl version):
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.5-gke.1091000

Environment:

• Cloud provider or hardware configuration: Google Cloud Platform, GKE cluster
• OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS
• Kernel (e.g. uname -a): Linux

How was the ingress-nginx-controller installed:
Helm Chart.

ingress-test               default 15              2024-07-11 11:30:07.167762801 +0300 EEST        deployed          ingress-nginx-4.7.0                     1.8.0

USER-SUPPLIED VALUES:
controller:
  containerPort:
    redis-dev: 6379
    redis-prod: 6380
  ingressClass: research-nginx
  ingressClassResource:
    controllerValue: k8s.io/research-ingress-nginx
    name: research-nginx
  publishService:
    enabled: true
  service:
    internal:
      nodePorts:
        tcp:
        - 31100
        - 31101
    loadBalancerSourceRanges:
    - xxx.xxx.xxx.xxx/xx
    type: LoadBalancer
tcp:
  "6379": default/redis-stack-server-dev:6379
  "6380": default/redis-stack-server-prod:6380

Current State of the controller:
Ingress controller is functional, but the traffic on port 6379 is being handled incorrectly.

Name: research-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-test
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.8.0
helm.sh/chart=ingress-nginx-4.7.0
Annotations: meta.helm.sh/release-name: ingress-test
meta.helm.sh/release-namespace: default
Controller: k8s.io/research-ingress-nginx
Events:

Name:             ingress-test-ingress-nginx-controller-985dbdcj4nhh
Namespace:        default
Priority:         0
Service Account:  ingress-test-ingress-nginx
Node:             gke-common-t4-node-po-53125293-9gw8/10.100.0.38
Start Time:       Wed, 10 Jul 2024 17:39:04 +0300
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-test
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.8.0
                  helm.sh/chart=ingress-nginx-4.7.0
                  pod-template-hash=985dbdc74
Annotations:      cni.projectcalico.org/containerID: 04d5736e5a6e97eca16b21c05bad032f4db0be89aa236728ef3845e1f42cc2a8
                  cni.projectcalico.org/podIP: 10.48.3.249/32
                  cni.projectcalico.org/podIPs: 10.48.3.249/32
Status:           Running
IP:               10.48.3.249
IPs:
  IP:           10.48.3.249
Controlled By:  ReplicaSet/ingress-test-ingress-nginx-controller-985dbdc74
Containers:
  controller:
    Container ID:  containerd://a617fa6b84112609ac290b58459de2c977c3929cf8cef30c669077181f88533c
    Image:         registry.k8s.io/ingress-nginx/controller:v1.8.0@sha256:744ae2afd433a395eeb13dc03d3313facba92e96ad71d9feaafc85925493fee3
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:744ae2afd433a395eeb13dc03d3313facba92e96ad71d9feaafc85925493fee3
    Ports:         80/TCP, 443/TCP, 6379/TCP, 6380/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 6379/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-test-ingress-nginx-controller
      --election-id=ingress-test-ingress-nginx-leader
      --controller-class=k8s.io/research-ingress-nginx
      --ingress-class=research-nginx
      --configmap=$(POD_NAMESPACE)/ingress-test-ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/ingress-test-ingress-nginx-tcp
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
    State:          Running
      Started:      Wed, 10 Jul 2024 17:39:05 +0300
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:      ingress-test-ingress-nginx-controller-985dbdcj4nhh (v1:metadata.name)
      POD_NAMESPACE:  default (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9w7vm (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-test-ingress-nginx-admission
    Optional:    false
  kube-api-access-9w7vm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Name:                        ingress-test-ingress-nginx-controller
Namespace:                   default
Labels:                      app.kubernetes.io/component=controller
                             app.kubernetes.io/instance=ingress-test
                             app.kubernetes.io/managed-by=Helm
                             app.kubernetes.io/name=ingress-nginx
                             app.kubernetes.io/part-of=ingress-nginx
                             app.kubernetes.io/version=1.8.0
                             helm.sh/chart=ingress-nginx-4.7.0
Annotations:                 meta.helm.sh/release-name: ingress-test
                             meta.helm.sh/release-namespace: default
Selector:                    app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-test,app.kubernetes.io/name=ingress-nginx
Type:                        LoadBalancer
IP Family Policy:            SingleStack
IP Families:                 IPv4
IP:                          10.52.8.129
IPs:                         10.52.8.129
LoadBalancer Ingress:        34.31.132.131
Port:                        http  80/TCP
TargetPort:                  http/TCP
NodePort:                    http  31388/TCP
Endpoints:                   10.48.3.249:80
Port:                        https  443/TCP
TargetPort:                  https/TCP
NodePort:                    https  32016/TCP
Endpoints:                   10.48.3.249:443
Port:                        6379-tcp  6379/TCP
TargetPort:                  6379-tcp/TCP
NodePort:                    6379-tcp  31100/TCP
Endpoints:
Port:                        6380-tcp  6380/TCP
TargetPort:                  6380-tcp/TCP
NodePort:                    6380-tcp  31024/TCP
Endpoints:                   10.48.3.249:6380
Session Affinity:            None
External Traffic Policy:     Cluster
LoadBalancer Source Ranges:  xxx.xxx.xxx.xxx/xx
Events:                      <none>

Current state of ingress object, if applicable:

Name:             redis-stack-server-dev
Labels:           app=redis-stack-server-dev
                  app.kubernetes.io/managed-by=Helm
Namespace:        default
Address:          34.31.132.131
Ingress Class:    research-nginx
Default backend:  <default>
TLS:
  redis-dev-tls terminates my-redis.domain.name
Rules:
  Host                        Path  Backends
  ----                        ----  --------
  my-redis.domain.name
                              /   redis-stack-server-dev:6379 (<none>)
Annotations:                  cert-manager.io/cluster-issuer: acme-devops-delivery
                              kubernetes.io/ingress.class: research-nginx
                              meta.helm.sh/release-name: redis-dev
                              meta.helm.sh/release-namespace: default
                              nginx.ingress.kubernetes.io/proxy-buffering: off
                              nginx.ingress.kubernetes.io/proxy-read-timeout: 86400
                              nginx.ingress.kubernetes.io/proxy-send-timeout: 86400
                              nginx.ingress.kubernetes.io/ssl-passthrough: true
                              nginx.ingress.kubernetes.io/ssl-redirect: true
Events:                       <none>

How to reproduce this issue:
To reproduce the problem, you need to have a Kubernetes cluster with the redis-stack-server installed on port 6379. The problem is very simple to reproduce: you just need to expose port 6379 on the redis-stack-server service, port needs to be exposed through the Helm chart (it is necessary for the "-tcp" suffix to be added to the targetPort), and try to run redis-cli -h your-redis.domain.name -p 6379 from the local machine several times.

Anything else we need to know:
I would like to add that without the "-tcp" suffix the problem disappears and the service responds consistently. But with this suffix, the connection becomes unstable.

$ redis-cli -h my-redis.domain.name -p 6379
Could not connect to Redis at my-redis.domain.name:6379: Connection refused
not connected>
$ redis-cli -h my-redis.domain.name -p 6379
my-redis.domain.name:6379>
$ redis-cli -h my-redis.domain.name -p 6379
my-redis.domain.name:6379>
$ redis-cli -h my-redis.domain.name -p 6379
Could not connect to Redis at my-redis.domain.name:6379: Connection refused
not connected>
$ redis-cli -h my-redis.domain.name -p 6379
Could not connect to Redis at my-redis.domain.name:6379: Connection refused
not connected>

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug
/kind support

Delete the ingress named "redis-stack-server-dev"

[a1expol]

/remove-kind bug /kind support

Delete the ingress named "redis-stack-server-dev"

But for me, the whole point is to use this ingress with the domain. If you do it manually using the example https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/exposing-tcp-udp-services.md then everything works perfectly, which is what I want to achieve through helm. But I can't understand what the -tcp suffix does in targetPort, what is its role there in general.

[longwuyuan]

I am not clear on the question. I am wondering if you are asking that you can set a value like "6379 -tcp" to the field

% k explain service.spec.ports.targetPort
KIND:       Service
VERSION:    v1

FIELD: targetPort <IntOrString>


DESCRIPTION:
    Number or name of the port to access on the pods targeted by the service.
    Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If
    this is a string, it will be looked up as a named port in the target Pod's
    container ports. If this is not specified, the value of the 'port' field is
    used (an identity map). This field is ignored for services with
    clusterIP=None, and should be omitted or set equal to the 'port' field. More
    info:
    https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
    IntOrString is a type that can hold an int32 or a string.  When used in JSON
    or YAML marshalling and unmarshalling, it produces or consumes the inner
    type.  This allows you to have, for example, a JSON field that can accept a
    name or number.

[a1expol]

I am not clear on the question. I am wondering if you are asking that you can set a value like "6379 -tcp" to the field

This suffix is ​​added independently in the template https://github.com/kubernetes/ingress-nginx/blob/a6727d81e7cd510ac5fed4bb1773f9aefefe8fbe/charts/ingress-nginx/templates/controller-service.yaml#L84C1-L85C1. I understand that this is a reference to the port name, but it is not working correctly in my case. Because of it, I can't use expose tcp service

[longwuyuan]

Sorry, with the little I understand, it may be that I will need to helm template the chart with a value suggested by you, to a key in the values.yaml file . That will be relatvely more ambiguous when compared to directly asking you about what precise value do you desire to set to which field of which K8S object.

So can you kindly help and mention both the precise field like my example "k explain service.spec.ports.targetPort" and that what precise value you want to set to that field, in the final K8S object that is created on the cluster.

After reading all the posts here I made a guess of the field and the value and asked you that directly. So kindly help by going past all the templating and the yaml and json to apiserver etc etc, and kindly type out the precise field and the complete precise value you want to set, during a helm install.

Confused because I am not aware of any field in any K8S object where "-tcp" is a valid value (in the context of the ingress-controller I mean)

[a1expol]

Sorry, with the little I understand, it may be that I will need to helm template the chart with a value suggested by you, to a key in the values.yaml file . That will be relatvely more ambiguous when compared to directly asking you about what precise value do you desire to set to which field of which K8S object.

So can you kindly help and mention both the precise field like my example "k explain service.spec.ports.targetPort" and that what precise value you want to set to that field, in the final K8S object that is created on the cluster.

After reading all the posts here I made a guess of the field and the value and asked you that directly. So kindly help by going past all the templating and the yaml and json to apiserver etc etc, and kindly type out the precise field and the complete precise value you want to set, during a helm install.

Confused because I am not aware of any field in any K8S object where "-tcp" is a valid value (in the context of the ingress-controller I mean)

If I understand you correctly, I need to change the targetPort value in line 84

ingress-nginx/charts/ingress-nginx/templates/controller-service.yaml

Line 84 in a6727d8

targetPort: {{ if $.Values.portNamePrefix }}{{ $.Values.portNamePrefix }}-{{ end }}{{ $key }}-tcp

to
{{ (split ":" $value)._1 }}
as a reference to the service port.
How it should works:

Let me know if I misunderstood you. Thank you.

[longwuyuan]

Hi,

Sorry, I am not sure what you understood as there is no clear confirmed conclusive info on what is your desired expected end-result.

Is there any chance you can name the exact name of the field. For example is this the field that you want to configure ;

service.spec.ports.targetPort

if yes, what is the real exact complete value you want to set for this field ?

[a1expol]

Hi,

Sorry, I am not sure what you understood as there is no clear confirmed conclusive info on what is your desired expected end-result.

Is there any chance you can name the exact name of the field. For example is this the field that you want to configure ;

service.spec.ports.targetPort

if yes, what is the real exact complete value you want to set for this field ?

Yes, this is the field I want to configure. In the value of this field, I want to see the destination port of the service that I refer to in the "tcp" value

[longwuyuan]

I already mentioned to you that the K8S spec for valid values is this

% k explain service.spec.ports.targetPort
KIND:       Service
VERSION:    v1

FIELD: targetPort <IntOrString>


DESCRIPTION:
    Number or name of the port to access on the pods targeted by the service.
    Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If
    this is a string, it will be looked up as a named port in the target Pod's
    container ports. If this is not specified, the value of the 'port' field is
    used (an identity map). This field is ignored for services with
    clusterIP=None, and should be omitted or set equal to the 'port' field. More
    info:
    https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
    IntOrString is a type that can hold an int32 or a string.  When used in JSON
    or YAML marshalling and unmarshalling, it produces or consumes the inner
    type.  This allows you to have, for example, a JSON field that can accept a
    name or number.

So if you want to put a number there, then put the number like 6379.

Its not clear to me if you want to put a name there like "6379-tcp". Or you want to put a number and also a flag there like "6379 -tcp" .

In either case, the spec field is explained by the apiserver as per my copy/paste above. It contains precise desciption of what values are valid and accepted.

[a1expol]

I already mentioned to you that the K8S spec for valid values is this

% k explain service.spec.ports.targetPort
KIND:       Service
VERSION:    v1

FIELD: targetPort <IntOrString>


DESCRIPTION:
    Number or name of the port to access on the pods targeted by the service.
    Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If
    this is a string, it will be looked up as a named port in the target Pod's
    container ports. If this is not specified, the value of the 'port' field is
    used (an identity map). This field is ignored for services with
    clusterIP=None, and should be omitted or set equal to the 'port' field. More
    info:
    https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
    IntOrString is a type that can hold an int32 or a string.  When used in JSON
    or YAML marshalling and unmarshalling, it produces or consumes the inner
    type.  This allows you to have, for example, a JSON field that can accept a
    name or number.

So if you want to put a number there, then put the number like 6379.

Its not clear to me if you want to put a name there like "6379-tcp". Or you want to put a number and also a flag there like "6379 -tcp" .

In either case, the spec field is explained by the apiserver as per my copy/paste above. It contains precise desciption of what values are valid and accepted.

I want to set the 'service.spec.ports.targetPort' field to a numeric value (for example '6379'). The issue I'm having is that when creating the Load Balancer resource via Helm, a '-tcp' suffix is getting added on (making it '6379-tcp') which is causing my application to malfunction. I need to figure out how to prevent this '-tcp' suffix from being added when using Helm.

[longwuyuan]

Thank you for finally providing the simple easy understandable discussion.

Redis is not HTTP or HTTPS so creating a ingress for redis is INVALID. Delete the ingress.

[a1expol]

Thank you for finally providing the simple easy understandable discussion.

Redis is not HTTP or HTTPS so creating a ingress for redis is INVALID. Delete the ingress.

Understood, Redis does not use HTTP or HTTPS protocols, thus creating an Ingress for Redis is incorrect. However, when I manually expose the TCP service, everything works as presented in this repository's documentation. Therefore, I am curious as to why the Helm configuration differs from the documentation?

[longwuyuan]

Can you please explain in complete and elaborate details by what you mean different

…

On Wed, 17 Jul, 2024, 13:18 Oleksandr Polishchuk, ***@***.***> wrote: Thank you for finally providing the simple easy understandable discussion. Redis is not HTTP or HTTPS so creating a ingress for redis is INVALID. Delete the ingress. Understood, Redis does not use HTTP or HTTPS protocols, thus creating an Ingress for Redis is incorrect. However, when I manually expose the TCP service, everything works as presented in this repository's documentation. Therefore, I am curious as to why the Helm configuration differs from the documentation? — Reply to this email directly, view it on GitHub <#11603 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWWY2TXIMUFR6ZQEMJLZMYOWPAVCNFSM6AAAAABKW63S4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZSGY2TKMJZGU> . You are receiving this because you commented.Message ID: ***@***.***>

[a1expol]

Can you please explain in complete and elaborate details by what you mean
different

By 'different', I mean why is there a discrepancy between the manual TCP service exposure method outlined in the documentation and the Helm chart deployment. In the documentation, we manually specify the destination service port for the exposure in targetPort. However, in the Helm chart, this isn't specified and only the value from the 'name' parameter is duplicated. Why is that?

[longwuyuan]

I don't understand the difference clearly.
Can you please help and copy paste the 2 different information pieces that are conflicting or different. Kindly ensure to provide links also.

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.