External Postman to Pod 404 not found (new to k8)

[j4t1nd3r]

What happened:

We have a k8 cluster which contains apis that run in pods. Each API has its own namespace and pod.
The deployment for the pods is done with Terraform and Azure CI/D Pipelines.
The src files for previous API deployments that run in pods all run on dotnet6, they have no routing issues.
The devs have created a new set of source files on dotnet7 for a new API. They can reach the health endpoint locally in Visual Studio.

Once the pod is deployed to our cluster, we can connect to generic-dotnet7-api pod, curl the health endpoint, 200 ok return.
We can curl the same endpoint from a pod in another namespace, 200 ok return.
We can also curl the same endpoint from the ingress-nginx-controller, 200 ok return.

When we run an external (internet) postman request to the same endpoint, 404 not found.
We can see the request reach the ingress-nginx-controller via log analytics workspace logs for the ingress-nginx-controller pod.
We can reach other endpoints on other pods in different namespaces via postman. We do not believe this to be an Azure Front Door issue.

What you expected to happen:
From postman, making an external request to the endpoint receive an ok 200 response.

We currently believe there is a routing issue or some kind of problem once the external request is received by the ingress-nginx-controller it cannot route the request to the pod.
It is possible there is a misconfiguration in the src files when going from dotnet6 to dotnet7, I cant say I am not a dev.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

ingress-nginx-controller-6c5dcd58d-bsksw:/$ nginx-ingress-controller --version
bash: nginx-ingress-controller: command not found

NGINX Ingress controller
  Release:       v1.7.0
  Build:         72ff21ed9e26cb969052c753633049ba8a87ecf9
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Kubernetes version (use kubectl version):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.25.6
WARNING: version difference between client (1.27) and server (1.25) exceeds the supported minor version skew of +/-1

Environment:

• Cloud provider or hardware configuration: AKS
• OS (e.g. from /etc/os-release): Alpine Linux v3.17
• Kernel (e.g. uname -a): Linux ingress-nginx-controller-6c5dcd58d-bsksw 5.15.0-1041-azure #48-Ubuntu SMP Tue Jun 20 20:34:08 UTC 2023 x86_64 Linux
• Install tools:
  • CI/CD pipeline in Azure Devops
• Basic cluster related info:
  • kubectl version 1.25
  • kubectl get nodes -o wide

NAME                              STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-default-34372747-vmss00002v   Ready    agent   13d   v1.25.6   10.10.21.10   <none>        Ubuntu 22.04.2 LTS   5.15.0-1042-azure   containerd://1.6.18+azure-1
aks-default-34372747-vmss00002w   Ready    agent   13d   v1.25.6   10.10.21.29   <none>        Ubuntu 22.04.2 LTS   5.15.0-1041-azure   containerd://1.6.18+azure-1
aks-default-34372747-vmss00002x   Ready    agent   13d   v1.25.6   10.10.21.47   <none>        Ubuntu 22.04.2 LTS   5.15.0-1041-azure   containerd://1.6.18+azure-1

• How was the ingress-nginx-controller installed:
  • If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx   ingress-nginx   18   2023-05-10 13:15:57.5873403 +0000 UTC   deployed   ingress-nginx-4.6.0   1.7.0

• If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
  service:
    annotations:
      service.beta.kubernetes.io/azure-dns-label-name: acp-stg-team-core-infra

• if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
  Just the one ingress controller

• Current State of the controller:

  • kubectl describe ingressclasses

Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.7.0
              helm.sh/chart=ingress-nginx-4.6.0
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   k8s.io/ingress-nginx
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

NAME                                           READY   STATUS    RESTARTS   AGE   IP            NODE                              NOMINATED NODE   READINESS GATES
pod/ingress-nginx-controller-6c5dcd58d-bsksw   1/1     Running   0          9d    10.10.21.51   aks-default-34372747-vmss00002x   <none>           <none>

NAME                                         TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)                      AGE    SELECTOR
service/ingress-nginx-controller             LoadBalancer   10.0.7.161   20.108.203.206   80:31656/TCP,443:30843/TCP   133d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
service/ingress-nginx-controller-admission   ClusterIP      10.0.159.3   <none>           443/TCP                      133d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                                                                                    SELECTOR
deployment.apps/ingress-nginx-controller   1/1     1            1           133d   controller   registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

NAME                                                  DESIRED   CURRENT   READY   AGE    CONTAINERS   IMAGES                                                                                                                    SELECTOR
replicaset.apps/ingress-nginx-controller-59bcdf459b   0         0         0       133d   controller   k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a        app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=59bcdf459b
replicaset.apps/ingress-nginx-controller-6c5dcd58d    1         1         1       132d   controller   registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=6c5dcd58d

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:             ingress-nginx-controller-6c5dcd58d-bsksw
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             aks-default-34372747-vmss00002x/10.10.21.47
Start Time:       Sun, 13 Aug 2023 22:18:25 +0100
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=ingress-nginx
                  app.kubernetes.io/part-of=ingress-nginx
                  app.kubernetes.io/version=1.7.0
                  helm.sh/chart=ingress-nginx-4.6.0
                  pod-template-hash=6c5dcd58d
Annotations:      <none>
Status:           Running
IP:               10.10.21.51
IPs:
  IP:           10.10.21.51
Controlled By:  ReplicaSet/ingress-nginx-controller-6c5dcd58d
Containers:
  controller:
    Container ID:  containerd://e04bca85daa6a890e5a50d50d38184d7e3f1f9acdf98370863ef6970b5fd5fa0
    Image:         registry.k8s.io/ingress-nginx/controller:v1.7.0@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:7612338342a1e7b8090bef78f2a04fffcadd548ccaabe8a47bf7758ff549a5f7
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Sun, 13 Aug 2023 22:18:25 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6c5dcd58d-bsksw (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jjjj7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-jjjj7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.7.0
                          helm.sh/chart=ingress-nginx-4.6.0
Annotations:              meta.helm.sh/release-name: ingress-nginx
                          meta.helm.sh/release-namespace: ingress-nginx
                          service.beta.kubernetes.io/azure-dns-label-name: acp-stg-team-core-infra
                          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.0.7.161
IPs:                      10.0.7.161
LoadBalancer Ingress:     20.108.203.206
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31656/TCP
Endpoints:                10.10.21.51:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30843/TCP
Endpoints:                10.10.21.51:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Name:              ingress-nginx-controller-admission
Namespace:         ingress-nginx
Labels:            app.kubernetes.io/component=controller
                   app.kubernetes.io/instance=ingress-nginx
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=ingress-nginx
                   app.kubernetes.io/part-of=ingress-nginx
                   app.kubernetes.io/version=1.7.0
                   helm.sh/chart=ingress-nginx-4.6.0
Annotations:       meta.helm.sh/release-name: ingress-nginx
                   meta.helm.sh/release-namespace: ingress-nginx
Selector:          app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.0.159.3
IPs:               10.0.159.3
Port:              https-webhook  443/TCP
TargetPort:        webhook/TCP
Endpoints:         10.10.21.51:8443
Session Affinity:  None
Events:            <none>

• Current state of ingress object, if applicable:
  • kubectl -n <appnnamespace> get all,ing -o wide

NAME                                       READY   STATUS    RESTARTS   AGE   IP            NODE                              NOMINATED NODE   READINESS GATES
pod/generic-dotnet7-api-6b7d66bddd-67r86   1/1     Running   0          9d    10.10.21.34   aks-default-34372747-vmss00002w   <none>           <none>

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/generic-dotnet7-api   ClusterIP   10.0.8.83    <none>        80/TCP    18d   app.kubernetes.io/component=api,app.kubernetes.io/name=generic-dotnet7-api,app.kubernetes.io/part-of=anchor-platform,bounded-context=generic,platform=acp,scope-name=d01

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                                            SELECTOR
deployment.apps/generic-dotnet7-api   1/1     1            1           18d   api          apukssubshcorinfcr.azurecr.io/acp/generic/dotnet7-api:0.1.1-dotnet7-template.10   app.kubernetes.io/component=api,app.kubernetes.io/name=generic-dotnet7-api,app.kubernetes.io/part-of=anchor-platform,bounded-context=generic,platform=acp,scope-name=d01

NAME                                             DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                                            SELECTOR
replicaset.apps/generic-dotnet7-api-6b7d66bddd   1         1         1       18d   api          apukssubshcorinfcr.azurecr.io/acp/generic/dotnet7-api:0.1.1-dotnet7-template.10   app.kubernetes.io/component=api,app.kubernetes.io/name=generic-dotnet7-api,app.kubernetes.io/part-of=anchor-platform,bounded-context=generic,platform=acp,pod-template-hash=6b7d66bddd,scope-name=d01

NAME                                                      REFERENCE                        TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/generic-dotnet7-api   Deployment/generic-dotnet7-api   1%/95%    1         1         1          18d

NAME                                            CLASS    HOSTS                              ADDRESS          PORTS     AGE
ingress.networking.k8s.io/generic-dotnet7-api   <none>   d01-uks.core.anchor-platform.dev   20.108.203.206   80, 443   18d

• kubectl -n <appnamespace> describe ing <ingressname>

Name:             generic-dotnet7-api
Labels:           app.kubernetes.io/component=api
                  app.kubernetes.io/name=generic-dotnet7-api
                  app.kubernetes.io/part-of=anchor-platform
                  app.kubernetes.io/version=0.1.1-dotnet7-template.10
                  bounded-context=generic
                  platform=acp
                  scope-name=d01
Namespace:        generic-dotnet7-d01
Address:          20.108.203.206
Ingress Class:    <none>
Default backend:  <default>
TLS:
  tls-secret terminates d01-uks.core.anchor-platform.dev
Rules:
  Host                              Path  Backends
  ----                              ----  --------
  d01-uks.core.anchor-platform.dev
                                    /(api/generic/dotnet7(?:/?|$).*)   generic-dotnet7-api:80 (10.10.21.34:80)
Annotations:                        app.kubernetes.io/component: api
                                    app.kubernetes.io/name: generic-dotnet7-api
                                    app.kubernetes.io/part-of: anchor-platform
                                    app.kubernetes.io/version: 0.1.1-dotnet7-template.10
                                    bounded-context: generic
                                    cert-manager.io/cluster-issuer: letsencrypt
                                    kubernetes.io/ingress.class: nginx
                                    nginx.ingress.kubernetes.io/force-ssl-redirect: false
                                    nginx.ingress.kubernetes.io/rewrite-target: /$1
                                    nginx.ingress.kubernetes.io/ssl-redirect: false
                                    platform: acp
                                    scope-name: d01
Events:                             <none>

• If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

curl https://d01-api.anchor-platform.dev/api/generic/dotnet7/_health -v                                                                                                      ─╯
*   Trying 13.107.213.64:443...
* Connected to d01-api.anchor-platform.dev (13.107.213.64) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
> GET /api/generic/dotnet7/_health HTTP/1.1
> Host: d01-api.anchor-platform.dev
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Wed, 23 Aug 2023 13:38:55 GMT
< Content-Length: 0
< Connection: keep-alive
< x-correlation-id: a4bdad64-63e9-4125-96a4-c419719b809c
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< X-Frame-Options: deny
< X-Content-Type-Options: nosniff
< Content-Security-Policy: frame-ancestors 'none'
< Cache-Control: no-store
< x-azure-ref: 20230823T133855Z-yq4dc9duz97p10fcsuwh1mdax80000000atg00000000f2ww
< X-Cache: CONFIG_NOCACHE
<
* Connection #0 to host d01-api.anchor-platform.dev left intact

• Others:
  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Unfortunately I cannot show how to reproduce the issue.
I am new to K8 and this role.

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug

• You got a 404 but the logs from the controller pod are not posted here
• The formatting is bad and not legible
• You can help by posting logs
• You can help by formatting issue description as per markdown so it becomes easy to read

[j4t1nd3r]

Sorry for the delay in response, I have updated the original post to fix the formatting.

The log for the failed CURL on the ingress controller:

10.10.21.29 - - [23/Aug/2023:13:38:55 +0000] "GET /api/generic/dotnet7/_health HTTP/1.1" 404 0 "-" "curl/8.0.1" 478 0.020 [generic-dotnet7-d01-generic-dotnet7-api-80] [] 10.10.21.34:80 0 0.021 404 9f673c7a63fb64d37be576542e27996d

Here is a working example of the ability to CURL a pod in another namespace which is on dotnet6:

10.10.21.47 - - [23/Aug/2023:13:49:29 +0000] "GET /api/test/test/health HTTP/1.1" 200 62 "-" "PostmanRuntime/7.32.3" 686 0.005 [test-test-d01-test-test-api-80] [] 10.10.21.19:80 62 0.004 200 853883bbd19bb5757d7a0613b0ece96c

[longwuyuan]

That 404 error message shows the routing was not a problem.

But because the failing curl request, in full, as it is, raw, from outside the cluster needs to be compared to a working exact same curl request, originating from a adjacent pod, inside the cluster, that returns response 200, its hard to know more from the data posted.

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[j4t1nd3r]

Closing, I believe this is not an nginx issue.