Skip to content

Regression when using ExternalName services #13081

New issue
New issue
Closed
#13154
Closed
Regression when using ExternalName services#13081
#13154
Labels
kind/bugCategorizes issue or PR as related to a bug.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
@philpep

Description

@philpep
philpep
opened on Mar 25, 2025 · edited by philpep

After upgrade from chart 4.11.3 to 4.12.1 my ingresses using ExternalName services aren't working anymore (http 503 Service Temporarily Unavailable).

Logs of the controller shows:

2025/03/25 10:22:17 [error] 26#26: *33847 lua entry thread aborted: runtime error: /etc/nginx/lua/balancer.lua:78: bad argument #1 to 'ipairs' (table expected, got nil)
stack traceback:
coroutine 0:
        [C]: in function 'ipairs'
        /etc/nginx/lua/balancer.lua:78: in function 'resolve_external_names'
        /etc/nginx/lua/balancer.lua:114: in function 'sync_backend'
        /etc/nginx/lua/balancer.lua:148: in function </etc/nginx/lua/balancer.lua:146>, context: ngx.timer

Example:

apiVersion: v1
kind: Service
metadata:
  name: example
spec:
  type: ExternalName
  externalName: internal.example.com
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example.com
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: example
                port:
                  number: 443
  tls:
    - hosts:
      - example.com

Activity

philpep
added
kind/bugCategorizes issue or PR as related to a bug.
on Mar 25, 2025
k8s-ci-robot
added
needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
on Mar 25, 2025
k8s-ci-robot

k8s-ci-robot commented on Mar 25, 2025

@k8s-ci-robot
k8s-ci-robot
on Mar 25, 2025
Contributor

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot
added
needs-priority
on Mar 25, 2025
strongjz
added this to [SIG Network] Ingress NGINXon Mar 25, 2025
philpep

philpep commented on Mar 25, 2025

@philpep
philpep
on Mar 25, 2025
Author

It might relate to #13076 but seems a different issue.

philpep

philpep commented on Mar 25, 2025

@philpep
philpep
on Mar 25, 2025
Author

Seems the bug was introduced in 4.11.5 release (4.11.4 is OK).

philpep

philpep commented on Mar 25, 2025

@philpep
philpep
on Mar 25, 2025
Author

I think issue is from this commit c6c5b48

tolix1

tolix1 commented on Mar 26, 2025

@tolix1
tolix1
on Mar 26, 2025 · edited by tolix1

same issue here. 4.11.5 and 4.12.1 impacted

Gacko

Gacko commented on Mar 26, 2025

@Gacko
Gacko
on Mar 26, 2025
Member

Please do not override the issue template and instead fill it as requested. This is important for reproducing your issue.

Also please add information about what internal.example.com is pointing at. Are these IP addresses? Is it a CNAME?

philpep

philpep commented on Mar 26, 2025

@philpep
philpep
on Mar 26, 2025
Author

Please do not override the issue template and instead fill it as requested. This is important for reproducing your issue.

Sorry will do better next time.

Also please add information about what internal.example.com is pointing at. Are these IP addresses? Is it a CNAME?

Yes it's a resolvable CNAME.

sepich

sepich commented on Mar 26, 2025

@sepich
sepich
on Mar 26, 2025
Contributor

Another possibly related issue is that nginx.ingress.kubernetes.io/default-backend stopped working for externalName Ingresses.

Here is example yamls 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
spec:
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - image: inanimate/echo-server
          name: echo
      enableServiceLinks: false
---
apiVersion: v1
kind: Service
metadata:
  name: echo
spec:
  selector:
    app: echo
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: google.com
    nginx.ingress.kubernetes.io/custom-http-errors: "301"
    nginx.ingress.kubernetes.io/default-backend: echo
    prometheus.io/probe: "false"
  name: test
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: ext
      port:
        number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: ext
spec:
  type: ExternalName
  externalName: google.com

So in example we trying to use custom errors:
https://kubernetes.github.io/ingress-nginx/examples/customization/custom-errors/
In this case we access google.com via http/80 and get redirect to https:

$ curl -si google.com | head -1
HTTP/1.1 301 Moved Permanently

Then we try to forward this 301 to be served from out service echo.
It works in v1.11.4:

127.0.0.1 - - [26/Mar/2025:15:22:12 +0000] "GET / HTTP/1.1" 200 1774 "-" "curl/8.7.1" 67 0.033 [custom-default-backend-default-echo] [] 142.250.186.174:80 : 10.244.0.8:8080 0 : 1774 0.030 : 0.003 301 : 200 0cc21efb7ceba0b69745bc9afc359738

but broken in v1.11.5:

127.0.0.1 - - [26/Mar/2025:15:15:00 +0000] "GET / HTTP/1.1" 502 150 "-" "curl/8.7.1" 67 0.036 [custom-default-backend-default-echo] [] 216.58.206.78:80 : 0.0.0.1:80 0 : 0 0.035 : 0.001 301 : 502 6958aeaac151beea2b82922fd8da30fd
2025/03/26 15:15:00 [error] 38#38: *24724 connect() failed (113: Host is unreachable) while connecting to upstream, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "test"
2025/03/26 15:15:00 [warn] 38#38: *24724 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, server: _, request: "GET / HTTP/1.1", upstream: "http://0.0.0.1:80/", host: "test"
Gacko

Gacko commented on Mar 26, 2025

@Gacko
Gacko
on Mar 26, 2025
Member

Yes it's a resolvable CNAME.

Out of curiosity: Can you make it an A / AAAA record? Just wanna see if it's related to the record type.

philpep

philpep commented on Mar 27, 2025

@philpep
philpep
on Mar 27, 2025
Author

Yes it's a resolvable CNAME.

Out of curiosity: Can you make it an A / AAAA record? Just wanna see if it's related to the record type.

In my case it's a A record (without AAAA) resolving into a RFC1918 reserved ip address, outside of the k8s cluster (e.g. 192.168.42.12)

vasili439

vasili439 commented on Mar 27, 2025

@vasili439
vasili439
on Mar 27, 2025

Yes it's a resolvable CNAME.

Out of curiosity: Can you make it an A / AAAA record? Just wanna see if it's related to the record type.

In my case it's a A record (without AAAA) resolving into a RFC1918 reserved ip address, outside of the k8s cluster (e.g. 192.168.42.12)

in my case the same: I've tried as a quick fix to replace RFC1918 IP with temp domain name (A record) but no luck. HTTP503 with plain IP address and with DNS A record.

24 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    [SIG Network] Ingress NGINX

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @neerfri@philpep@Gacko@jasonhadi@Confushion

      Issue actions
        Regression when using ExternalName services · Issue #13081 · kubernetes/ingress-nginx