namespaced ingress doesn't work as expected #11222
Labels
kind/feature
Categorizes issue or PR as related to a new feature.
lifecycle/frozen
Indicates that an issue or PR should not be auto-closed due to staleness.
needs-priority
needs-triage
Indicates an issue or PR lacks a `triage/foo` label and requires one.
triage/needs-information
Indicates an issue needs more information in order to work on it.
Comments
k8s-ci-robot
added
the
needs-triage
|
This issue is currently awaiting triage. If Ingress contributors determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
10 tasks
|
raise PR #11223 |
|
|
/remove-kind bug |
k8s-ci-robot
added
needs-kind
|
@longwuyuan thanks for your quick response. as requested in PR #11223 please check information from my env as below
# kind get clusters
ingress-nginx-dev
# kubectl get po
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-2vf6g 0/1 Completed 0 16m
ingress-nginx-admission-patch-cxtx8 0/1 Completed 2 16m
ingress-nginx-controller-659c6c4948-pr8jm 1/1 Running 0 16m
test0-574c47cb97-fzhjf 1/1 Running 0 8m11s
# kubectl get clusterrolebinding |grep ingres
ingress-nginx ClusterRole/ingress-nginx 18m
ingress-nginx-admission ClusterRole/ingress-nginx-admission 18m
# kubectl get clusterrolebinding ingress-nginx -ojsonpath='{.roleRef} {"\n"} {.subjects}'
{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"ingress-nginx"}
[{"kind":"ServiceAccount","name":"ingress-nginx","namespace":"ingress-nginx"}]3) for namespaced deployment, nginx-ingress-controller is not supposed to have cluster level permission. so remove cluster rolebinding "ingress-nginx" # kubectl delete clusterrolebinding ingress-nginx
clusterrolebinding.rbac.authorization.k8s.io "ingress-nginx" deleted
apiVersion: v1
kind: Service
metadata:
labels:
app: test0
name: test0
namespace: ingress-nginx
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: test0
type: ClusterIPmore test0.deploy.yaml apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: test0
name: test0
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app: test0
template:
metadata:
labels:
app: test0
spec:
containers:
- image: nginx:alpine
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
restartPolicy: Alwaysmore test0.ing.yaml apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-class-name-no-perm
namespace: ingress-nginx
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- http:
paths:
- backend:
service:
name: test0
port:
number: 80
path: /demo/http1
pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-invalid-ingress-class-name-no-perm
namespace: ingress-nginx
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx-not-match
rules:
- http:
paths:
- backend:
service:
name: test0
port:
number: 80
path: /demo/http2
pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx-annotation
nginx.ingress.kubernetes.io/rewrite-target: /
name: ingress-from-annotation
namespace: ingress-nginx
spec:
rules:
- http:
paths:
- backend:
service:
name: test0
port:
number: 80
path: /demo/http3
pathType: Prefix
# kubectl get po|grep ingress-nginx-controller-7fd476c957-6t5cp
ingress-nginx-controller-7fd476c957-6t5cp 1/1 Running 0 14m
# kubectl exec -it ingress-nginx-controller-7fd476c957-6t5cp -- bash -c 'more /etc/nginx/nginx.conf|grep /demo'
location ~* "^/demo/http1" {
set $location_path "/demo/http1";
rewrite "(?i)/demo/http1" / break;
8) Check deployed demo application. "/demo/http1" is picked up when "nginx-ingress-controller" pod don't have permission to access the cluster level resource "IngressClass"
# kubectl get svc -n$ns|grep ingress
ingress-nginx-controller NodePort 10.96.31.140 <none> 80:32691/TCP,443:31098/TCP 67m
# ip=10.96.31.140
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' --noproxy $ip http://$ip:80/demo/http1 ; echo $http_code
200
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' --noproxy $ip http://$ip:80/demo/http2 ; echo $http_code
404
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' --noproxy $ip http://$ip:80/demo/http3 ; echo $http_code
404
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' -k --noproxy $ip https://$ip:443/demo/http1 ; echo $http_code
200
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' -k --noproxy $ip https://$ip:443/demo/http2 ; echo $http_code
404
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' -k --noproxy $ip https://$ip:443/demo/http3 ; echo $http_code
404
|
|
@yong-jie-gong I request some detailed information which helps reduce the work to be done by others. Is it possible for you ti kindly edit the above message and post information as per hints below ;
And other such information. This is to see the live state of the resources like clusterrole and others from your changes as well the curl command and the other commands that explains how ingress is working after your changes |
|
/kind feature |
k8s-ci-robot
added
kind/feature
|
/triage needs-information |
k8s-ci-robot
added
the
triage/needs-information
Add more information as requested
/root$ helm ls -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-2vf6g 0/1 Completed 0 47h
pod/ingress-nginx-admission-patch-cxtx8 0/1 Completed 2 47h
pod/ingress-nginx-controller-6484977b56-tp7tr 1/1 Running 0 46h
pod/test0-574c47cb97-fzhjf 1/1 Running 0 47h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.96.31.140 <none> 80:32691/TCP,443:31098/TCP 47h
service/ingress-nginx-controller-admission ClusterIP 10.96.33.172 <none> 443/TCP 47h
service/test0 ClusterIP 10.96.41.149 <none> 80/TCP 47h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 47h
deployment.apps/test0 1/1 1 1 47h
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-6484977b56 1 1 1 46h
replicaset.apps/ingress-nginx-controller-659c6c4948 0 0 0 47h
replicaset.apps/ingress-nginx-controller-77d966f98c 0 0 0 47h
replicaset.apps/ingress-nginx-controller-7fd476c957 0 0 0 47h
replicaset.apps/test0-574c47cb97 1 1 1 47h
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 73s 47h
job.batch/ingress-nginx-admission-patch 1/1 86s 47h
$ kubectl describe clusterrole ingress-nginx
Name: ingress-nginx
Labels: app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch]
services [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
nodes [] [] [list watch get]
endpointslices.discovery.k8s.io [] [] [list watch get]
configmaps [] [] [list watch]
endpoints [] [] [list watch]
namespaces [] [] [list watch]
pods [] [] [list watch]
secrets [] [] [list watch]
leases.coordination.k8s.io [] [] [list watch]
ingresses.networking.k8s.io/status [] [] [update]
$ kubectl describe role ingress-nginx
Name: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
events [] [] [create patch]
leases.coordination.k8s.io [] [] [create]
configmaps [] [] [get list watch]
endpoints [] [] [get list watch]
pods [] [] [get list watch]
secrets [] [] [get list watch]
services [] [] [get list watch]
ingressclasses.networking.k8s.io [] [] [get list watch]
ingresses.networking.k8s.io [] [] [get list watch]
leases.coordination.k8s.io [] [ingress-nginx-leader] [get update]
namespaces [] [] [get]
endpointslices.discovery.k8s.io [] [] [list watch get]
ingresses.networking.k8s.io/status [] [] [update]
# kubectl describe clusterrolebindings.rbac.authorization.k8s.io ingress-nginx
Name: ingress-nginx-admission
Labels: app.kubernetes.io/component=admission-webhook
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
Role:
Kind: ClusterRole
Name: ingress-nginx-admission
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount ingress-nginx-admission ingress-nginx
# kubectl describe rolebindings.rbac.authorization.k8s.io ingress-nginx
Name: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: <none>
Role:
Kind: Role
Name: ingress-nginx
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount ingress-nginx ingress-nginx
$ kubectl describe sa
Name: default
Namespace: ingress-nginx
Labels: <none>
Annotations: <none>
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
Name: ingress-nginx
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: <none>
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
Name: ingress-nginx-admission
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=admission-webhook
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.10.0
helm.sh/chart=ingress-nginx-4.10.0
Annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
$ kubectl get all,ing
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-2vf6g 0/1 Completed 0 3d
pod/ingress-nginx-admission-patch-cxtx8 0/1 Completed 2 3d
pod/ingress-nginx-controller-6484977b56-tp7tr 1/1 Running 0 2d23h
pod/test0-574c47cb97-fzhjf 1/1 Running 0 3d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.96.31.140 <none> 80:32691/TCP,443:31098/TCP 3d
service/ingress-nginx-controller-admission ClusterIP 10.96.33.172 <none> 443/TCP 3d
service/test0 ClusterIP 10.96.41.149 <none> 80/TCP 3d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 3d
deployment.apps/test0 1/1 1 1 3d
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-6484977b56 1 1 1 2d23h
replicaset.apps/ingress-nginx-controller-659c6c4948 0 0 0 3d
replicaset.apps/ingress-nginx-controller-77d966f98c 0 0 0 3d
replicaset.apps/ingress-nginx-controller-7fd476c957 0 0 0 3d
replicaset.apps/test0-574c47cb97 1 1 1 3d
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 73s 3d
job.batch/ingress-nginx-admission-patch 1/1 86s 3d
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/ingress-class-name-no-perm nginx * 10.96.31.140 80 2d23h
ingress.networking.k8s.io/ingress-from-annotation <none> * 80 2d23h
ingress.networking.k8s.io/ingress-invalid-ingress-class-name-no-perm nginx-not-match * 80 2d23h
$ kubectl describe ing
Name: ingress-class-name-no-perm
Labels: <none>
Namespace: ingress-nginx
Address: 10.96.31.140
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/demo/http1 test0:80 (10.244.0.8:80)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /
Events: <none>
Name: ingress-from-annotation
Labels: <none>
Namespace: ingress-nginx
Address:
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/demo/http3 test0:80 (10.244.0.8:80)
Annotations: kubernetes.io/ingress.class: nginx-annotation
nginx.ingress.kubernetes.io/rewrite-target: /
Events: <none>
Name: ingress-invalid-ingress-class-name-no-perm
Labels: <none>
Namespace: ingress-nginx
Address:
Ingress Class: nginx-not-match
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/demo/http2 test0:80 (10.244.0.8:80)
Annotations: nginx.ingress.kubernetes.io/rewrite-target: /
Events: <none>
$ kubectl get events
No resources found in ingress-nginx namespace.
# curl test0.local -v
> GET http://test0.local/ HTTP/1.1
> Host: test0.local
> User-Agent: curl/7.76.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Connection: close
<
* Closing connection 0
DNS lookup failed
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.31.140 <none> 80:32691/TCP,443:31098/TCP 3d
ingress-nginx-controller-admission ClusterIP 10.96.33.172 <none> 443/TCP 3d
test0 ClusterIP 10.96.41.149 <none> 80/TCP 3d
# kubectl exec -it ingress-nginx-controller-6484977b56-tp7tr sh
#
/etc/nginx $ curl http://test0.local -v
* Could not resolve host: test0.local
* Closing connection
curl: (6) Could not resolve host: test0.local
/etc/nginx $ curl http://test0 -v
* Host test0:80 was resolved.
* IPv6: (none)
* IPv4: 10.96.41.149
* Trying 10.96.41.149:80...
* Connected to test0 (10.96.41.149) port 80
> GET / HTTP/1.1
> Host: test0
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.25.4
< Date: Thu, 18 Apr 2024 08:23:19 GMT
< Content-Type: text/html
< Content-Length: 615
< Last-Modified: Wed, 14 Feb 2024 16:20:36 GMT
< Connection: keep-alive
< ETag: "65cce854-267"
< Accept-Ranges: bytes
<
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
* Connection #0 to host test0 left intact
|
|
kubectl get rolebinding ingress-nginx -oyamlapiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.10.0
helm.sh/chart: ingress-nginx-4.10.0
name: ingress-nginx
namespace: ingress-nginx
resourceVersion: "646"
uid: 1a2703cc-a10e-46be-94b5-b2eedfe8d4ea
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
It break back-award compatiblity. to use ingressClassName in ingress.spec, cluter permission is mandatory for nginx-ingress-controller. that is why i raise this enhancement. with this enhancement, nginx-ingress-controller can manage ingresses with ingress.spec.ingressClassName in specific namespaces without cluster level permission. |
|
Apologies. I am 100% lost. Wait for other comments. |
|
This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach |
github-actions
bot
added
the
lifecycle/frozen
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
From kubernetes 1.18, kubernetes deprecate ingress annotation "kubernetes.io/ingress.class", instead, it is replaced with ingress.Spec.IngressClass. for cluster Ingress, it is ok. but for namespaced ingress. cluster don't want to grant any cluster resource permission to ingress-controller. it means nginx-ingrss-controller have no permissions to access the IngressClass object. in current nginx-ingress-controller, it mandate the IngressClass existence referred as ingress.Spec.IngressClassName.
As a result, for Namespaced ingress scenario, ingress annnotation "kubernetes.io/ingress.class" is the only choice. it works at this time, but from kubernets 1.28, kubernetes server keep printing warning if ingress has annotation "kubernetes.io/ingress.class". it is not ideal.
What you expected to happen:
so it is better support namespaced ingressClass without accessing the IngresClass object and using the annotation.
suggestions:
NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
Kubernetes version (use
kubectl version): v1.29.2Environment:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g.
uname -a):Install tools:
Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.Basic cluster related info:
kubectl versionkubectl get nodes -o wideHow was the ingress-nginx-controller installed:
helm ls -A | grep -i ingresshelm -n <ingresscontrollernamespace> get values <helmreleasename>/nginx-ingress-controller --kubeconfig=/root/.kube/config
--default-ssl-certificate=core/demo1-nginx-secret
--v=0
--configmap=core/demo1-ingress-controller-conf
--watch-namespace=core
--annotations-prefix=ingress.kubernetes.io
--enable-ssl-chain-completion=false
--http-port=8080
--https-port=8443
--enable-annotation-validation=true
--update-status=false
--ingress-class=demo1-nginx
--metrics-per-host=false
--enable-metrics=false "
Current State of the controller:
kubectl describe ingressclasseskubectl -n <ingresscontrollernamespace> get all -A -o widekubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>Current state of ingress object, if applicable:
kubectl -n <appnamespace> get all,ing -o widekubectl -n <appnamespace> describe ing <ingressname>Others:
kubectl describe ...of any custom configmap(s) created and in useHow to reproduce this issue:
Anything else we need to know:
The text was updated successfully, but these errors were encountered: