K8s IngressClass Blocks Ingress Controllers Deployed on Namespace Scope

[bschoenbach]

Hi k8s community,

After upgrading of k8s cluster to v1.18+ we get the following error during startup of several ingress-controllers (traefik v2.3+, nginx v1.9.0+)

Traefik v2.3+:
E0107 06:45:40.895430 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.2/tools/cache/reflector.go:125: Failed to list *v1beta1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:traefik:traefik" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

NGINX v1.9.0+:
F0305 07:02:12.462868 1 main.go:300] Error when getting IngressClass nginx: ingressclasses.networking.k8s.io "nginx" is forbidden: User "system:serviceaccount:nginx:nginx" cannot get resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

The problem:
Within k8s v1.18+ the "IngressClass" was introduced to avoid bad annotations on "Ingress" resource objects themselves (see here) . However, the idea of that concept is pretty cool but lacks in implementation because it requires access on cluster scope level. Hence, developers or teams which can only work on namespace scope cannot use ingress controllers on k8s > 1.18.x.

Possible solutions could be:
Provide access to "ingressclasses" on k8s v1.18+ on namespace scope, too. In my opinion it must be possible to define (i.e. read/write) ingressclasses also on namespace scope.

Temporary Solution:
Currently we downgraded our ingress-controllers to lower versions where listing of "ingressclasses" is not implemented and uses the old annotation scheme within ingress specs. We also raised issues for the ingress-controller suppliers. Nothing happened so far. Shameful.

• Ingressclasses not available on namespace scope level traefik/traefik#7729

[neolit123]

/sig network

[maplain]

similar discussions in Service API:

1. Add support for selecting a specific in-cluster proxy instance as implementer of Gateway kubernetes-sigs/gateway-api#491
2. Support ClusterGatewayClass and GatewayClass kubernetes-sigs/gateway-api#136

As GatewayClass is also a Cluster scoped resource, like IngressClass.

[robscott]

Thanks to @SantoDE and @maplain for linking this issue to the corresponding Gateway API discussions. I'm quite hesitant to add a new namespace-scoped {Ingress/Gateway}Class resource to these APIs but I do think we need to provide a better path forward for namespace-scoped Ingress/Gateway implementations.

One option that's been mentioned for Gateway API would be to make GatewayClass an optional resource and allow the gatewayClassName field on Gateway to represent a name that was already meaningful to the controller. I think a similar approach could potentially work for Ingress as well.

We're hard at work trying to make this and many other concepts better with the Gateway API. We'd love to have more participants in our efforts to improve these APIs. Of course we're also pulling back whatever we can from the Gateway API to Ingress, such as the recent addition of namespace-scoped parameters references for Ingress.

I think it will likely be easier to continue this discussion on kubernetes-sigs/gateway-api#567 since this is so closely related. Once we reach a conclusion there I'm hoping it will be equally helpful for both APIs. So if you're interested in improving these APIs, please join the discussions we're having there.

I'm assigning this issue to myself, but that doesn't mean I can solve it on my own. Continued discussion and likely eventually a KEP will be necessary to solve this.

/assign
/kind feature

[robscott]

Accepting this for triage. I don't think we have a clear solution yet, but I think this merits further discussion.

/triage accepted

[robscott]

Follow up - this has transitioned to a PR on Gateway API: kubernetes-sigs/gateway-api#614.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[bschoenbach]

/remove-lifecycle stale

[thockin]

@robscott Updates?

[robscott]

@thockin Unfortunately this lost steam. We explored some options for Gateway API but unfortunately none of them really panned out as well as we'd hoped. Unfortunately I'm not sure how much time I'll have to push this forward in the next few months.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[robscott]

I don't think I have the bandwidth to fix this soon, if anyone has time, contributions are very welcome.

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[wangrzneu]

/remove-lifecycle rotten

let me have a try

[wangrzneu]

/assign

[wangrzneu]

I think a namespace scope ingress class resource should be added. The new resource can be named NamespaceIngressClass, and the NamespaceIngressClass can be referenced in Ingress like,

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
spec:
  ingressClassName: external-lb
  ingressClassNamespace: infra

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[robscott]

Some more recent discussion on kubernetes-sigs/gateway-api#567

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.