Ingress crossnamespace should be allowed by default

[deepio]

Is your feature request related to a problem? Please describe.

Yes, for context I've used https://github.com/k3s-io/k3s-ansible to install k3s in my cluster.

If I create an ingress which points to a service in a namespace that is not default, it is not accessible. Why is this the default? If I wanted to create something completely within a namespace nothing happens. To allow different namespaces, it appears we need to turn on AllowCrossNamespaces in traefik? I believe this should be turned on by default.

From what I understand the recommended method of enabling CrossNamespaces in traefik https://doc.traefik.io/traefik/providers/kubernetes-crd/#allowcrossnamespace is by creating a HelmChartConfig with --providers.kubernetescrd.allowCrossNamespace=true as a global argument. When I try to enable CrossNamespaces, however, all my ingresses become suddenly unavailable (even the default ones). I'm not sure how to recover from this without nuking the cluster and starting over but that's not a big deal because I have backups.

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    globalArguments:
    - "--providers.kubernetescrd.allowCrossNamespace=true"

Describe the solution you'd like

I have zero preferences for ingresses with traefik vs nginx or another one. The default should just be that we can deploy any ingress with a namespace that is not "default" and it's viewable from outside the cluster. This way, when I install longhorn I can see the UI too without having to mess with the ingress. https://rancher.com/docs/k3s/latest/en/storage/ because as it is now I can't have k3s with the ansible playbook AND deploy longhorn without changing all the namespaces to default.

[brandond]

If I remember correctly, the cross-namespace restriction just requires that the ingress and service it passes traffic to are in the same namespace, as a security measure. If you want to use a single Ingress for services in multiple namespaces then yes, you would need to allow cross-namespace ingress and ensure that your ingress config points at the correct namespace for each service. Can you provide an example configuration that's not working for you?

[deepio]

Sure, I tried just following the docs:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.2.3/deploy/longhorn.yaml

then

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: longhorn-ingress
  namespace: default 
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required '
    nginx.ingress.kubernetes.io/proxy-body-size: 10000m
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: longhorn.<snip>.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: longhorn-frontend
            port:
              number: 80

Am I supposed to use an externalName instead to get across the namespace issue? That would suggest we should update the longhorn documentation instead.

[brandond]

Is the longhorn-frontend service in the default namespace, or in the longhorn-system namespace? The Ingress should go into the same namespace as the service.

The Longhorn docs are probably designed for RKE, RKE2, kubeadm, or other clusters that ship with ingress-nginx, not Traefik. Any functionality configured by the nginx annotations (auth middleware, tls config, etc) may need to be adapted to work with Traefik or any other ingress controller.

[brandond]

Did that work?

[deepio]

It did not for me but this seems more like a traefik and longhorn problem and less of a k3s problem so I've shared my concern with them and will be trying the nginx ingress. My priority is just to have it accessible and then I can figure out why it doesn't work in some situations and why it does in others. Then I can release a guide or blog somewhere on how to configure it together.

[ilanKushnir]

did you manage to set allowCrossNamespace to true ?