Parametrize IngressClass name

[gbonnefille]

Description

On a K3S kubernetes, the default ingress controller is traefik. It uses a "traefik" IngressClass.

Currently, the Ingress generated by devworkspace-controler is hardcoded to "nginx"

devworkspace-operator/controllers/controller/devworkspacerouting/solvers/basic_solver.go

Line 34 in 7e8aa72

"kubernetes.io/ingress.class": "nginx",

Having a parameter to control this name would help to reuse in different K8S.

[gbonnefille]

As a workaround, I created a nginx IngressClass to reroute to traefik

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
spec:
  controller: traefik.io/ingress-controller

Traefik is now interested in the generated Ingresses but:

time="2023-03-16T16:32:11Z" level=error msg="Error while updating ingress status: failed to update ingress status default/workspace31cb5bbeca574a4d-che-code: Operation cannot be fulfilled on ingresses.networking
.k8s.io \"workspace31cb5bbeca574a4d-che-code\": the object has been modified; please apply your changes to the latest version and try again" namespace=default providerName=kubernetes ingress=workspace31cb5bbeca5
74a4d-che-code                                                                                                                                                                                                     

I suspect a race condition around Status: traefik try to update and devworkspace-controller reconcille the Ingress.

devworkspace-controller {"level":"info","ts":"2023-03-16T16:32:38Z","logger":"controllers.DevWorkspaceRouting","msg":"Ingresses not in sync","Request.Namespace":"default","Request.Name":"routing-workspace31cb5bb
eca574a4d","devworkspace_id":"workspace31cb5bbeca574a4d"}                                                                                                                                                          
devworkspace-controller {"level":"info","ts":"2023-03-16T16:32:38Z","logger":"controllers.DevWorkspaceRouting","msg":"Reconciling DevWorkspaceRouting","Request.Namespace":"default","Request.Name":"routing-worksp
ace31cb5bbeca574a4d","devworkspace_id":"workspace31cb5bbeca574a4d"}                                                                                                                                                
devworkspace-controller {"level":"info","ts":"2023-03-16T16:32:38Z","logger":"controllers.DevWorkspaceRouting","msg":"Reconciling DevWorkspaceRouting","Request.Namespace":"default","Request.Name":"routing-worksp
ace31cb5bbeca574a4d","devworkspace_id":"workspace31cb5bbeca574a4d"}                                                                                                                                                

[gbonnefille]

Humh... the Status is supposed to be ignored

devworkspace-operator/pkg/provision/sync/diffopts.go

Line 91 in 7e8aa72

cmpopts.IgnoreFields(networkingv1.Ingress{}, "TypeMeta", "ObjectMeta", "Status"),

[gbonnefille]

I did some tries. I suspect something also introduce the field metadata.annotations.field.cattle.io/publicEndpoints. I don't know if this is the issue...
But it is supposed to be excluded as it is part of ObjectMeta.

[gbonnefille]

@amisevsk is it possible to disable adding the nginx related annotations by default?

Or, having a config routing.annotations with the following behavior:

• if not set, use a actual code to automatically set some defaults
• if set, do not set the default and only apply the specified ones.

[gbonnefille]

Related to #1143