chore: remove compatibility for legacy networking

[jawnsy]

I've tagged a bunch of people for review because this is a pretty big change. I have not tested this yet, but it does at least pass the kind integration and very limited unit tests we have... Recommendations for a test strategy would be most welcome, I was just going to give it a smoke test on dev-4.

[shortcut-integration]

This pull request has been linked to Shortcut Story #18924: Remove compatibility code for old ingress controller.

[deansheather]

This looks ok but the removal of nginx-ingress load balancer is worrying. Also the inconsistent indenting/newlines around toYaml should be fixed

[mterhar]

Looks great. I think this deserves some sort of new version or name or something when we release it.

[kylecarbs]

Why is the removal of nginx-ingress worrying @deansheather?

[deansheather]

@kylecarbs the old load balancer service is called ingress-nginx, so we can't change the name of it for existing deployments or load balancer IPs will change

[kylecarbs]

Oh I see. Yea, that is sketchy.

[coadler]

I like the use of nindent. Makes it much more readable

[mterhar]

kylecarbs the old load balancer service is called ingress-nginx, so we can't change the name of it for existing deployments or load balancer IPs will change

Can we remove the ingress-nginx loadbalancer from control of helm kinda like we did for the workspace provider service account? I think we can abandon that object since the future paradigm is that they manage their own LB.

[jawnsy]

I tested this manually in our dev cluster, and since the release has been cut, I think this is safe to merge, so that we get more testing done in preparation for the next release.