[2.6] Automatic creation/sync of services <-> workload ports permanently lost after 2.5.8->2.6.0

[GiGurra]

Rancher Server Setup

• Rancher version: 2.6.0
• Installation option (Docker install/Helm Chart): docker standalone/single instance
• Proxy/Cert Details: letsencrypt

Information about the Cluster

• Kubernetes version: 1.20.8
• Cluster Type (Local/Downstream): imported digitalocean managed kubernetes

Describe the bug
after upgrading from 2.5.8 -> 2.6.0, services are no longer automatically created/removed when configuring ports in the new "edit deployment config" gui. All existing services were seemingly preserved in the upgrade, but all corresponding deployments ended up in "do not create service" in the workload config editor. I can no longer edit the ports in any workloads, if I try to set them to "cluster IP" i get errors saying "service already exists". If I manually attempt to delete the service objects, they are instantly automatically recreated. If I add new ports to the workload config, these changes are not automatically added to the service objects.

To Reproduce
just guessing:

1. create a 2.5.8 cluster
2. create deployment with cluster ip ports
3. create ingress to it
4. upgrade to 2.6.0
5. Check if port config under deployment ended up in "do not create service"
6. try to change to "create service" or add any new ports. None of these settings have any affect on service objects (automatic service <-> workload sync is permanently lost)

manually Deleting all ports, then the service object, and then re-adding ports to the deployment config doesnt help. (actually it creates a new service, even though the webpage says "service already exists", but then it ends up in the same de-synced state as explained above where you cant edit anything on the workload)

Result
automatic service <-> workload sync is permanently lost. I can delete all ports on workload and services are still kept. This is a security issues in some sense. Can no longer reliably create port mappings in deployment config gui, so starting from 2.6.0. I must manually create services separately.

Expected Result
Same functionality as in previous rancher < 2.6, that is, you can edit ports on workloads and services are edited automatically.

Other info
Have been using rancher since 1.x days and just about every single 2.x version. Never seen this issue before, and it is one of the worst issues in terms of usability Ive had so far. I would not dare upgrading 2.5.x production clusters to 2.6.0 risking hundreds of deployments to lose sync with service configurations/services maintained by rancher.

[GiGurra]

I can verify that if I create entirely new deployments AFTER the upgrade from 2.5.8->2.6.0, service<->deployment port config is synchronized and automatic (both adding and removing ports work fine), but for all deployments that existed from prior to the upgrade, sync is permanently lost

[GiGurra]

Well this is bad... I cannot create services for the workloads even manually (if svc name same as workload name). They are auto reverted to the configuration that existed before the rancher upgrade. Even if I delete all ports from workloads and then delete/edit the services, the services are reverted back to their pre 2.6 config (after a few minutes)

[GiGurra]

This is even worse.. even if I delete the entire workload and recreate it again, the issue persists.
It seems like rancher remembers the workload name, and all workload names ever used before are broken.

[anupama2501]

Reproduced this on v2.5.8 to v2.6.0 rancher server.

• Created a downstream rke1 cluster on v2.5.8
• Create a few wokrloads with port ClusterIP, Nodeport, Hostport
• Created ingress for each of the workload
• Upgraded from v2.5.8 to v2.6.0 rancher version.
• Workloads have ports Do not create a service status

- If we edit the config by removing the previous ports and add a new port with service type `ClusterIP` , the config fails with the error:

type: "error", links: {}, code: "AlreadyExists", message: "services "nodeport1" already exists",…}
code: "AlreadyExists"
links: {}
message: "services \"nodeport1\" already exists"
status: 409

This is even worse.. even if I delete the entire workload and recreate it again, the issue persists.
It seems like rancher remembers the workload name, and all workload names ever used before are broken.

This is happening as the Service discovery entry for workload is present in the Services tab. If we delete the SD entry and re-create the workload, able to create the deployment.

Issues seen:

• Services should be deleted when the corresponding workload is deleted
• Workload edit config should let add new ports [if workload is created with a service type port eg cluster IP on v2.6.0 and when edited if we add new ports with the same service type, this is not erroring out].

[cbron]

Waiting on that UI issue ^, once that is merged this can go to test. At this point there is no dev work to be done.

[deniseschannon]

Since both issues linked here have been labeled to test in the UI, can we please test to see if this is still an issue @kinarashah before we verify and move it over to QA to finalize testing on this use case?

[kinarashah]

We should release-note this for users upgrading to 2.6.1, all legacy ui services will need to be deleted manually when workloads are deleted rancher/dashboard#4212 (comment)

On upgrade, if workloads created from legacy UI are deleted, the corresponding services won't be deleted. Users will have to manually clean them up. UI now shows a text msg (screenshot here rancher/dashboard#4212 (comment)) to inform users when they're deleting such workloads.

[kinarashah]

Validation update:

This issue needs to be tested for 3 issues mentioned in rancher/dashboard#4159, out of which first one is failing for the following scenario:

• Create workload in old ui without passing any ports
• In new UI, try to edit workload and add port
• Edit/save is blocked with an error msg: service already exists

[gaktive]

Per rancher/dashboard#4159 (comment), UI is running into a backend issue tied to some Norman vs. Steve differences. Not sure if we need to spawn a new ticket to look into that but for UI, there is a workaround to allow editing at least through the old UI.

[kinarashah]

Fresh Install v2.6.1-rc7:

• Fresh Install v2.6.1-rc7 single node docker Install
• Create workload in Ember with clusterIP, service for workload gets created automatically
• Edit Config the workload in Vue, the ports show up correctly as clusterIP
• Add a new port clusterIP/NodePort, no error shows up and Edit goes through successfully
• Delete the workload, Warning shows up in UI saying this is a legacy workload and service needs to be deleted manually

Upgrade v2.5.8 to v2.6.1-rc7:

• Fresh Install v2.5.8 single node docker Install
• Create workload with clusterIP port, service for workload gets created automatically
• Upgrade to v2.6.1-rc7
• Edit Config the workload, the ports show up correctly as clusterIP
• Add a new port clusterIP/NodePort, no error shows up and Edit goes through successfully
• Delete the workload, Warning shows up in UI saying this is a legacy workload and service needs to be deleted manually

Upgrade v2.5.8 to v2.6.0 to v2.6.1-rc7:

• Fresh Install v2.5.8 single node docker Install
• Create workload with clusterIP port, service for workload gets created automatically
• Upgrade to v2.6.0
• Edit Config the workload, the service shows up as Do not create service
• Try to add clusterIP port, save gives an error mentioned in the issue, service already exists
• Try to save multiple times, gives same error.
• Upgrade to v2.6.1-rc7
• Edit Config the workload, the ports show up correctly as clusterIP
• Add a new port clusterIP/NodePort, no error shows up and Edit goes through successfully
• Delete the workload, Warning shows up in UI saying this is a legacy workload and service needs to be deleted manually

[kinarashah]

@sowmyav27 ^ This issue has been validated, ok to close.