Trying to edit/update the statefulset through the UI is not working

[gaktive]

Internal reference: SURE-7109
Reported in 2.7.5, confirmed also in 2.8.0

Repro steps:

step 1: on Rancher 2.7.5 install monitoring from the apps marketplace in downstream RKE1 cluster
step 2: go to workloads > statefulsets > "prometheus-rancher-monitoring-prometheus" > edit config
step 3: on prometheus tab, go to resources and changed CPU reservation from 750 mCPUs to 700mCPUs. Select save
step 4: observe that the statefulset was redeploying but the CPU reservation stayed at 750
step 5: kubectl edit <statfulset> change the prometheus CPU res from 750 to 700 and the change took and reflected in both the Rancher UI and in the cluster itself

Issue description:
A user is trying to update the CPU/Mem resources limits for one of the stateful sets (monitoring installed from apps marketplace) from the rancher UI and it's being reset again back to the old limits for some reason. When the resources are updated from the CLI using the kubectl edit command, it gets updated and then Rancher UI shows the update.

Support was able to duplicate this in their lab.

Business impact:
Editing the config from the UI should, in fact, push change down to the cluster. The user shouldn't have to do it from the kubectl CLI

Troubleshooting steps:

Changed Rancher logs to debug
reproduced the issue
no clues....

Repro steps:

step 1: on Rancher 2.7.5 install monitoring from the apps marketplace in downstream RKE1 cluster
step 2: go to workloads > statefulsets > "prometheus-rancher-monitoring-prometheus" > edit config
step 3: on prometheus tab, go to resources and changed CPU reservation from 750 mCPUs to 700mCPUs. Select save
step 4: observe that the statefulset was redeploying but the CPU reservation stayed at 750
step 5: kubectl edit <statfulset> change the prometheus CPU res from 750 to 700 and the change took and reflected in both the Rancher UI and in the cluster itself

Workaround:
Use the kubectl CLI on the cluster to edit the resource. Not ideal but it is a workaround

Actual behavior:
Editing the config in Rancher and it doesn't actually change the statefulset

Expected behavior:
An edit from Rancher should change the value in the statefulset

Addenda:

• @nwmac confirmed that this still happens in 2.8.0
• Belief is that you should be able to edit these, but the caveat is if you then redeployed the app via Helm or updated it via Helm, those changes would get overwritten

[nwmac]

Doesn't seem to affect all Stateful Sets - we use PUT to apply changes, where as kubectl uses PATCH - we should switch to do the same - note this requires a backend change as per rancher/rancher#40712

[gaktive]

Let's sync with backend first to check if PUT or PATCH is the expected mechanism here.

[gaktive]

Perhaps related code that touches near where this is: #9577 though this problem still happens in Chrome.

[gaktive]

Similar efforts around permissions: #7897

[aalves08]

After a bit of investigation, there seems to be some problem with the updates we are getting via websockets...

Here is a video where we do the changes on the UI, then I show each relevant part of the Networks tab:

Screen.Recording.2023-12-18.at.16.10.48.mov

Recreating the repro steps, when we successfully edit this prometheus-rancher-monitoring-prometheus statefulset, we do a PUT to /v1/apps.statefulsets/cattle-monitoring-system/prometheus-rancher-monitoring-prometheus, which has been successful and in both the payload and the response we see the corresponding changes.

What happens is that the resource get's overwritten on the UI side by the resource.change coming via websockets (socket k8s/clusters/local/v1/subscribe?sockId=3), where we can clearly see that the resource data comes with the old data it had on the resource cpu and memory reservation.

A full page refresh returns the old object (without the changes employed by the PUT method).

@richard-cox @nwmac Is this just a PUT vs PATCH thing? 🤔

[aalves08]

Moved to backend blocked with creation of rancher/rancher#43916

[samjustus]

@aalves08 so this is blocked by both 43916 and #40712 ?

[aalves08]

@samjustus at least rancher/rancher#43916 for sure. Not entirely sure about rancher/rancher#40712, but I trust Neil's judgement.

I would have to repro it all again and check the implications.

[samjustus]

@aalves08 don't worry about that - we will take a look at them

[aalves08]

Thank you @samjustus