RBAC project role (caching?) issue in Rancher v2.5 Cluster Explorer

[axeal]

What kind of request is this (question/bug/enhancement/feature request):

Bug

Steps to reproduce (least amount of steps as possible):

1. Provision a Rancher v2.5.5 instance and custom cluster with three nodes
2. Enabling monitoring v2 for the custom cluster in the Cluster Explorer UI
3. Create a user foo
4. Create a custom project role monitoring, granting get/list/watch on replicationcontrollers, statefulsets, endpoints, cronjobs, daemonsets, jobs, replicasets, podmonitors and services/proxy. (All of the permissions required for the monitoring view in the Cluster Explorer UI, except deployments).
5. Grant the monitoring project role to the user foo on the System project of the custom cluster
6. Log in as user foo and open the Monitoring view for the custom cluster in the Cluster Explorer UI
7. Observe links are not clickable and an error of the following format in the javascript console:

getters.js:222 Uncaught (in promise) Error: Unknown schema for type: apps.deployment
    at Object.<anonymous> (getters.js:222)
    at actions.js:137
    at d (runtime.js:45)
    at Generator._invoke (runtime.js:274)
    at Generator.O.forEach.t.<computed> [as next] (runtime.js:97)
    at r (asyncToGenerator.js:3)
    at l (asyncToGenerator.js:25)
    at asyncToGenerator.js:32
    at new Promise (<anonymous>)
    at asyncToGenerator.js:21

8. Observe user foo receives a permission error performing kubectl -n kube-system get deployments
9. Add get/list/watch for deployments to the monitoring project role
10. Observe user foo can successfully perform kubectl -n kube-system get deployments
11. Observe links are not clickable and the same error is shown in the javascript console when user foo attempts to view the Monitoring view for the custom cluster.
12. Remove the monitoring role grant on the System project for user and foo and then re-add the grant.
13. Observe Monitoring view for user foo now behaves as expected, links are clickable and javascript console error is absent

Environment information

• Rancher v2.5.5

[gaktive]

Initial read points to a schema error. Will need to repro here to isolate if it's UI and/or Server.

[codyrancher]

@Jono-SUSE-Rancher @gaktive

This appears to be a backend issue. After step 9 I looked to see if the schema exists and the backend is reporting that it doesn't.

After step 12 I looked to see if the schema exists and the backend is reporting that it does.

These screenshots were captured during the same session in the same tab just refreshing without caching.

A quick note to whoever picks this up: I was able to reproduce with an rke cluster instead of a custom cluster in case that makes things easier for you.

[gaktive]

@Jono-SUSE-Rancher should we transfer this over to rancher/rancher to have someone look on the backend?

[jiaqiluo]

The bug is reproduced on v2.5-head 7426ba8

Following the steps from the issue and reproduce the bug on a node-driver cluser.

[kravciak]

Currently blocked on rancher/dashboard#2926 (can't create new project role)

[richard-cox]

Currently blocked on rancher/dashboard#2926 (can't create new project role)

I'm currently investigating this, you should still be able to create project roles via the old ui though.

[aiyengar2]

@kravciak @richard-cox I investigated this issue and figured out the root cause. The problem is that the /v1/schema isn't being updated on PRTB, CRTB, or RoleTemplate updates afaik, so the backend still omits deployments from the schema returned for the user. This was confirmed by @codyrancher's comments in #31982 (comment) as well.

The fix for this will involve triggering a refresh of the data that is displayed on /v1/schema on changes to Rancher RBAC.

[kravciak]

I'm currently investigating this, you should still be able to create project roles via the old ui though.

I was able to create roles this way, thanks.
But now I have problem with monitoring deployment, blocking issue: #32652

[anupama2501]

@JacieChao I tried the following tests but was not able to reproduce the issue on rancher v2.5.10.

• Rancher installed on a docker container
• Custom cluster rke1 with 1 node[AWS] all roles.
• Create a local standard user
• From security --> roles --> projects --> create a custom role with resource deployments and verbs get/list/watch -- role1
• Assign user1 to the default project in the custom cluster with the role1
• Login as user1 in a different browser and verify user can list the deployments
• As an admin, edit role1 and add resource config maps with verbs get/list/watch
• Verify user1 can list all config maps
• Delete the resource configmaps and veirfy user1 cannot list config maps anymore
• Create a new project role with resources daemonsets -- role2
• Create a new user and assign the user to the default project in the custom cluster with role2
• Login as user2 and verify user can list all the daemon sets in the default project
• Edit role2 as an admin and add the resource ingresses with verbs get/list/watch and remove daemon sets
• Verify user2 can list all the ingresses and cannot watch daemonsets anymore.

[ryansann]

@anupama2501 it seems @JacieChao is seeing this on Rancher with an HA local cluster, i.e. it sounds like the access control cache could be inconsistent on some nodes. Can you try a Rancher HA install instead of docker install?

I can also try this repro if needed

[anupama2501]

I was able to reproduce on HA with rke1 as local cluster. Re-opening the issue.

• node driver cluster rke1 with 1 node[DO] all roles.
• Create a local standard user
• From security --> roles --> projects --> create a custom role with resource configmaps and verbs get/list/watch -- role1
• Assign user1 to the default project in the custom cluster with the role1
• Login as user1 in a different browser and verify user can list the configmaps
• As an admin, edit role1 and add deployments as resources verbs get/list/watch and remove config maps
• refresh the UI as user1 and the user can still list the config maps.

[ryansann]

Thanks @anupama2501

I'll take a look at this to see if I can identify what's going on in the HA case

[anupama2501]

Verified on v2.5-head f5cf749 HA installation

• Create a local standard user
• Created a custom project role with verbs get/list/watch with configmaps as resource - role1
• Assigned user1 to the default project in the custom cluster with the role1
• Login as user1 in a different browser and verify user can list the configmaps
• As an admin, edit role1 and add deployments as resources verbs get/list/watch and remove config maps
• refresh the UI as user1 and the user can still list the config maps.

[anupama2501]

Verified by enabling openLDAP auth

• Created a custom project role with verbs get/list/watch with deployments as resource - role1
• Assigned user1 to the default project in the custom cluster with the role1
• Login as user1 in a different browser and verify user can list the deployments
• As an admin, edit role1 and add config maps as resources verbs get/list/watch and remove deployments
• refresh the UI as user1 and the user can still list the config maps.
  repeated the same steps by adding a group as user and logging in as user from that group.

[ryansann]

Release note:

Fixed access control caching issues that caused certain resources not to display properly in UI #31982 #32045