Docker install is unable to view users

[Josh-Diamond]

Rancher Server Setup

• Rancher version: v2.6-b26b920420e47e99d5c96356b96c393360dd5739-head
• Installation option (Docker install/Helm Chart): Docker
  • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):
• Proxy/Cert Details: self-generated

Information about the Cluster

• Kubernetes version: 1.23.3+k3s1
• Cluster Type (Local/Downstream): Local
  • If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):
    N/A

User Information

• What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
  Admin

Describe the bug
When selecting the hamburger menu from the rancher home page (upper-left hand corner) and then selecting "Users & Authentication", right away you'll notice no users are listed, not even the admin, which exists. Upon clicking "create" to attempt to create a new user, the form seems incomplete in the UI and the "create" button in bottom right, is disabled, even when all required fields are filled in. (my initial thoughts are that the UI "create" form for "Users & Auth" is not displaying the entire form, so if that is correct, there might be no way to actually fill in all the required fields to allow the "create" button to become enabled, assuming that the button is only enabled when ALL required fields are filled, and in this instance the UI form itself may not be displaying all input field options. Further investigating, I noticed that if you log out, the UI will get caught in a "hang" state of "Loading...", but with a hard refresh, you are able to log back in and the bug seems to be resolved. I was also able to create a new user through the api with Postman. When attempting to log out of rancher, the UI seemed to "hang" on the "Logging out..." screen. With a hard refresh, I was able to log back in as the new user, created through Postman, and then I was able to see ALL users (new user + admin) as well as having the correct UI display on the "create" user form page)

To Reproduce

1. Fresh install rancher with Docker, using v2.6-b26b920420e47e99d5c96356b96c393360dd5739-head
2. Once logged in, as admin, from the home page, select the hamburger menu found in the upper-left corner
3. You should now be able to view "all" users on the server, however, even admin (currently logged in as) is not listed
4. To continue, the issue also persists when attempting to "create" a new user, if "create" button is selected, it seems an incomplete form page is shown in UI, also no matter what data the user inputs, the "create" button is never enabled

Result

Expected Result

Additional Comments
Successful creation of a new user through api with Postman:

Logs:

2022/03/08 19:39:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-9wg8x': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-9wg8x), requeuing
2022/03/08 19:39:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-9wg8x': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-9wg8x), requeuing
2022/03/08 19:39:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-9wg8x': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-9wg8x), requeuing
2022/03/08 19:39:52 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-local-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/03/08 19:39:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-9wg8x': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-9wg8x), requeuing
2022/03/08 19:39:52 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-m2vwh to namespace=cattle-fleet-local-system
2022/03/08 19:39:52 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/03/08 19:39:52 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-local-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/03/08 19:39:52 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-m2vwh to namespace=cattle-fleet-local-system
2022/03/08 19:39:52 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2022/03/08 19:39:52 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-m2vwh to namespace=cattle-fleet-local-system
2022/03/08 19:39:52 [ERROR] error syncing 'cattle-fleet-system/helm-operation-9wg8x': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-9wg8x), requeuing
2022/03/08 19:40:06 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-03-08 19:39:11 +0000 UTC notAfter=2023-03-08 19:40:06 +0000 UTC
2022/03/08 19:40:06 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-03-08 19:39:11 +0000 UTC notAfter=2023-03-08 19:40:06 +0000 UTC
2022/03/08 19:40:06 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-03-08 19:39:11 +0000 UTC notAfter=2023-03-08 19:40:06 +0000 UTC
2022/03/08 19:40:06 [INFO] Updating TLS secret for serving-cert (count: 6): map[field.cattle.io/projectId:local:p-m2vwh listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-IP_ADDRESS_REDACTED:IP_ADDRESS_REDACTED listener.cattle.io/cn-ANOTHER_IP_ADDRESS_REDACTED:ANOTHER_IP_ADDRESS_REDACTED listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=HIDDEN_VALUE]
2022/03/08 19:40:06 [INFO] Active TLS secret serving-cert (ver=3265) (count 6): map[field.cattle.io/projectId:local:p-m2vwh listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-IP_ADDRESS_REDACTED:IP_ADDRESS_REDACTED listener.cattle.io/cn-ANOTHER_IP_ADDRESS_REDACTED:ANOTHER_IP_ADDRESS_REDACTED listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=HIDDEN_VALUE]
2022/03/08 19:40:07 [ERROR] Error during subscribe write tcp IP_ADDRESS_REDACTED:443->ALT_IP_ADDRESS_REDACTED:59558: use of closed network connection
2022/03/08 19:40:09 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:40:10 [ERROR] error syncing 'cattle-fleet-system/helm-operation-c8ndm': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-c8ndm), requeuing
2022/03/08 19:42:10 [INFO] starting duplicate binding cleanup
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] starting bindings cleanup
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] total PRTB duplicate clusterRoleBindings 0, roleBindings 0
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] binding with deterministic name not found, will delete all except the oldest binding
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] binding with deterministic name not found, will delete all except the oldest binding
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] binding with deterministic name not found, will delete all except the oldest binding
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] binding with deterministic name not found, will delete all except the oldest binding
2022/03/08 19:42:10 [INFO] [clean-dupe-bindings] total CRTB duplicate clusterRoleBindings 0, roleBindings 0
2022/03/08 19:42:10 [INFO] successfully cleaned up duplicate bindings
2022/03/08 21:39:10 [INFO] Updating global catalog system-library
2022/03/08 21:39:10 [INFO] Catalog sync done. 0 templates created, 1 templates updated, 0 templates deleted, 0 templates failed
2022/03/08 22:38:01 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-9lmpq
2022/03/08 22:38:01 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-9lmpq for user u-gzcr4 with role cattle-globalrole-user
2022/03/08 23:22:51 [ERROR] Error during subscribe write tcp IP_ADDRESS_REDACTED:443->ALT_IP_ADDRESS_REDACTED:62661: write: broken pipe

[gaktive]

Possible internal reference: SURE-4255. Re-reading just to make sure.

[gaktive]

This is also popping up on Keycloak. Internal reference: SURE-4195

[catherineluse]

(my initial thoughts are that the UI "create" form for "Users & Auth" is not displaying the entire form, so if that is correct, there might be no way to actually fill in all the required fields

You're correct - in your second screenshot, the Global Permissions section is missing. At least one checkbox in Global Permissions must be selected in order for the Create button to be enabled.

For me, on a fresh Docker install, when I log in for the first time, I get a fail whale. Then if I go to the Users & Authentication section, it's a blank page:

And the console is filled with certificate errors.

However, after refreshing the page, everything works as expected.

Possibly related issues:

• Dashboard launch is flaky it sometimes fails to load and redirected to /dashboard/fail-whale dashboard#393
• Fail Whale if end user visits dashboard before Rancher is "ready" dashboard#3750 (this is the behavior I'm seeing)

[catherineluse]

@gaktive This issue is less critical than I thought at first glance because it does have a workaround - it works if you refresh the page. So I think it can be bumped to v2.6.5.

[gaktive]

@jtravee something to release note for 2.6.4. Per Catherine's comment, a refresh of the page is a workaround.

[jtravee]

@jtravee something to release note for 2.6.4. Per Catherine's comment, a refresh of the page is a workaround.

Thank you, added!

[samjustus]

SURE-4255

[samjustus]

will reevaluate after #39094 as it might fix this

[MKlimuszka]

According to the reporter, this has been remedied as of Rancher 2.6.6.