[Stack Monitoring] Missing remote_cluster_client role is reported as "access denied"

[matschaffer]

This was originally reported as #125756 to which #120384 will provide a workaround, but the error case is still quite confusing.

When attempting to load Stack Monitoring UI with monitoring.ui.ccs.enabled: true (the current default and proposed ongoing default in #120384) but missing the remote_cluster_client role, the user will get this error message:

Which says nothing about what needs to be done to resolve the error (either disable css or add the remote_cluster_client node role).

We should update the error handling in our access check to make the error clearer for both the end user, and ourselves fielding support or forum issues like these:

• https://discuss.elastic.co/t/kibana-upgrade-to-8-x-cant-enter-monitoring/299987
• https://discuss.elastic.co/t/cannot-access-stack-monitoring/312589/2

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[liza-mae]

I just came across this same issue in upgrade testing, will try remedy in discuss ticket.

[liza-mae]

I tried a user with kibana_admin and monitoring_user roles and it did not work, the error still is present.

[matschaffer]

@liza-mae check if the ES nodes have remote_cluster_client. This error sounds like user adjustment is required, but usually I see it happen because the ES node roles are unable to support CCS.

[bar0n36]

@matschaffer - I have replicated this issue, but even with monitoring.ui.ccs.enabled: false, I was unable to access Stack Monitoring without the remote_cluster_client role assigned to my cluster nodes (2 x nodes)

This is a new build of 8.2.1.

[matschaffer]

Thanks @bar0n36

Can you provide more details about your test environment? ESTF or ESS? Just regular downloads to a laptop? cluster roles, possibly an ES diagnostic, etc