[Metrics] Observability requirements for Hosted Che

[l0rd]

Description

There have been some discussions about metrics we would like to be able to observe on hosted Che (e.g. che.openshift.io) based on the issues we had in the past and these are the most relevant ones:

Che Server metrics

• The % of workspaces started successfully
• The % of workspaces stopped successfully
• The % of workspaces started in under N seconds

Workspace metrics

• [MVP] Collecting Che Workspace metrics in single cluster mode with Prometheus [MVP] Collecting Che Workspace metrics in single cluster mode with Prometheus #14888
• Collecting Che Workspace metrics in multiple cluster mode with help of Prometheus Federation mechanism Collecting Che Workspace metrics in multi-cluster mode with help of Prometheus Federation mechanism #14889
• Test Prometheus federation in simplified mode. Test Prometheus federation in simplified mode. #15030
• Investigate alternative possibilities to reduce Prometheus's permissions required to monitor workspaces Investigate alternative possibilities to reduce Prometheus's permissions required to monitor workspaces #15031
• The % of workspaces projects cloned successfully
• The % of workspaces projects cloned under N seconds
• The % of Language Servers requests that get a successful response
• The % of Language Servers requests that get a successful response in under N seconds

cc @skabashnyuk @ibuziuk

[l0rd]

I have added this issue to epic #10329

[ibuziuk]

I believe the first step would still be adding what we already have available on dsaas - redhat-developer/rh-che#1336

[l0rd]

@ibuziuk are you talking about the monitoring infrastructure (prometheus, grafana etc...)? This issue is about implementing the prometheus endpoints.

[ibuziuk]

@l0rd Of course, this is not a blocker for implementing those endpoints upstream, but until we have a proper setup in downstream we will not be able to take full advantage of those metrics for Hosted Che

[mkuznyetsov]

we already have in Grafana "Workspace Detailed" section with heatmaps, which could be used for getting "The % of workspaces started in under N seconds" (if we want to do exactly that, we have metric endpoints specifically for it)

we also have "The % of workspaces started successfully" displayed on Grafana, yet we don't have the "The % of workspaces stopped successfully"

[aditya-konarde]

/cc @skryzhny can you please provide some feedback on the metrics and provide suggestions here?

Beyond the classic USE and RED metrics, we can look at some application specific metrics other than the current ones that either:

• Provide business insights to someone running Eclipse Che (number of workspaces, number of users, number of signups)
• Provide feedback into development (Workspace start time, average runtime of a workspace, workspace aggregate errors)

I believe we're already well covered looking at the dashboard. But may have scope fro some more :)

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.