Workspace diagnosis capabilities in Eclipse Che

[skabashnyuk]

Is your task related to a problem? Please describe.

A workspace is a set of microservices and there is a lot of complexity hidden from the end-user.
When a workspace does not behave as expected, it is very hard to understand what is going on, where to get the information, where to get the logs. It makes the diagnosis of bugs very hard

Describe the solution you'd like

Workspace startup

• All logs of the workspace startup should be persisted and available to the end user.
  • All the containers logs of the workspace should be available.
  • The plugin broker logs
  • The JWT proxy logs
  • Che server (when reading the devfile)
  • The language servers, the editor and any plugins logs (note that those logs are generally not available as container logs)
• Logs should be downloadable, so they can be exported in an issue and shared.
• We would keep history of the latest 5 startup of the workspace
• Logs should be available from chectl (to start)

Running workspace

• We should have a widget displaying the status of the workspace. This could be an option of the “workspace panel”.
  • We should display the resource usage in each of the containers
  • We should allow the user to access to the logs of a container (links to command on getting the logs, or links to the openshift console)
• Chectl should allow to export the logs of a currently running workspace.

Implementation

• Eclipse Che Server doesn't notify user when OOM happens during workspace start Eclipse Che Server doesn't notify user when OOM happens during workspace start #13511
• Publish plugin broker pod events Publish plugin broker pod events #14854
• Misleading message form CHE server when a container stopped Misleading message from CHE server when a container stopped  #14781
• [POC] Access to the workspace containers logs [POC] Access to the workspace containers logs #15134
• [ ] [POC] Access to the workspace lifetime event log [POC] Access to the workspace lifetime event log #15135

Access to the workspace logs

as a result of #15134

• [ ] Write workspace's containers stdout/err log to file Write workspace's containers stdout/err log to file #15834
• [ ] Create LogCollector component Create LogCollector component #15835
• [ ] Archive logs on che-server Archive logs on che-server #15836
• [ ] User can access workspace logs from che-server User can access workspace logs from che-server #15837
• Workspace start in debug mode -> watch the logs Workspace start in debug mode -> watch the logs #15983
• [dashboard] - restart workspace in debug mode after startup failure [dashboard] - restart workspace in debug mode after startup failure #16050
• Watch plugin broker logs in workspace startup debug mode Watch plugin broker logs in workspace startup debug mode #16061
• new metrics related to workspace startup log watch new metrics related to workspace startup log watch #16062
• [dashboard] - make it possible to download all workspace startup logs [dashboard] - make it possible to download all workspace startup logs #16063
• Watch plugin broker logs Watch plugin broker logs #16117
• [chectl] - offer to run workspace in debug mode [chectl] - offer to run workspace in debug mode #16119
• Watch mkdir pod logs in workspace startup debug mode Watch mkdir pod logs in workspace startup debug mode #16161

Access to the workspace lifetime event log

as a result of #15135
TBD

Describe alternatives you've considered

Additional context

[skabashnyuk]

@l0rd @slemeur

1. The description of this epic is more concentrated around logs. Should we invest our time on k8s events [POC] Access to the workspace lifetime event log #15135 or logs should be our p1?

2. Can we put resource usage out of the scope of this task? It is very connected to Dashboard: provide target k8s namespace information  #13484 maybe we should implement it as part of that epic?

[sparkoo]

Here's some proposal draft of possible solution:

1. archive stdout logs

1. stdout/err logs of all ws containers to file on containers. The location has to be on workspace PV and has to be known.
2. Watch ws exit event -> start collector pod that will grab all logs and send them to the backend.

What is backend is open question. At this point, we will send logs to the master's logs endpoint, which does not exist yet.

2. access archived logs

1. Design che-server API to provide logs to the user
2. Provide the logs on request via the API. This very much depends on chosen storage backend.

3. collect file logs

Some components might be logging into files somewhere in the container. We can't know where all the logs are, thus we have to give the option to define the directory, where the component is logging, to be able to collect them later. We need to enhance plugin's meta.yaml as well as the devfile. Once we have that, we can collect the logs with collector pod, same as in 1..

1. Design meta.yaml and devfile changes needed to be able to define file logs dir to collect
2. Somehow grab these logs with collector

extras: things to not forget to at least think about them

• quotas management
• what if logs are big (gigs)? collect only last 100M?
• what if ephemeral mode ?
• what all should be configurable? (number of archived workspace starts, max size of the logs per user, cleanup old logs based on time)

[sparkoo]

all issues int the scope of this epic are done, closing