Not enough memory for sidecar container

[svor]

Description

Sometimes sidecar container has not enough memory to work perfect. In that case it is very hard to understand that problem is with memory limit. It would be nice to detect and notice somehow this problem, but I am not sure if it is possible.
@benoitf WDYT?

[tsmaeder]

@svor I discussed something related with @l0rd on thursday: #13485

[benoitf]

@svor yes it would be nice. Maybe report the kind of issues that you've faced when there was not enough memory.
Process killed ?

[svor]

@tsmaeder it is not just about sidecars with LS, it is about all containers, we should know if something can't work because of OOM.

@benoitf i think the cases can be different. In my case the process was killed (for java plugin), problems with web socket connection (for apache-camel plugin https://github.com/svor/files/blob/master/theia-ide_log). It is difficult to predict what might be broken

[benoitf]

@svor ok thx. For websocket connection errors we can easily display reports to the user.

For the process killed, as they're started by the vs code extensions (and the vscode language server library is more difficult. Probably propose patchs to upstream vscode project and add new events) or enhance vscode-java to check process after it's spawn.
We could detect new processes but it's hard to know from the framework if the process should be with a long life a short life (as extensions can spawn any processes they want)

[AndrienkoAleksandr]

Related:
#13511
#13512

[tsmaeder]

As I understand it, the container gets killed as well, when more memory is requested than available. Is this something we can detect in theia or does it need to be determined from kubernetes? Do we understand in which cases the container gets killed and in which cases only the process dies?

[l0rd]

If it can help here is some k8s doc and in this is a command to retrieve a Che workspace pod failures reasons (a couple of containers were OOMKilled):

$ k get po -o json workspacezkdyy2ccc7vjs60h.workspace-7d87b984d6-zkdxh | jq  --raw-output '.status.containerStatuses[] | "\(.image) \(.name) \(.lastState.terminated.reason)"'
eclipse/che-remote-plugin-runner-java8:next 3g0 Error
eclipse/che-machine-exec:latest che-machine-exec25j Error
eclipse/che-theia-endpoint-runtime:next kcm OOMKilled
maven:3.6.0-jdk-11 maven-container Error
centos/mysql-57-centos7:latest mysql Error
mariolet/petclinic:d2831f9b spring-boot Error
eclipse/che-theia:next theia-ideb3h Error
eclipse/che-remote-plugin-kubernetes-tooling-1.0.0:next xey OOMKilled

In practice we may add this information to the containers view or create a new plugin that watch for errors in a Che workspace pod and show them in the error panel.

[mmorhun]

I have similar use case, but instead of OOM, yaml plugin just doesn't work.
From logs it is clear that problem is the memory limits:

(node:12) UnhandledPromiseRejectionWarning: Error: ENOMEM: not enough memory, write

Setting memory limit to 512 Mb resolved the problem.

So, at least yaml plugin doesn't work with default settings. I think we should check other plugins and make them work with default settings.

[benoitf]

@mmorhun do you have trace on your UnhandledPromiseRejectionWarning, to know in which code we could handle this error ?

[mmorhun]

All what I have is logs from container with autogenerated name

Logs

 Starting the deployer with the list of resolvers [ LocalDirectoryPluginDeployerResolver {},
  GithubPluginDeployerResolver { unpackedFolder: '/tmp/github-remote' },
  HttpPluginDeployerResolver { unpackedFolder: '/tmp/http-remote' },
  VsCodePluginDeployerResolver { vscodeExtensionsFolder: '/tmp/vscode-extension-marketplace' } ]
Theia Endpoint 13/pid listening on port 2503
Found the list of default plugins ID on env: undefined
Found the list of plugins ID on env: local-dir:///plugins/sidecars/redhat_vscode_yaml_latest
Found the list of default plugins ID from CLI: undefined
unzipping the plugin ProxyPluginDeployerEntry {
  deployer:
   PluginVsCodeFileHandler { unpackedFolder: '/tmp/vscode-unpacked' },
  delegate:
   PluginDeployerEntryImpl {
     originId: 'local-dir:///plugins/sidecars/redhat_vscode_yaml_latest',
     pluginId:
      'redhat.vscode-yaml.latest.qagupxptni.redhat.vscode-yaml-0.4.0.vsix',
     map: Map {},
     changes: [],
     acceptedTypes: [],
     currentPath:
      '/plugins/sidecars/redhat_vscode_yaml_latest/redhat.vscode-yaml.latest.qagupxptni.redhat.vscode-yaml-0.4.0.vsix',
     initPath:
      '/plugins/sidecars/redhat_vscode_yaml_latest/redhat.vscode-yaml.latest.qagupxptni.redhat.vscode-yaml-0.4.0.vsix',
     resolved: true,
     resolvedByName: 'LocalDirectoryPluginDeployerResolver' },
  deployerName: 'PluginVsCodeFileHandler' }
(node:13) UnhandledPromiseRejectionWarning: Error: ENOMEM: not enough memory, write
(node:13) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:13) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. 

[benoitf]

thx @mmorhun

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.