[jb]: monitor low memory notifications

[akosyakov]

Description

As for now we monitor JVM used and max memory. It is not enough though to detect perf degradation, since GC can kick off and release memory. It turned out though that JB backend already monitors performance degradation internally and notifies a user about it with low memory notification after GC. This PR adds a counter of low memory notifications. Steady increase of such notifications means performance degradation for a user.

Related Issue(s)

Monitoring for #8704

How to test

Reproducing low memory notifications

• Configure prev envs with IntelliJ IDEA: https://ak-jb-gc-pause.preview.gitpod-dev.com/preferences
• Start a workspace in prev envs with low max heap to emulate low memory usage: https://ak-jb-gc-pause.preview.gitpod-dev.com/#INTELLIJ_VMOPTIONS=-Xmx384m/https://github.com/eclipse/xtext-core
• Connect with IntelliJ and confirm loading gradle project.

• Double Shift → Find and open LanguageServerImpl.java file
• Find some method and start typing to get auto triggered completions:
  • for local symbols like parameters
  • for global symbols like IllegalStateExceptions
• Now try to trigger completions manually via Ctrl+Space.
• Repeat 2 steps above till you get low memory notifications. It usually happens after indexing of JDK is done, IntellIJ is trying to index gradle project itself and serve completions already. After you seen it do above a bit more to trigger more notifications. You won't see them on UI new but backend will count them.

Monitoring

• Start a dev workspace https://gitpod.io#https://github.com/gitpod-io/gitpod/pull/10558
• Run ./dev/preview/portforward-monitoring-satellite.sh -c harvester to port forward prometheus API endpoint.
• Run gp preview $(gp url 3000)/d/oamBLUC7k/jetbrains-overview?orgId=1 --external to open JetBrains Overview dashboard.
• You can see rate of low memory after GC notifications in the last 5 minutes as well as top 10 pods.

Release Notes

NONE

Documentation

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.20 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.21 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.22 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.23 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.24 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.25 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.26 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.27 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.28 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.29 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.30 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.31 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.33 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.34 because the annotations in the pull request description changed
(with .werft/ from main)

[akosyakov]

/werft run with-clean-slate-deployment=true

👍 started the job as gitpod-build-ak-jb-gc-pause.35
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-ak-jb-gc-pause.36 because the annotations in the pull request description changed
(with .werft/ from main)

[akosyakov]

@mustard-mh Could you have a look please? 🙏

[mustard-mh]

Prometheus and Grafana graph view as expected (I know it's shown, but not clear what the numbers means)

P	G

When I click option Configuration of Low Mem Notification, restart IDE, and re-conn, it seems not work

Set	New Conn

[mustard-mh]

Code LGTM

/hold

Not sure if I need to test with observability too? Since we can see it in dev Grafana

[akosyakov]

Not sure if I need to test with observability too? Since we can see it in dev Grafana

dev Grafana should be enough

Prometheus and Grafana graph view as expected (I know it's shown, but not clear what the numbers means)

First graph show overall in the system (among all pods) how many low memory cases happen. Second show top 10 pods with highest number of notifications. First graph should be clean ideally. Second should be used to investigate worst cases.

When I click option Configuration of Low Mem Notification, restart IDE, and re-conn, it seems not work

Yes, it does not work. I will file an issue to investigate how to fix it.

[akosyakov]

/unhold

[akosyakov]

a follow-up to upgrade of Xmx: #10715