[supervisor] Make resource status request more resilient

[andreafalzetti]

Description

Makes the resource status request more resilient by avoiding calling syncronously upstream to fetch resource status at every requests. With more clients adding functionalities such as gp top or workspace cpu/memory in jetbrains ide control center, the risk of getting rate-limited is guaranteed.

With the approached used in this PR, we retrieve data from upstream at every second and when clients request it, we return the latest available data. Additional, an exponential backoff strategy is in place to be more resilient.

Related Issue(s)

Fixes #12083

How to test

Compare how this prev env behaves in comparison with gitpod.io by running a workspace and run watch -n 0.1 gp top. In gitpod.io (prod) it will almost immediately fail by returning gRPC errors because ResourceStatus endpoint is pulled too frequently and we hit the rate-limit.

In the prev env, you should be able to run multiple watch -n 0.1 gp top without any errors.

Release Notes

NONE

Documentation

Werft options:

• /werft with-preview

[akosyakov]

@andreafalzetti How do you test failure case?

[andreafalzetti]

How do you test failure case?

@akosyakov That's what is mainly left now to get this merged. If you have any suggestions or can point me to some similar tests, I would appreciate it. In the meantime I will attempt some ideas.

[akosyakov]

Run several watch gp top in parallel and then manipulate with sock file.

[akosyakov]

@andreafalzetti I actually cannot start a workspace it gets stopped immediately.

[andreafalzetti]

@andreafalzetti I actually cannot start a workspace if gets stopped immediately.

@akosyakov Weird. I was able to reproduce. Before today's changes it was working fine. Maybe it's the prev env, I will try to do a clean deployment

[andreafalzetti]

/werft run with-clean-slate-deployment

👍 started the job as gitpod-build-afalz-12083-supervisor-extend-rate-limiting.9
(with .werft/ from main)

[akosyakov]

You can check workspace logs, maybe supervisor panics somehow after last change, i.e. not initialised channel and then nil pointer exception.

[andreafalzetti]

@akosyakov If I look for the file pathes used to read memory/cpu, in my workspace, I don't see them. What am I missing?

e.g. /sys/fs/cgroup/memory/memory.limit_in_bytes -> https://github.com/gitpod-io/gitpod/blob/main/components/supervisor/pkg/supervisor/top.go#L67

cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

the sock file /.supervisor/info.sock is there, but I am not sure how to mock/alter it yet

[akosyakov]

If I look for the file pathes used to read memory/cpu, in my workspace, I don't see them. What am I missing?

for cgroup v2 there is not such files only via /.supervisor/info.sock It is expected if sock is missing then we try to compute with cgroup v1 if not possible we fail completely, one cannot compute such info for v2 from within of the workspace.

[andreafalzetti]

@akosyakov workspace starts now, it was re-closing a closed channel

[akosyakov]

works as advertised

/hold
if you want to clean up something

[akosyakov]

(nit) I think for usual clients it should be just NewTopService()

[andreafalzetti]

@akosyakov do you mean without parameters?

[akosyakov]

I left some nits, can be merged without addressing them as well. Not sure about the test, using timeouts can lead to flakiness.

[andreafalzetti]

/unhold