[ws-daemon] Add plugin to set custom process priorities #11159

aledbf · 2022-07-05T21:11:40Z

Description

Increase supervisor and code-node process priority inside the workspaces.
The goal is to improve responsiveness in busy workspaces/nodes.

How to test

Start a workspace.
Wait 30s
Check process priorities running
- ps ax -o pid,ni,cmd|grep supervisor
- ps ax -o pid,ni,cmd|grep gitpod-code

Should be similar to

# ps ax -o pid,ni,cmd|grep gitpod-code
 365036  -5 sh /ide/bin/gitpod-code --install-builtin-extension gitpod.gitpod-theme --install-builtin-extension github.vscode-pull-request-github --install-extension ms-python.python --port 23000 --host 0.0.0.0 --without-connection-token --server-data-dir /workspace/.vscode-remote --do-not-sync --start-server

# ps ax -o pid,ni,cmd|grep supervisor
 365017 -10 supervisor run

Release Notes

NONE

Werft options:

/werft with-preview

Furisto · 2022-07-05T23:21:27Z

/werft run with-preview

👍 started the job as gitpod-build-aledbf-prio2.5
(with .werft/ from main)

aledbf · 2022-07-05T23:23:26Z

@Furisto this will not work until cgroup v2. You can test this change with workspace-preview

components/ws-daemon/pkg/cgroup/plugin_process_priority_v2.go

utam0k · 2022-07-06T03:42:04Z

/werft run with-preview

👍 started the job as gitpod-build-aledbf-prio2.12
(with .werft/ from main)

utam0k · 2022-07-06T03:42:18Z

I edited this preview env to enable cgroup v2

utam0k · 2022-07-06T04:07:01Z

/werft run with-preview

👍 started the job as gitpod-build-aledbf-prio2.16
(with .werft/ from main)

utam0k

It looks fine

root@aledbf-prio2:/# ps ax -o pid,ni,cmd | grep gitpod-code | grep -v grep
 549984  -5 sh /ide/bin/gitpod-code --install-builtin-extension gitpod.gitpod-theme --install-builtin-extension github.vscode-pull-request-github --install-extension golang.go --port 23000 --host 0.0.0.0 --without-connection-token --server-data-dir /workspace/.vscode-remote --do-not-sync --start-server
root@aledbf-prio2:/# ps ax -o pid,ni,cmd | grep supervisor | grep -v grep
 549820   0 /.supervisor/workspacekit ring0
 549853   0 /run/containerd/io.containerd.runtime.v2.task/k8s.io/d9b51abd5584b80e80335ad908b3b31f0977ea8bee6b1ed9eb10f06bbb1d9e2c/rootfs/.supervisor/fuse-overlayfs -o lowerdir=/run/containerd/io.containerd.runtime.v2.task/k8s.io/d9b51abd5584b80e80335ad908b3b31f0977ea8bee6b1ed9eb10f06bbb1d9e2c/rootfs,upperdir=/var/gitpod/workspaces/49f3c282-b61e-4652-bb1c-0afa91d1c3d6-daemon/upper,workdir=/var/gitpod/workspaces/49f3c282-b61e-4652-bb1c-0afa91d1c3d6-daemon/work,uidmapping=0:33333:1:1:100000:65534,gidmapping=0:33333:1:1:100000:65534 none /var/gitpod/workspaces/49f3c282-b61e-4652-bb1c-0afa91d1c3d6-daemon/mark
 549868   0 supervisor init
 549925 -10 supervisor run

akosyakov · 2022-07-06T06:33:11Z

I don't see any changes to NI for relevant IDE processes in the preview env, only to shell scripts starting IDEs.

VS Code Browser

I see following in this preview env:

F   UID     PID    PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4 33333      53      33  20   0   2620   528 do_wai S    ?          0:00  \_ sh /ide/bin/gitpod-code --install-builtin-extension gitpod.gitpod-theme --install-builtin-extension github.vscode-pull-request-github --install-extension redha
0 33333    1131      53  20   0 965448 89040 ep_pol Sl   ?          0:27  |   \_ /ide/node /ide/out/server-main.js --install-builtin-extension gitpod.gitpod-theme --install-builtin-extension github.vscode-pull-request-github --install-e
0 33333    1167    1131  20   0 896072 34036 -      Sl   ?          0:01  |       \_ /ide/node /ide/out/bootstrap-fork --type=ptyHost
0 33333    2159    1167  20   0  12388  9172 do_wai Ss   pts/2      0:00  |       |   \_ /usr/bin/bash
0 33333    4209    2159  20   0   8820  2948 -      R+   pts/2      0:00  |       |       \_ ps -lafx
0 33333    1547    1131  20   0 834036 21176 ep_pol Sl   ?          0:00  |       \_ /ide/node /ide/out/bootstrap-fork --type=fileWatcher
0 33333    1561    1131  20   0 985076 101220 ep_pol Sl  ?          0:06  |       \_ /ide/node /ide/out/bootstrap-fork --type=extensionHost --transformURIs --useHostProxy=false
0 33333    1590    1561  20   0 778068 44352 ep_pol Sl   ?          0:00  |           \_ /ide/node /ide/extensions/redhat.vscode-yaml/dist/languageserver.js --node-ipc --clientProcessId=1561

Important processes are:

1131 - VS Code Server itself
1167 - terminal host process related to responsiveness to terminals
1547 - fs watcher process related to responsiveness of file explorer and editors
1561 - ext host process related to responsiveness of loading and activation of extensions

VS Code Desktop

I see following in this preview env:

F   UID     PID    PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4 33333    3045      33  20   0   3468  3036 do_pol S    ?          0:00  \_ sshd: gitpod [priv]
1 33333    3047    3045  20   0   3924  2812 do_pol S    ?          0:00      \_ sshd: gitpod@notty
0 33333    3048    3047  20   0  10436  6964 do_wai Ss   ?          0:00          \_ -bash
0 33333    3115    3048  20   0   7552  3892 pipe_r S    ?          0:00              \_ bash
0 33333    3187    3115  20   0   2620   596 do_wai S    ?          0:00                  \_ sh /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/bin/code-server --start-server --host=127.0.0.1 --accept-server-lic
0 33333    3197    3187  20   0 973788 112220 ep_pol Sl  ?          0:33                      \_ /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbc
0 33333    3233    3197  20   0 655776 43716 ep_pol Sl   ?          0:00                          \_ /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151
0 33333    3378    3197  20   0 834296 40720 ep_pol Sl   ?          0:00                          \_ /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151
0 33333    3488    3197  20   0 985876 124452 ep_pol Sl  ?          0:04                          \_ /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node /home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151
0 33333    3675    3488  20   0 5833964 918508 futex_ Sl ?          0:48                              \_ /home/gitpod/.vscode-server/extensions/redhat.java-1.8.0-linux-x64/jre/17.0.3-linux-x86_64/bin/java --add-modules=ALL-SYSTEM --add-

Important processes are:

3197 - VS Code Server itself


/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node�/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/out/server-main.js�--start-server�--host=127.0.0.1�--accept-server-license-terms�--enable-remote-auto-shutdown�--port=0�--telemetry-level�all�--install-extension�gitpod.gitpod-remote-ssh�--connection-token-file�/home/gitpod/.vscode-server/.30d9c6cd9483b2cc586687151bcbcd635f373630.token�

3233 - terminal host process related to responsiveness to terminals


/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node�/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/out/bootstrap-fork�--type=ptyHost�

3378 - fs watcher process related to responsiveness of file explorer and editors


/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node�/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/out/bootstrap-fork�--type=fileWatcher�

3488 - ext host process related to responsiveness of loading and activation of extensions


/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/node�/home/gitpod/.vscode-server/bin/30d9c6cd9483b2cc586687151bcbcd635f373630/out/bootstrap-fork�--type=extensionHost�--transformURIs�--useHostProxy=false�

JB backends

Let's do it in another PR. I'm not sure yet what processes are important there and how they are different between different products. We should have a follow up issue though.

utam0k · 2022-07-06T06:36:28Z

@akosyakov
Did you use with cgroup v2?

I see following in this preview env:

You have to see that from the root pid namespace. It means you enter the target host and run ps command because ws-daemon is in root pid namesapce.

akosyakov · 2022-07-06T09:12:44Z

@utam0k oh, i see, no i was not able

but looking at the code it does not seem it will change niceness of relevant processes, please have a look at #11159 (comment)

I did not list gitpod-code at all

aledbf · 2022-07-06T16:09:25Z

@akosyakov I will open a PR adding support for VS Code Desktop and additional VS Code Web processes.

roboquat added do-not-merge/work-in-progress do-not-merge/release-note-label-needed size/L labels Jul 5, 2022

aledbf force-pushed the aledbf/prio2 branch from 9c33515 to dbd44b0 Compare July 5, 2022 22:20

roboquat added release-note-none and removed do-not-merge/release-note-label-needed labels Jul 5, 2022

aledbf marked this pull request as ready for review July 5, 2022 22:29

aledbf requested review from a team July 5, 2022 22:29

roboquat removed the do-not-merge/work-in-progress label Jul 5, 2022

github-actions bot added team: delivery Issue belongs to the self-hosted team team: workspace Issue belongs to the Workspace team labels Jul 5, 2022

utam0k reviewed Jul 6, 2022

View reviewed changes

components/ws-daemon/pkg/cgroup/plugin_process_priority_v2.go Outdated Show resolved Hide resolved

aledbf force-pushed the aledbf/prio2 branch from cab11bc to e06224e Compare July 6, 2022 02:38

[ws-daemon] Add plugin to set custom process priorities

5d1f3a8

aledbf force-pushed the aledbf/prio2 branch from e06224e to 5d1f3a8 Compare July 6, 2022 02:44

utam0k approved these changes Jul 6, 2022

View reviewed changes

Furisto approved these changes Jul 6, 2022

View reviewed changes

mrsimonemms approved these changes Jul 6, 2022

View reviewed changes

roboquat merged commit 0564200 into main Jul 6, 2022

roboquat deleted the aledbf/prio2 branch July 6, 2022 08:09

csweichel mentioned this pull request Jul 6, 2022

Adjust process niceness within a workspace #11167

Closed

roboquat added the deployed: workspace Workspace team change is running in production label Jul 6, 2022

aledbf mentioned this pull request Jul 6, 2022

[ws-daemon] Add additional process type priorities #11186

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ws-daemon] Add plugin to set custom process priorities #11159

[ws-daemon] Add plugin to set custom process priorities #11159

aledbf commented Jul 5, 2022 •

edited

Furisto commented Jul 5, 2022 •

edited by werft-gitpod-dev-com bot

aledbf commented Jul 5, 2022

utam0k commented Jul 6, 2022 •

edited by werft-gitpod-dev-com bot

utam0k commented Jul 6, 2022

utam0k commented Jul 6, 2022 •

edited by werft-gitpod-dev-com bot

utam0k left a comment •

edited

akosyakov commented Jul 6, 2022 •

edited

utam0k commented Jul 6, 2022 •

edited

akosyakov commented Jul 6, 2022 •

edited

aledbf commented Jul 6, 2022 •

edited

[ws-daemon] Add plugin to set custom process priorities #11159

[ws-daemon] Add plugin to set custom process priorities #11159

Conversation

aledbf commented Jul 5, 2022 • edited

Description

How to test

Release Notes

Werft options:

Furisto commented Jul 5, 2022 • edited by werft-gitpod-dev-com bot

aledbf commented Jul 5, 2022

utam0k commented Jul 6, 2022 • edited by werft-gitpod-dev-com bot

utam0k commented Jul 6, 2022

utam0k commented Jul 6, 2022 • edited by werft-gitpod-dev-com bot

utam0k left a comment • edited

Choose a reason for hiding this comment

akosyakov commented Jul 6, 2022 • edited

VS Code Browser

VS Code Desktop

JB backends

utam0k commented Jul 6, 2022 • edited

akosyakov commented Jul 6, 2022 • edited

aledbf commented Jul 6, 2022 • edited

aledbf commented Jul 5, 2022 •

edited

Furisto commented Jul 5, 2022 •

edited by werft-gitpod-dev-com bot

utam0k commented Jul 6, 2022 •

edited by werft-gitpod-dev-com bot

utam0k commented Jul 6, 2022 •

edited by werft-gitpod-dev-com bot

utam0k left a comment •

edited

akosyakov commented Jul 6, 2022 •

edited

utam0k commented Jul 6, 2022 •

edited

akosyakov commented Jul 6, 2022 •

edited

aledbf commented Jul 6, 2022 •

edited