Job manager using systemd #49

bzizou · 2024-04-04T15:09:55Z

bzizou
Apr 4, 2024
Maintainer

Dans le job manager, phase d'init (préparation du slice):

oar@bigfoot-gh1:~$ oardodo systemd-run --uid=bzizou --slice oar_job_bzizou_6666 -p Delegate=yes sleep infinity
oar@bigfoot-gh1:~$ oardodo systemctl set-property oar_job_bzizou_6666.slice AllowedCPUs=12-13

On obtient ça:

oar@bigfoot-gh1:~$ systemd-cgls -u oar_job_bzizou_6666.slice
Unit oar_job_bzizou_6666.slice (/oar_job_bzizou_6666.slice):
└─run-ra3b4750f3e7d4169802a44c98c9a6a21.service
  └─15486 /usr/bin/sleep infinity

Dans oarsh_shell, lancement du shell via systemd:

oar@bigfoot-gh1:~$ oardodo systemd-run --uid=bzizou --scope --slice oar_job_bzizou_6666 bash
bzizou@bigfoot-gh1:/var/lib/oar$

On voit bien le shell et ses process dans un scope du slice:

bzizou@bigfoot-gh1:/var/lib/oar$ systemd-cgls -u oar_job_bzizou_6666.slice
Unit oar_job_bzizou_6666.slice (/oar_job_bzizou_6666.slice):
├─run-r70d4f5d40f164f879e98d0e2ecb6bbcb.scope
│ ├─15553 /usr/bin/bash
│ ├─15555 systemd-cgls -u oar_job_bzizou_6666.slice
│ └─15556 less
└─run-ra3b4750f3e7d4169802a44c98c9a6a21.service
  └─15486 /usr/bin/sleep infinity

On tourne bien sur les cpu spécifiées:

bzizou@bigfoot-gh1:/var/lib/oar$ ps -axo pid,cpuid,comm |grep 15553
  15553    12 bash

Et dans le job_manager, au clean, on peut faire:

oar@bigfoot-gh1:~$ oardodo systemctl kill -s 9 oar_job_bzizou_6666.slice

Hop:

bzizou@bigfoot-gh1:/var/lib/oar$ Killed
oar@bigfoot-gh1:~$

En déléguant tout à systemd, j'ai l'impression qu'on aurait un truc bien propre, et on n'a même plus besoin de se soucier qu'on soit en cgroup v1 ou v2!

npf · 2024-04-04T15:48:07Z

npf
Apr 4, 2024
Maintainer

Tu as regardé pour les GPU/block devices ?

2 replies

bzizou Apr 5, 2024
Maintainer Author

It's on my TODO list for today ;-)

bzizou Apr 5, 2024
Maintainer Author

For block Devices, it should also work:
https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#Device%20Access

I tried a POC with GPUS and the command nvidia-smi:

Test without /dev/nvidia0 into the allowed devices:

systemd-run --uid=bzizou --slice oar_job_bzizou_6666 -p Delegate=yes sleep infinity
systemctl set-property oar_job_bzizou_6666.slice AllowedCPUs=12-13 DevicePolicy=closed DeviceAllow=/dev/nvidiactl
systemd-run --uid=bzizou --scope --slice oar_job_bzizou_6666 bash
nvidia-smi

Test with /dev/nvidia0 into the allowed devices:

systemd-run --uid=bzizou --slice oar_job_bzizou_6666 -p Delegate=yes sleep infinity
systemctl set-property oar_job_bzizou_6666.slice AllowedCPUs=12-13 DevicePolicy=closed DeviceAllow=/dev/nvidiactl DeviceAllow=/dev/nvidia0
systemd-run --uid=bzizou --scope --slice oar_job_bzizou_6666 bash
nvidia-smi

In both cases, the nvidia-smi command reports the list of the GPUS... but maybe the usage is not allowed, we have to check with a real usage (pytorch ?). And without /dev/nvidiactl , the nvidia-smi commands reports an error in both cases.... more tests to come...

bzizou · 2024-04-11T13:58:57Z

bzizou
Apr 11, 2024
Maintainer Author

Simple Cuda code that detects GPUS and print some informations (may behave differently than nvidia-smi, so useful to check if a GPU is accessible for programming and not only for displaying with nvidia-smi)

#include <stdio.h>             
                                                               
int main() {
 int nDevices;                                                 
                               
 cudaGetDeviceCount(&nDevices); 
 printf("Found %d devices\n", nDevices);
 for (int i = 0; i < nDevices; i++) {
   cudaDeviceProp prop;       
   cudaGetDeviceProperties(&prop, i);     
   printf("Device Number: %d\n", i);
   printf("  Device name: %s\n", prop.name);
   printf("  Memory Clock Rate (KHz): %d\n",
          prop.memoryClockRate);
   printf("  Memory Bus Width (bits): %d\n",
          prop.memoryBusWidth); 
   printf("  Peak Memory Bandwidth (GB/s): %f\n\n",
          2.0*prop.memoryClockRate*(prop.memoryBusWidth/8)/1.0e6);
 }
}

(compile with nvcc -o test test.cu)

0 replies

bzizou · 2024-04-12T10:27:06Z

bzizou
Apr 12, 2024
Maintainer Author

POC For Nvidia GPU access crontrol with systemd

Fake a JOB

JOB_ID=6671

Create a transcient slice

oardodo systemd-run --uid=bzizou --slice oar_job_bzizou_$JOB_ID -p Delegate=yes sleep infinity

Set up properties of the slice without device /dev/nvidia0 allocation

oardodo systemctl set-property oar_job_bzizou_$JOB_ID.slice \
  AllowedCPUs=0-71 \
  AllowedMemoryNodes=""  \
  DevicePolicy=closed \
  DeviceAllow="/dev/nvidia-caps/nvidia-cap1 rwm" \
  DeviceAllow="/dev/nvidia-caps/nvidia-cap2 rwm" \
  DeviceAllow="/dev/nvidiactl rwm" \
  DeviceAllow="/dev/nvidia-modeset rwm" \
  DeviceAllow="/dev/nvidia-uvm rwm" \
  DeviceAllow="/dev/nvidia-uvm-tools rwm"

Launch a shell inside the slice ant test

oardodo systemd-run --uid=bzizou -p Delegate=yes --scope --slice oar_job_bzizou_$JOB_ID --unit oar_job_bzizou_$JOB_ID bash

Running scope as unit: oar_job_bzizou_6670.scope
bzizou@bigfoot-gh1:/var/lib/oar$ ~/tmp/test
Found 0 devices
bzizou@bigfoot-gh1:/var/lib/oar$ 
exit

Now allow the /dev/nvidia0 device and test again

oardodo systemctl set-property oar_job_bzizou_$JOB_ID.slice DeviceAllow="/dev/nvidia0 rwm"

Running scope as unit: oar_job_bzizou_6672.scope
bzizou@bigfoot-gh1:/var/lib/oar$ ~/tmp/test 
Found 1 devices
Device Number: 0
  Device name: NVIDIA GH200 480GB
  Memory Clock Rate (KHz): 2619000
  Memory Bus Width (bits): 6144
  Peak Memory Bandwidth (GB/s): 4022.784000

Kill the job

systemctl stop oar_job_bzizou_$JOB_ID

3 replies

npf May 23, 2024
Maintainer

The problem with DevicePolicy=close is that it's a white listing approach instead of a black list like we have with cgroup v1.

Here is an example of a problem: unless /dev/infiniband/umad* is whitelisted, iblinkinfo does not work.

root@drac-3:/dev# iblinkinfo 
ibwarn: [2357] mad_rpc_open_port: can't open UMAD port ((null):0)

npf May 23, 2024
Maintainer

device.deny is not welcomed in systemd for now :-(

A DeviceDeny directive could be useful systemd/systemd#19222
Add DeviceAllow=~ or DeviceDeny systemd/systemd#30780
Add a blacklist inverse of DeviceAllow systemd/systemd#31695
and explicit whitelisting is ugly :-(

npf May 24, 2024
Maintainer

ebpf explained: https://dropbear.xyz/2023/05/23/devices-with-cgroup-v2/

bzizou · 2024-04-17T12:14:39Z

bzizou
Apr 17, 2024
Maintainer Author

Working solution:

https://github.com/oar-team/oar3/tree/systemd , resulting in 2 files, OAR2/OAR3 compatibles:

https://github.com/oar-team/oar3/blob/systemd/oar/tools/job_resource_manager_systemd.pl # To set up into oar.conf
https://github.com/oar-team/oar3/blob/systemd/oar/tools/oarsh/oarsh_shell.in # To replace on the nodes you want to be managed by systemd
and touch /etc/oar/use_systemd on the nodes to be managed by systemd, cgroup v2 enabled (systemd.unified_cgroup_hierarchy=true)

currently in production on my "bigfoot" cluster (OAR 2.5.9) with 14 normal nodes and 1 experimental systemd node

0 replies

npf · 2024-05-17T08:56:36Z

npf
May 17, 2024
Maintainer

Some refs:

About services vs scopes:
https://unix.stackexchange.com/questions/688298/what-is-the-difference-between-a-systemd-scope-and-a-systemd-slice
About creating a scope with busctl
https://unix.stackexchange.com/questions/525740/how-do-i-create-a-systemd-scope-for-an-already-existing-process-from-the-command
Systemd dbus API
https://www.freedesktop.org/software/systemd/man/latest/org.freedesktop.systemd1.html
DBus data types
https://www.alteeve.com/w/List_of_DBus_data_types
resource control
https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#
Slumd doc
https://slurm.schedmd.com/cgroup_v2.html

0 replies

npf · 2024-05-17T12:29:14Z

npf
May 17, 2024
Maintainer

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartTransientUnit 'ssa(sv)a(sa(sv))' oar-firstname_lastname-jobid-c4.scope fail 4 Delegate b 1 PIDs au 1 6830 Slice s oar-firstname_lastname-jobid.slice AllowedCPUs ay 2 0x00 0x80 0

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager AttachProcessesToUnit 'ssau' oar-firstname_lastname-jobid-c4.scope / 1 30217

busctl introspect org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/oar_2dfirst_5fname_2d16041977_2dc4_2escope

busctl introspect org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/oar_2dfirst_5fname_2d16041977_2dc4_2escope org.freedesktop.systemd1.Scope

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/oar_2dfirst_5fname_2djobid_2dc4_2escope org.freedesktop.systemd1.Scope AttachProcesses sau / 1 46842

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartUnit ss toto-tata-titi.slice fail

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager SetUnitProperties 'sba(sv)' toto.slice 1 1 AllowedCPUs ay 1 6

0 replies

npf · 2024-05-17T14:31:53Z

npf
May 17, 2024
Maintainer

Create job slice with hierarchy oar/user/jobid and set cpuset procs (1+2 here)

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartUnit ss oar-f_n-jobid.slice fail
busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager SetUnitProperties 'sba(sv)' oar-f_n-jobid.slice 1 1 AllowedCPUs ay 1 6

Create a scope for the job and attach an existing process to it (60800 here)

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartTransientUnit 'ssa(sv)a(sa(sv))' oar-f_n-jobid-60800.scope fail 3 Delegate b 1 PIDs au 1 60800 Slice s oar-f_n-jobid.slice 0

Create subjob slice with a subset of procs (1 only)

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartUnit ss oar-f_n-jobid-c1.slice fail
busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager SetUnitProperties 'sba(sv)' oar-f_n-jobid-c1.slice 1 1 AllowedCPUs ay 1 2

Create a scope for the job and attach an existing process to it (60801 here)

busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager StartTransientUnit 'ssa(sv)a(sa(sv))' oar-f_n-jobid-c1-60801.scope fail 3 Delegate b 1 PIDs au 1 60801 Slice s oar-f_n-jobid-c1.slice 0

0 replies

bzizou · 2024-06-17T11:33:00Z

bzizou
Jun 17, 2024
Maintainer Author

Nvidia just gave me this link as a reference to try to fix the original issue that urged me to use systemd and cgroupv2 (original problem was that a GH200 host frozed each time a nvidia-smi command was passed from inside a cpuset v1)

https://developer.nvidia.com/blog/improving-cuda-initialization-times-using-cgroups-in-certain-scenarios/

1 reply

npf Jun 28, 2024
Maintainer

Ok, then mounting CG v1 for devices only, on top of CG v2, permits to deny devices quite easily (tested ok on a debian12 which mounts CG v2 only by default).
Not sure however how long this "workaround" will be valid.

CG v2 device deny using eBPF will also be ok soon (the oarcgdev utility is almost ready).

npf · 2024-10-09T16:16:56Z

npf
Oct 9, 2024
Maintainer

I missed the net_cls conversion from cgroup v1 to v2... This seems to involve cgroup v2 namespace: https://unix.stackexchange.com/questions/412775/iptables-and-cgroups-v2-netfilters-xt-cgroup

Also, need to see how to work nicely the AllowedMemoryNodes property with the dbus API syntax...

0 replies

npf · 2024-10-21T20:42:58Z

npf
Oct 21, 2024
Maintainer

root@dahu-21:~# busctl call org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager SetUnitProperties 'sba(sv)' oar-f_n-jobid.slice 1 2 AllowedCPUs $(/home/pneyron/scm/hwloc/utils/hwloc/hwloc-calc --cpuset-output-format systemd-dbus-api node:1) AllowedMemoryNodes $(/home/pneyron/scm/hwloc/utils/hwloc/hwloc-calc --nodeset-output-format systemd-dbus-api node:1)
root@dahu-21:~# cat /sys/fs/cgroup/oar.slice/oar-f_n.slice/oar-f_n-jobid.slice/cpuset.cpus.effective 
1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
root@dahu-21:~# cat /sys/fs/cgroup/oar.slice/oar-f_n.slice/oar-f_n-jobid.slice/cpuset.mems.effective 
1

😎

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Job manager using systemd #49

{{title}}

Replies: 10 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Job manager using systemd #49

bzizou Apr 4, 2024 Maintainer

Dans le job manager, phase d'init (préparation du slice):

On obtient ça:

Dans oarsh_shell, lancement du shell via systemd:

On voit bien le shell et ses process dans un scope du slice:

On tourne bien sur les cpu spécifiées:

Et dans le job_manager, au clean, on peut faire:

Hop:

Replies: 10 comments · 6 replies

npf Apr 4, 2024 Maintainer

bzizou Apr 5, 2024 Maintainer Author

bzizou Apr 5, 2024 Maintainer Author

bzizou Apr 11, 2024 Maintainer Author

bzizou Apr 12, 2024 Maintainer Author

POC For Nvidia GPU access crontrol with systemd

Fake a JOB

Create a transcient slice

Set up properties of the slice without device /dev/nvidia0 allocation

Launch a shell inside the slice ant test

Now allow the /dev/nvidia0 device and test again

Kill the job

npf May 23, 2024 Maintainer

npf May 23, 2024 Maintainer

npf May 24, 2024 Maintainer

bzizou Apr 17, 2024 Maintainer Author

npf May 17, 2024 Maintainer

npf May 17, 2024 Maintainer

npf May 17, 2024 Maintainer

bzizou Jun 17, 2024 Maintainer Author

npf Jun 28, 2024 Maintainer

npf Oct 9, 2024 Maintainer

npf Oct 21, 2024 Maintainer

bzizou
Apr 4, 2024
Maintainer

Replies: 10 comments 6 replies

npf
Apr 4, 2024
Maintainer

bzizou Apr 5, 2024
Maintainer Author

bzizou Apr 5, 2024
Maintainer Author

bzizou
Apr 11, 2024
Maintainer Author

bzizou
Apr 12, 2024
Maintainer Author

npf May 23, 2024
Maintainer

npf May 23, 2024
Maintainer

npf May 24, 2024
Maintainer

bzizou
Apr 17, 2024
Maintainer Author

npf
May 17, 2024
Maintainer

npf
May 17, 2024
Maintainer

npf
May 17, 2024
Maintainer

bzizou
Jun 17, 2024
Maintainer Author

npf Jun 28, 2024
Maintainer

npf
Oct 9, 2024
Maintainer

npf
Oct 21, 2024
Maintainer