/dev/null: Operation not permitted when using cgroup, and running as root

[hpckorea]

I found following bugs when I use singularity-3.10.0-rc1 with option '--cpus' and '--memory'.

• I got following error when I execute /usr/sbin/sshd
  bash: /dev/null: Operation not permitted
• It shows more cores in the top command than assigning by '--cpus'
• It shows more memory in the top command than assigning by '--memory'

[dtrudg]

It shows more cores in the top command than assigning by '--cpus'

This is normal. cgroups cpu limits will limit the amount of cpu usage a container can make, by restricting the amount of time a container runs on the system's CPUs. It will not reduce the number of visible CPUs.

It shows more memory in the top command than assigning by '--memory'

This is normal. cgroups limits can place restrictions on memory usage for a process / container, but they do not change the system level information that top/htop displays. You must look at the cgroup to see the applied limits, not system level memory information.

Both of these behaviors are the same as e.g. docker.

got following error when I execute /usr/sbin/sshd
bash: /dev/null: Operation not permitted

What is the container you are using? What is the full command line you are using? We will need some more information to identify the issue.

Note that generally sshd cannot be run in a non-root container. It requires privilege separation and must be started as root.

[hpckorea]

Thank you for your interesting.

I did it like below as root user.
o singularity pull rckl.sif docker://rockylinux
o singularity build --sandbox rckl rckl.sif
o singularity shell -w --cpus 1 rckl
o singularity> dnf install openssh-server openssh-clients lsof
o singularity> /usr/libexec/openssh/sshd-keygen rsa
o singularity> /usr/libexec/openssh/sshd-keygen ecdsa
o singularity> /usr/libexec/openssh/sshd-keygen ed25519
o singularity> /usr/sbin/sshd
bash: /dev/null: Operation not permitted

If I use a command with 'singularity shell -w rckl', it works fine.
But when I add a option '--cpus #', it gives that error.

[dtrudg]

Thanks for the additional information - that does help isolate the issue!

[dtrudg]

Can also be replicated on master with an empty toml file and --apply-cgroups as root:

$ touch null.toml
$ sudo singularity exec -w --apply-cgroups null.toml centos8 cat /dev/null
bash: /dev/null: Operation not permitted

Switching to cgroupfs (systemd cgroups = no) does not fix the issue:

On Singularity 3.9.9 it works as expected:

$ singularity --version
singularity-ce version 3.9.

$ sudo singularity exec --apply-cgroups null.toml -w centos8 cat /dev/null