nspawn ignores limits

[jeffka11]

Submission type

• Bug report
• Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

232

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Debian stretch

In case of bug report: Expected behaviour you didn't see

a change in max open files

In case of bug report: Unexpected behaviour you saw

process 1142 is a nspawn container started with systemd-nspawn@container.service

jeff@host:/etc/systemd/system/systemd-nspawn@.service.d# cat /proc/1142/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    0                    bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             100                  100                  processes 
Max open files            100                  100                  files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       64030                64030                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us      

with the same data (I only checked max files) when looking at the container

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64030
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

In case of bug report: Steps to reproduce the problem

I tried the following ways to increase the limit:

1. created a ulmit.cnf file in /etc/systemd/system/systemd-nspawn@.service.d which adds LimitNOFILE=infinity in the [Service] section, systemctl daemon-reload, then start the container with machinectl start container
2. same file in /etc/systemd/system/systemd-nspawn@container.service.d and same reload and retry process as above
3. gave the container Capabilities=CAP_SYS_RESOURCE CAP_SYS_ADMIN in a /etc/systemd/nspawn/container.nspawn file and tried updating limits via ulimit -n 10000 inside the container and same reload and retry process as above

if I make a container2 with the same OS as container, except I start it with systemd-nspawn -bD container2 instead of the service file (systemctl start systemd-nspawn@container2.service), Item 3 works, although item 1 and item 2 don't. 1. Note: use PrivateUsers=0 and PrivateUsersChown=false in the .nspawn file, in addition to the capabilities listed above.

On a possibly related note, inside container2, if I try to remove the ulimit -n 10000 step by creating a service file /etc/systemd/system/application.service.d/ulimit.cnf, similar to above, the application complains about too many open files. Using the ulimit -n 10000 command allows the application to open up the files.

Ideally, I'd be able to use the service drop-ins to change the ulimit, but another option is to change the default from 100 (see the cat /proc/1142/limits output) to something else, although I don't know where the 100 is coming from.

At all times during my testing, on the host, the /etc/security/limits.conf file contains:

  * - nofile 1048576
  root - nofile 1048576
  1: - nofile 1048576

edit: clarified steps to reproduce
edit2: after an additional test of creating a separate service file without the -U flag, I can start container2 with this service file. This was after the above testing, so something above may have permanently changed the situation, just thought this was a relevant thing to say:

root@host:/etc/systemd/system# diff /lib/systemd/system/systemd-nspawn@.service systemd-nspawn-alt@.service
16c16
< ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=%i
---
> ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth --settings=override --machine=%i

[evverx]

created a ulmit.cnf
/etc/systemd/system/application.service.d/ulimit.cnf

Try *.conf. Does it work?

mkdir -p /etc/systemd/system/systemd-nspawn@.service.d/
cat <<'EOF' >/etc/systemd/system/systemd-nspawn@.service.d/ulimit.conf
[Service]
LimitNOFILE=infinity
EOF
systemctl daemon-reload
systemctl stop systemd-nspawn@CONTAINER
systemctl start systemd-nspawn@CONTAINER

[evverx]

Anyway, .cnf doesn't work. I'm closing.