Skip to content

Commit

Permalink
Decrease maximum limit for number of possible open files
Browse files Browse the repository at this point in the history
When installing python packages via apt, a pycompile process would get
stuck in a seemingly infinite loop.

Debian (Ubuntu), seems to run a post install script after every (?) python
package installed via apt. That script calls pycompile [0], which in
turn makes a call to Popen() with close_fds=True [1]. In (C)Python
2.7, at least, subprocess (Popen()) would then try to close all possibly
open files [2].

I'm guessing (and strace'ing the stuck process seems to agree) that it
gets stuck trying to close "file descriptors" that are not actually
linked to any file. I guess for a low OPEN_MAX this would be ok. But in
my machine I have these limits:
 $ docker run debian:buster getconf OPEN_MAX
 1073741816

Although locally (not in a docker container) I have this:
 $ getconf OPEN_MAX
 1024
which I guess comes from a default config for systemd? (Archlinux)
/etc/systemd/system.conf: #DefaultLimitNOFILE=1024:524288

Not sure if this has to do with a later version of docker, or a
combination of new docker version and new kernel? More investigation is
required here to know how OPEN_MAX gets its value in a docker container
(and during build).

For now, this is a way of fixing the problem for _running_ containers.
During build however, I don't think this will work. For that you might
want to change your systemd unit for docker to something like:

 LimitNOFILE=1048576

Instead of "infinity", which is what moby (docker) now ships with [3].

There's a chance this is what was causing [4] as well? Unsure.

[0]: https://salsa.debian.org/cpython-team/python-defaults/-/blob/2.7.18-2/pycompile
[1]: https://salsa.debian.org/cpython-team/python-defaults/-/blob/2.7.18-2/pycompile#L134-135
[2]: https://github.com/python/cpython/blob/v2.7.16/Lib/subprocess.py#L873-L874
[3]: moby/moby@80039b4
[4]: moby/buildkit#1785

[DHDO-789]
  • Loading branch information
lsago committed Feb 28, 2023
1 parent f812a25 commit 97ab4fd
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions docker-compose-irods.yml
Expand Up @@ -50,6 +50,10 @@ services:
- net.ipv4.tcp_keepalive_intvl=75
- net.ipv4.tcp_keepalive_probes=9
- net.ipv4.tcp_keepalive_time=600
ulimits:
nofile:
soft: 1048576
hard: 1048576
depends_on:
- irods-db
hostname: icat.dh.local
Expand Down Expand Up @@ -132,6 +136,10 @@ services:
- net.ipv4.tcp_keepalive_intvl=75
- net.ipv4.tcp_keepalive_probes=9
- net.ipv4.tcp_keepalive_time=600
ulimits:
nofile:
soft: 1048576
hard: 1048576
hostname: ires-hnas-um.dh.local
env_file:
- .env
Expand Down Expand Up @@ -214,6 +222,10 @@ services:
- net.ipv4.tcp_keepalive_intvl=75
- net.ipv4.tcp_keepalive_probes=9
- net.ipv4.tcp_keepalive_time=600
ulimits:
nofile:
soft: 1048576
hard: 1048576
hostname: ires-hnas-azm.dh.local
env_file:
- .env
Expand Down

0 comments on commit 97ab4fd

Please sign in to comment.