HTTP fails and stops the container

[simonmysun]

Specifications

• Image hash or version: d99fbaf65c5d63b186e1e997d8dafeec3e3e2267eb36959da024b8056bff4f94
• Platform: Archlinux (linux 6.8.8.arch1-1) + Docker version 26.1.0, build 9714adc6c7
• Subsystem: x86-64

Expected Behavior

With SERVICE_ENABLE_HTTP=true one should be able to access something in the browser via binded address.

Actual Behavior

The container runs normally with SERVICE_ENABLE_HTTP=false. But if I enable it, the container yields:

...
fr24feed-piaware-1  | [piaware] ADS-B data program 'dump1090' is listening on port 30005, so far so good
fr24feed-piaware-1  | [piaware] Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr 127.0.0.1 --net-bo-port 30005 --stdout
fr24feed-piaware-1  | [piaware] Started faup1090 (pid 248) to connect to dump1090
fr24feed-piaware-1  | [piaware] UAT support disabled by local configuration setting: uat-receiver-type
fr24feed-piaware-1  | [piaware] piaware received a message from dump1090!
fr24feed-piaware-1  | [piaware] piaware has successfully sent several msgs to FlightAware!
fr24feed-piaware-1  | ./run: line 17:   227 Killed                  /thttpd -D -h 0.0.0.0 -p 8080 -d /usr/lib/fr24/public_html -l - -M 60 2>&1
fr24feed-piaware-1  |        228 Done                    | mawk -W interactive '{printf "%c[32m[http]%c[0m %s\n", 27, 27, $0}'
fr24feed-piaware-1  | s6-rc: info: service legacy-services: stopping
fr24feed-piaware-1  | s6-rc: info: service legacy-services successfully stopped
fr24feed-piaware-1  | s6-rc: info: service legacy-cont-init: stopping
fr24feed-piaware-1  | s6-rc: info: service radarbox: stopping
fr24feed-piaware-1  | s6-rc: info: service planefinder-log: stopping
fr24feed-piaware-1  | s6-rc: info: service piaware: stopping
fr24feed-piaware-1  | s6-rc: info: service opensky: stopping
fr24feed-piaware-1  | s6-rc: info: service netcat: stopping
fr24feed-piaware-1  | s6-rc: info: service http: stopping
fr24feed-piaware-1  | s6-rc: info: service fr24feed: stopping
...

and exits.

Steps to Reproduce the Problem

docker pull thomx/fr24feed-piaware

docker run -p 8080:8080 -p 8754:8754 \
  --device=/dev/bus/usb:/dev/bus/usb \
  -v '/etc/localtime:/etc/localtime:ro' \
  -e 'MLAT_EXACT_LAT=50.11116935343097' \
  -e 'MLAT_EXACT_LON=8.546335229860983' \
  -e 'MLAT_ALTITUDE_MSL_METERS=123.3' \
  -e 'SERVICE_ENABLE_HTTP=true' \
  -e 'SERVICE_ENABLE_DUMP1090=true' \
  -e 'SERVICE_ENABLE_PIAWARE=true' \
  -e 'PIAWARE_FEEDER_DASH_ID=*****' \
  -e 'SERVICE_ENABLE_FR24FEED=false' \
  -e 'SERVICE_ENABLE_ADSBEXCHANGE=false' \
  -e 'SERVICE_ENABLE_PLANEFINDER=false' \
  -e 'SERVICE_ENABLE_OPENSKY=false' \
  -e 'SERVICE_ENABLE_ADSBFI=false' \
  -e 'SERVICE_ENABLE_RADARBOX=false' \
  -e 'SERVICE_ENABLE_ADSBHUB=false' \
  thomx/fr24feed-piaware

I have made sure that the port is not engaged. Is there anyway to print more verbose log?

[Thom-x]

Hello you can disable http and exec httpd manually to see the error and debug. Don't hesitate to see closed issues, it sounds familiar to me.

[simonmysun]

Hi, thank you for your hints.

the thttpd is killed by OOM killer:

[Fr Mai  3 09:51:25 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-8bd35660a3190bbe7f501e1a3a8fc0226f392271f22e2fd86eefab8567427932.scope,task=thttpd,pid=2701529,uid=65534
[Fr Mai  3 09:51:25 2024] Out of memory: Killed process 2701529 (thttpd) total-vm:180355452kB, anon-rss:51083712kB, file-rss:320kB, shmem-rss:0kB, UID:65534 pgtables:100020kB oom_score_adj:0

It works by specifying ulimit according to #107 (comment)

In my case, it's adding

services:
  fr24feed-piaware:
    ...
    ulimits:
      nproc: 65535
      nofile:
        soft: 20000
        hard: 40000

to the docker compose file.

Interestingly, the adding this actually lower the limit, before:

root@111c7e4eb5c0:/# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 256081
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1073741816
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@51e95bc16fed:/# nproc 
20

after:

root@51e95bc16fed:/# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 256081
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 20000
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 65535
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@111c7e4eb5c0:/# nproc 
20

The difference in on "open files" and "max user processes".

and thttpd actually engages far less memory while running:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                      
    226 nobody    20   0    8936   7028    800 S   0.0   0.0   0:00.13 thttpd  

I'm quite confused why this works... and also why thttpd allocates so much memory in that condition.

[simonmysun]

I would suggest mentioning it in your README. You may close the issue if you find it unnecessary.

[simonmysun]

Ahh I see the SYSTEM_HTTP_ULIMIT_N option when going through your source code. Sorry for missing it while reading README.

Could you please explain why the number 1048576 is chosen?

[Thom-x]

Yes, it was for fr24, if I remember it was the max value ?

[simonmysun]

I see.

What I figured out:

• thttpd allocates its connection table with the number from getrlimit (ref: https://github.com/jpouellet/thttpd/blob/0066035fc65d190186486a478c61f0b592e7224c/fdwatch.c#L718 ). If the result is -1, use 8192 (ref: https://github.com/jpouellet/thttpd/blob/0066035fc65d190186486a478c61f0b592e7224c/fdwatch.c#L162 ).
• However from some version of docker the nofile limit insides containers becomes 1073741816 and some other software users encountered similar problems (ref: https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2020q1/013821.html , Removed limit in Docker Engine 23 breaks existing config file haproxy/haproxy#2043 )
• thttpd tries to initiallize its connection table with size of 1073741816 and triggers OOM killer

I think the proper solution here is to set a lower limit. In this case I believe 1024 is far beyond enough.