Container crashing without any apparent reason #107

fl0wfr · 2023-01-21T16:02:04Z

Specifications

Image hash or version: PR-106 and latest
Platform: amd64
Subsystem:

Expected Behavior

Simply start and feed FR24/PiAware

Actual Behavior

Crash loop without any error

Steps to Reproduce the Problem

Start the container with this command: docker run -d -p 8080:8080 -p 8754:8754
--device=/dev/bus/usb:/dev/bus/usb
-v "/etc/localtime:/etc/localtime:ro"
-e "FR24FEED_FR24KEY=<>"
-e "PIAWARE_FEEDER_DASH_ID=<>"
-e "HTML_SITE_LAT=<>"
-e "HTML_SITE_LON=<>"
-e "HTML_SITE_NAME=<>"
-e "SERVICE_ENABLE_ADSBEXCHANGE=false"
-e "MLAT_EXACT_LAT=<>"
-e "MLAT_EXACT_LON=<>"
-e "MLAT_ALTITUDE_MSL_METERS=<>"
-e "SERVICE_ENABLE_PLANEFINDER=false"
-e "TZ=Etc/GMT+0"
--tmpfs /run:exec,size=32M
--tmpfs /planefinder/log:exec,size=32M
--tmpfs /usr/lib/fr24/public_html/data:size=32M
--restart=unless-stopped
--network=myNetwork
--name fr24feed
thomx/fr24feed-piaware:PR-106
USB Tuner is: Bus 002 Device 003: ID 0bda:2838 Realtek Semiconductor Corp. RTL2838 DVB-T

I've attached some logs.
fr24feed.log

Thom-x · 2023-01-23T08:59:01Z

Could you try disabling services one by one to see the culprit ?

fl0wfr · 2023-01-23T09:04:16Z

Hi Thom-x, thanks for your reply
Seems to come from SERVICE_ENABLE_HTTP, no issue with :
SERVICE_ENABLE_DUMP1090
SERVICE_ENABLE_PIAWARE
SERVICE_ENABLE_FR24FEED
set to true

Thom-x · 2023-01-23T09:07:09Z

Ok, at least everything else is OK.
Could you try to start manually the HTTP server by executing the same command ?

docker-fr24feed-piaware-dump1090/root/etc/s6-overlay/s6-rc.d/http/run

Line 5 in 1afb9b3

    
           /thttpd -D -h 0.0.0.0 -p 8080 -d /usr/lib/fr24/public_html -l - -M 60 2>&1 | mawk -W interactive '{printf "%c[32m[http]%c[0m %s\n", 27, 27, $0}'

fl0wfr · 2023-01-23T09:10:37Z

No return of the command, I don't have ps in the image so I can't check if it's running, but I doubt it runs. Also, no message in logs following the start of thttpd. And no access to the page with the published port.

Thom-x · 2023-01-23T09:14:06Z

Probably not working (it should be blocking). You could check the exit code $?.
Could you change the port by a random one -p 4242?

fl0wfr · 2023-01-23T09:21:23Z

Same issue with port 4242, and I get a "1" exit code.

Thom-x · 2023-01-23T09:30:40Z

Strange, I would suggest changing path, ports, etc to see what's wrong.
I'm not sure if I can help more. 😕

fl0wfr · 2023-01-23T09:32:57Z

Okay, seems strange, I just popped the container on a new host, no persistent storage, basic options. In the meantime I will run without the WebUI.
Thanks anyway!

Thom-x · 2023-01-23T09:42:01Z

Sorry for that :/

fl0wfr · 2023-01-23T09:47:03Z

No worries, you already did a great work, thanks!

rpavlyuk · 2023-02-09T16:15:38Z

I can confirm the same problem: as soon as you enable SERVICE_ENABLE_HTTP -- it crashes. Starting thttpd inside the container crashes it as well.

@Thom-x, if you can provide any instructions on how and where to check thttpd logs (I'm RedHat guy, not Debian :)) so I can troubleshot the issue.

Thom-x · 2023-02-14T11:54:47Z

@Thom-x, if you can provide any instructions on how and where to check thttpd logs (I'm RedHat guy, not Debian :)) so I can troubleshot the issue.

thttpd is logging on stdout (no daemon), what strange is that I'm also using it on amd64 without error.

Littlejth · 2023-03-01T20:15:38Z

This is also occurring for me on my PI Zero 2 W now. Host OS is Manjaro ARM aarch64. I can do any troubleshooting steps that might be needed to get to the bottom of this :) Only thing common I can see with fl0wfr is the same RTL2838 SDR but not sure how that could affect thttpd.

Thom-x · 2023-03-02T10:54:53Z

@Littlejth Did you try starting the container with only HTTPD enabled ?

Littlejth · 2023-03-02T22:43:18Z

With only the HTTP service enabled, it still just starts and stops again. Attached is the log file of a single time I attempted to start it after creating the container with only HTTP enabled.

fr24feed-piaware.log

Thom-x · 2023-03-03T18:57:45Z

I created a branch to switch from thttpd to httpd, I'll let you now when the image is available so you can test and give me the feedback #114.

Thom-x · 2023-03-04T11:20:40Z

You can test it with docker pull thomx/fr24feed-piaware:PR-114

fl0wfr · 2023-03-04T16:20:42Z

Hello Thom-x,
Same issue, and I see this message in logs:
[http] /httpd-foreground: 7: exec: httpd: not found

Thom-x · 2023-03-04T16:24:56Z

What is the content of /usr/local/apache2/bin it's probably not the same between different arch ?

fl0wfr · 2023-03-04T16:32:26Z

Thom-x · 2023-03-04T16:36:13Z

Ok funny, it's probably not compatible for all arch...
I need to find another candidate 😄.

Littlejth · 2023-03-08T17:25:34Z

It did start working on my aarch64 pi but obviously not a great solution if it doesn't work for AMD64 as well :)

MaxWinterstein · 2023-04-20T12:51:17Z

@fl0wfr which docker version are you running? v23?

Users of my Home Assistant Add-On raised the same issue: MaxWinterstein/homeassistant-addons#149

I guess, thttpd does what is explained here: https://developers.home-assistant.io/blog/2023/04/13/new_limits_for_add_ons/

With Home Assistant OS 10, we update to the latest Docker release 23.0. With the new Docker version the maximum number of open file descriptors for add-ons changed to infinity (sort of).
...
Also in all three cases, memory got allocated dynamically depending on the number of open file descriptors allowed (which can be determined via prlimit64 syscall, which returns a limit of 1073741816).

fl0wfr · 2023-04-20T13:02:57Z

@MaxWinterstein yes I'm using Docker 23.0.4

MaxWinterstein · 2023-04-20T13:16:24Z

@fl0wfr no linux memory expert here, but can you give the ulimit idea a try?

https://docs.docker.com/engine/reference/commandline/run/#ulimit

Something like

docker run --ulimit nofile=1048576 ......

might be worth a shot

fl0wfr · 2023-04-20T13:31:04Z

@MaxWinterstein it works now. That should do the trick for now, thanks.

MaxWinterstein · 2023-04-20T13:37:25Z

@Thom-x this might hit more and more people who switch to docker 23

I created #118 as quickfix, but switching to some more recent webserver might be a better approach ;)

Thom-x · 2023-04-20T13:42:45Z

Thanks, I'll do a PR image for testing quickly and release of it's all good 👍 !

MaxWinterstein · 2023-04-20T13:55:17Z

Thanks, I'll do a PR image for testing quickly and release of it's all good 👍 !

Awesome! Can't really test it myself here, hope it works 🤞

fl0wfr · 2023-04-20T13:58:53Z

You let me know and I will test it ;)

CoMPaTech · 2023-04-20T14:09:48Z

I still ran into issues with Max' update on his HA-addon. Still OOM errors on thttp. See screenshot in MaxWinterstein/homeassistant-addons#149 (comment) if helpful

MaxWinterstein · 2023-04-20T14:21:15Z

I still ran into issues with Max' update on his HA-addon. Still OOM errors on thttp. See screenshot in MaxWinterstein/homeassistant-addons#149 (comment) if helpful

I guess the fix in my addon is placed wrongly. I think s6 starts the httpd stuff in another shell/session/thread/... where my adjustment is not read.

#118 is made a little bit different

Thom-x · 2023-04-25T19:18:30Z

You can try with the image thomx/fr24feed-piaware:PR-118

fl0wfr · 2023-04-26T09:17:04Z

It works, thank you :)

Thom-x · 2023-04-26T09:41:30Z

Nice 😄 let's do a release then.

Thom-x · 2023-04-26T10:55:03Z

It's now available with 1.21.0 or latest 👍

TheDom42 · 2023-04-27T07:58:45Z

It's now available with 1.21.0 or latest 👍

Running Unraid 6.11.5 (latest stable version), which has Docker v20.10.21.

At least on my system, this update to 1.21.0 does not seem to work. After I update, my (otherwise so far completely stable) container stops automatically without any error logged. This happens a few seconds after start of the container.

Downgrading back to v1.20.0 fixes the issue, so I would assume, that this is related to this issue? If I have time later on, I can try the dedicated PR tag/image, if that gives a clearer picture.

MaxWinterstein · 2023-04-27T11:53:58Z

It works, thank you :)

@fl0wfr did you see any difference with the cpu usage?

fl0wfr · 2023-04-27T13:22:52Z

It works, thank you :)

@fl0wfr did you see any difference with the cpu usage?

No, around 3 and 7% CPU usage, running on a Celeron N5105.

Thom-x linked a pull request Mar 3, 2023 that will close this issue

chore(http): change from thttpd to httpd #114

Closed

MaxWinterstein mentioned this issue Apr 20, 2023

Enforce ulimit to prevent possible OOM issues #118

Merged

MaxWinterstein mentioned this issue Apr 20, 2023

ADS-B Feeder failing after updating to Home Assistant Operating System OS 10.0 MaxWinterstein/homeassistant-addons#149

Closed

Thom-x closed this as completed Apr 26, 2023

MaxWinterstein mentioned this issue Jun 6, 2023

Possible CPU consumption issue with FR24Feed and no ulimit set #126

Closed

fl0wfr mentioned this issue Mar 8, 2024

Services in container keeps stopping #162

Closed

simonmysun mentioned this issue May 3, 2024

HTTP fails and stops the container #169

Closed

Container crashing without any apparent reason #107

Container crashing without any apparent reason #107

Comments

fl0wfr commented Jan 21, 2023 • edited Loading

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Thom-x commented Jan 23, 2023

fl0wfr commented Jan 23, 2023

Thom-x commented Jan 23, 2023

fl0wfr commented Jan 23, 2023

Thom-x commented Jan 23, 2023 • edited Loading

fl0wfr commented Jan 23, 2023

Thom-x commented Jan 23, 2023

fl0wfr commented Jan 23, 2023

Thom-x commented Jan 23, 2023

fl0wfr commented Jan 23, 2023

rpavlyuk commented Feb 9, 2023

Thom-x commented Feb 14, 2023 • edited Loading

Littlejth commented Mar 1, 2023

Thom-x commented Mar 2, 2023

Littlejth commented Mar 2, 2023

Thom-x commented Mar 3, 2023

Thom-x commented Mar 4, 2023

fl0wfr commented Mar 4, 2023

Thom-x commented Mar 4, 2023

fl0wfr commented Mar 4, 2023

Thom-x commented Mar 4, 2023

Littlejth commented Mar 8, 2023

MaxWinterstein commented Apr 20, 2023

fl0wfr commented Apr 20, 2023

MaxWinterstein commented Apr 20, 2023 • edited Loading

fl0wfr commented Apr 20, 2023

MaxWinterstein commented Apr 20, 2023

Thom-x commented Apr 20, 2023

MaxWinterstein commented Apr 20, 2023

fl0wfr commented Apr 20, 2023

CoMPaTech commented Apr 20, 2023

MaxWinterstein commented Apr 20, 2023

Thom-x commented Apr 25, 2023

fl0wfr commented Apr 26, 2023

Thom-x commented Apr 26, 2023

Thom-x commented Apr 26, 2023

TheDom42 commented Apr 27, 2023

MaxWinterstein commented Apr 27, 2023

fl0wfr commented Apr 27, 2023

fl0wfr commented Jan 21, 2023 •

edited

Loading

Thom-x commented Jan 23, 2023 •

edited

Loading

Thom-x commented Feb 14, 2023 •

edited

Loading

MaxWinterstein commented Apr 20, 2023 •

edited

Loading