Container is Unhealthy After a Few Days

[fender324]

Describe the bug

I'm running v5.1.7 of the omada controller on a raspberry pi 3. The container will run fine for a few days, but eventually it becomes unhealthy.

To Reproduce
Steps to reproduce the behavior:

1. Start the container
2. Wait a few days
3. Container reports unhealthy and can no longer connect
4. A restart of the container fixes the issue.

Expected behavior

The container to run without an error after a few days.

How you're launching your container

I'm running portainer, but I started the container initial from the command line using the example usage from the readme for armv7i.

docker run -d
--name omada-controller
--restart unless-stopped
-p 8088:8088
-p 8043:8043
-p 8843:8843
-p 29810:29810/udp
-p 29811:29811
-p 29812:29812
-p 29813:29813
-p 29814:29814
-e MANAGE_HTTP_PORT=8088
-e MANAGE_HTTPS_PORT=8043
-e PORTAL_HTTP_PORT=8088
-e PORTAL_HTTPS_PORT=8843
-e SHOW_SERVER_LOGS=true
-e SHOW_MONGODB_LOGS=false
-e SSL_CERT_NAME="tls.crt"
-e SSL_KEY_NAME="tls.key"
-e TZ=Etc/UTC
-v omada-data:/opt/tplink/EAPController/data
-v omada-work:/opt/tplink/EAPController/work
-v omada-logs:/opt/tplink/EAPController/logs
mbentley/omada-controller:5.1-armv7l

Container Logs

The full log is attached, but below is the log right before it stopped working (a couple of days ago).

04-06-2022 22:10:14.301 INFO [check-update-work-group-0] [] c.t.s.o.m.b.b(95): Checking Device Last Firmware...
04-06-2022 22:10:15.481 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(128): Checking last firmwares of managed devices succeed.
tail: '/opt/tplink/EAPController/logs/server.log' has become inaccessible: No such file or directory
tail: '/opt/tplink/EAPController/logs/server.log' has appeared;  following new file
04-07-2022 00:39:20.779 WARN [quartzScheduler_Worker-2] [] j.u.c.ThreadPoolExecutor$DiscardPolicy(113): FaceBookPeriodVerify Schedule ThreadPool is full, Discarding task.
04-07-2022 17:43:
45.054 INFO [maintenance-history-data-retention-283] [] c.t.s.o.m.d.c.a(40): clear omadac data with omadacId [dda303cad95d0aa6bcdcba4d6f23e64e]
04-07-2022 17:43:45.165 INFO [manager-data-retention-pool-0] [] c.t.s.c.u.l.LocalUpgradeFileCache(147): delete expired upgrade file, number = 0
04-07-2022 19:32:20.595 WARN [quartzScheduler_Worker-2] [] j.u.c.ThreadPoolExecutor$DiscardPolicy(113): FaceBookPeriodVerify Schedule ThreadPool is full, Discarding task.
04-07-2022 22:10:14.241 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(164): Cloud Access Not Enable, cancel sync cloud user.
04-07-2022 22:10:14.281 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(118): Start checking last firmwares of managed devices.
04-07-2022 22:10:14.282 INFO [check-update-work-group-0] [] c.t.s.o.m.b.b(95): Checking Device Last Firmware...
04-07-2022 22:10:15.415 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(128): Checking last firmwares of managed devices succeed.
tail: '/opt/tplink/EAPController/logs/server.log' has been replaced;  following new file
04-08-2022 07:39:05.958 WARN [quartzScheduler_Worker-1] [] j.u.c.ThreadPoolExecutor$DiscardPolicy(113): FaceBookPeriodVerify Schedule ThreadPool is full, Discarding task.
04-08-2022 07:39:07.664 INFO [quartzScheduler_Worker-1] [] c.t.s.o.c.b.c(61): cloud schedule queue is full, discard.
04-08-2022 17:43:45.056 INFO [maintenance-history-data-retention-307] [] c.t.s.o.m.d.c.a(40): clear omadac data with omadacId [dda303cad95d0aa6bcdcba4d6f23e64e]
04-08-2022 17:43:45.159 INFO [manager-data-retention-pool-0] [] c.t.s.c.u.l.LocalUpgradeFileCache(147): delete expired upgrade file, number = 0
04-08-2022 22:10:14.244 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(164): Cloud Access Not Enable, cancel sync cloud user.
04-08-2022 22:10:14.285 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(118): Start checking last firmwares of managed devices.
04-08-2022 22:10:14.286 INFO [check-update-work-group-0] [] c.t.s.o.m.b.b(95): Checking Device Last Firmware...
04-08-2022 22:10:15.510 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(128): Checking last firmwares of managed devices succeed.
tail: '/opt/tplink/EAPController/logs/server.log' has been replaced;  following new file
04-09-2022 17:43:45.049 INFO [maintenance-history-data-retention-331] [] c.t.s.o.m.d.c.a(40): clear omadac data with omadacId [dda303cad95d0aa6bcdcba4d6f23e64e]
04-09-2022 17:43:45.171 INFO [manager-data-retention-pool-0] [] c.t.s.c.u.l.LocalUpgradeFileCache(147): delete expired upgrade file, number = 0
04-09-2022 21:54:54.462 INFO [https-jsse-nio-8043-exec-3] [] c.t.s.o.c.d.a.l(210): connect failure list is empty
04-09-2022 21:55:08.301 INFO [https-jsse-nio-8043-exec-7] [] c.t.s.o.c.d.a.l(210): connect failure list is empty
04-09-2022 22:10:14.254 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(164): Cloud Access Not Enable, cancel sync cloud user.
04-09-2022 22:10:14.286 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(118): Start checking last firmwares of managed devices.
04-09-2022 22:10:14.288 INFO [check-update-work-group-0] [] c.t.s.o.m.b.b(95): Checking Device Last Firmware...
04-09-2022 22:10:15.540 INFO [check-update-work-group-0] [] c.t.s.o.m.b.e(128): Checking last firmwares of managed devices succeed.
04-09-2022 22:14:57.339 WARN [https-jsse-nio-8043-exec-6] [] o.s.w.s.m.s.DefaultHandlerExceptionResolver(207): Resolved [org.springframework.http.converter.HttpMessageNotWritableException: No converter for [class java.util.LinkedHashMap] with preset Content-Type 'application/javascript;charset=UTF-8']
04-10-2022 04:07:56.083 WARN [SessionValidationThread-1] [] c.t.s.o.i.p.c.a.c(69): omadacId=dda303cad95d0aa6bcdcba4d6f23e64e deleteSessionById sessionId=7f0edc7f12514b4dae0c4751dd604560 not exist.
tail: '/opt/tplink/EAPController/logs/server.log' has become inaccessible: No such file or directory
tail: '/opt/tplink/EAPController/logs/server.log' has appeared;  following new file
04-10-2022 14:20:01.642 WARN [quartzScheduler_Worker-1] [] j.u.c.ThreadPoolExecutor$DiscardPolicy(113): FaceBookPeriodVerify Schedule ThreadPool is full, Discarding task.
04-10-2022 14:23:10.875 WARN [quartzScheduler_Worker-1] [] j.u.c.ThreadPoolExecutor$DiscardPolicy(113): FaceBookPeriodVerify Schedule ThreadPool is full, Discarding task.

Additional context

output.log

[mbentley]

Can you grab the mongodb logs?

If you aren't sure how to do that, this'll work:
docker exec -it omada-controller cat /opt/tplink/EAPController/logs/mongod.log

I can only guess that you're hitting the same thing as reported in #197

[fender324]

I'm not seeing an out of memory error in the mongod.log file, but I've attached the whole thing just in case.
mongod.log

[mbentley]

Hmm, yeah, doesn't seem like a #197 after all. Are you seeing any excessive CPU or memory pressure when the controller becomes unresponsive? If you're not really sure, try out something like running top once, trimming the output to like 30 lines in case there are a lot of processes:

# top -bn1 | head -n30

The ThreadPool is full bit makes me wonder what exactly is going on as I don't recall ever seeing that but it could also be a red herring since it's just mentioned regarding FaceBookPeriodVerify Schedule.

[fender324]

The container is still healthy, but I ran the top command you provided. Running it multiple times I see java and mongod pop in and out of the list. It seems odd that homebridge is always using a fairly high amount of CPU, so I'll look into that. When the container becomes unhealthy again, I'll post an update with the top output.

top - 20:30:24 up 19 days,  5:26,  3 users,  load average: 0.63, 0.68, 0.64
Tasks: 192 total,   1 running, 191 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.8 us,  3.9 sy,  0.0 ni, 80.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    923.2 total,     30.5 free,    726.9 used,    165.8 buff/cache
MiB Swap:    100.0 total,      0.0 free,    100.0 used.     89.7 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2382 homebri+  20   0  190132  52344   4300 S  41.2   5.5   5614:32 homebridge
17622 pi        20   0   10292   2860   2460 R  11.8   0.3   0:00.04 top
20541 508       20   0 1412772 333452      0 S  11.8  35.3 109:34.51 java
   22 root      20   0       0      0      0 S   5.9   0.0   1:11.33 ksoftirqd/2
    1 root      20   0   34984   4512   2980 S   0.0   0.5  24:53.19 systemd
    2 root      20   0       0      0      0 S   0.0   0.0   0:06.10 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
   10 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
   11 root      20   0       0      0      0 S   0.0   0.0   4:43.40 ksoftirqd/0
   12 root      20   0       0      0      0 I   0.0   0.0  12:42.28 rcu_sched
   13 root      rt   0       0      0      0 S   0.0   0.0   0:00.80 migration/0
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
   16 root      rt   0       0      0      0 S   0.0   0.0   0:00.65 migration/1
   17 root      20   0       0      0      0 S   0.0   0.0   0:59.73 ksoftirqd/1
   20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2
   21 root      rt   0       0      0      0 S   0.0   0.0   0:00.63 migration/2
   25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3
   26 root      rt   0       0      0      0 S   0.0   0.0   0:00.57 migration/3
   27 root      20   0       0      0      0 S   0.0   0.0   2:32.53 ksoftirqd/3

[fender324]

The container went unhealthy again, but this time it looks like there are some errors in the logs. I've uploaded both server.log and mongod.log, but let me know if there is anything else I can provide.

mongod3.log
server3.log

[mbentley]

Hmm, it's hard to say exactly what is going on - all I can tell is that the controller app SpringBoot app can't talk to the MongoDB; or at least the MongoDB isn't responding. Any resource pressure when you were seeing an error? When you're seeing the controller unresponsive, it would be good to get the top output. It might also be a good idea to look in /var/log/syslog to see if there are any other errors showing up at the system level while the app is having issues.

Also just to give me an idea of the ports listening when you're in a good state now or later when it is unhealthy, also provide the output of:
docker exec -it omada-controller netstat -ltnpu

Then also grab the stats now and then when it is in a bad state:
docker stats --no-stream omada-controller

Just out of curiosity, which Raspberry Pi do you have? Looks like it has 1 GB of memory from the top output. With the issues that I am seeing people have with a Raspberry Pi, I might have to try to setup a Raspberry Pi to see if I can reproduce some of the issues people are hitting. I am thinking it is possible that the resources aren't sufficient and that the controller's default settings aren't ideal for something like a Raspberry Pi.

One thing that might be helpful would be to try to re-run the container but with reduced resources by decreasing the Xmx from 1024 to 512 by overriding the CMD:

docker run -d
--name omada-controller
--restart unless-stopped
-p 8088:8088
-p 8043:8043
-p 8843:8843
-p 29810:29810/udp
-p 29811:29811
-p 29812:29812
-p 29813:29813
-p 29814:29814
-e MANAGE_HTTP_PORT=8088
-e MANAGE_HTTPS_PORT=8043
-e PORTAL_HTTP_PORT=8088
-e PORTAL_HTTPS_PORT=8843
-e SHOW_SERVER_LOGS=true
-e SHOW_MONGODB_LOGS=false
-e SSL_CERT_NAME="tls.crt"
-e SSL_KEY_NAME="tls.key"
-e TZ=Etc/UTC
-v omada-data:/opt/tplink/EAPController/data
-v omada-work:/opt/tplink/EAPController/work
-v omada-logs:/opt/tplink/EAPController/logs
mbentley/omada-controller:5.1-armv7l \
/usr/bin/java -server -Xms128m -Xmx512m -XX:MaxHeapFreeRatio=60 -XX:MinHeapFreeRatio=30 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/tplink/EAPController/logs/java_heapdump.hprof -Djava.awt.headless=true -cp /opt/tplink/EAPController/lib/*::/opt/tplink/EAPController/properties: com.tplink.smb.omada.starter.OmadaLinuxMain

I'm just wondering if the JVM is using too much memory and not leaving enough for MongoDB or maybe even that the Omada Controller + Home Bridge is too much for the pi. The top and docker stats output should help me get a better idea though so it might be a good idea to investigate, pull those outputs for me and then later try to re-create the container.

[fender324]

I did take a look at top when I first noticed that the container was unhealthy, and I didn't notice anything unusual (but I didn't copy the status).

Here is the output of the netstat command when the container is healthy.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8843            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8043            0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:27217         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:29811           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:29812           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:29813           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:29814           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8088            0.0.0.0:*               LISTEN      -
udp        0      0 0.0.0.0:29810           0.0.0.0:*                           -
udp        0      0 0.0.0.0:27001           0.0.0.0:*                           -

And the output of stats when the container is healthy

CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
35f187898715   omada-controller   5.37%     0B / 0B             0.00%     12MB / 1.48MB   0B / 0B     248

This is running on a Raspberry Pi 3b with 1 GB of RAM. I'll try the new command after the container goes unhealthy again. I did disable a bunch of stuff in homebridge, so top now looks like

top - 13:08:21 up 20 days, 22:04,  3 users,  load average: 0.17, 0.20, 0.14
Tasks: 192 total,   1 running, 191 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.3 us,  5.7 sy,  0.0 ni, 92.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    923.2 total,     25.1 free,    707.8 used,    190.3 buff/cache
MiB Swap:    100.0 total,      0.0 free,    100.0 used.    110.1 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
28835 pi        20   0   10292   2876   2476 R  22.2   0.3   0:00.09 top
 4657 508       20   0 1411160 330208   2160 S   5.6  34.9  48:16.27 java
    1 root      20   0   34984   4036   2504 S   0.0   0.4  29:54.91 systemd
    2 root      20   0       0      0      0 S   0.0   0.0   0:07.04 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
   10 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
   11 root      20   0       0      0      0 S   0.0   0.0   5:11.66 ksoftirqd/0
   12 root      20   0       0      0      0 I   0.0   0.0  13:43.59 rcu_sched
   13 root      rt   0       0      0      0 S   0.0   0.0   0:00.86 migration/0
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
   16 root      rt   0       0      0      0 S   0.0   0.0   0:00.69 migration/1
   17 root      20   0       0      0      0 S   0.0   0.0   1:03.44 ksoftirqd/1
   20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2
   21 root      rt   0       0      0      0 S   0.0   0.0   0:00.67 migration/2
   22 root      20   0       0      0      0 S   0.0   0.0   1:14.85 ksoftirqd/2
   25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3
   26 root      rt   0       0      0      0 S   0.0   0.0   0:00.61 migration/3
   27 root      20   0       0      0      0 S   0.0   0.0   3:35.04 ksoftirqd/3
   30 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kdevtmpfs

[mbentley]

Thanks for the update. I will just mention that it looks like the memory cgroup isn't enabled which is why docker stats isn't reporting memory usage most likely:
https://forums.raspberrypi.com/viewtopic.php?p=1262431&sid=afae60666a4ab3df16197e46f1806564#p1262431

Not a huge deal but docker stats won't provide useful memory info. It's nothing that is contributing to any issues you're seeing but I might suggest updating your /boot/cmdline.txt for the next time you do a reboot.

This can also be confirmed with docker info as it will output warnings:

WARNING: No memory limit support
WARNING: No swap limit support

[fender324]

I edited /boot/cmdline.txt per that link, and now docker stats is providing more information

CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O   PIDS
35f187898715   omada-controller   6.83%     379.9MiB / 923.2MiB   41.15%    104kB / 41.8kB   0B / 0B     230

I'll provide another update when the container goes unhealthy again.

[drew-hill]

@fender324 How is this going? My 5.1.7 container running on an RPi B Rev 1.2 (1GB memory, 600mhz quad core arm64) is hitting some stability issues. If I use the Omada Controller web gui too much, it begins to slow and eventually the RPi becomes unresponsive and I need to reboot. I am wondering if it's a similar issue. I have enabled group options and relaunched the container using the 512Mb limits with fingers crossed.

[drew-hill]

It may very well be a memory problem, as implementing the following swap recommendations seems to have made a major difference for me: https://ikarus.sg/using-zram-to-get-more-out-of-your-raspberry-pi/.

I'm unsure of the longterm effects of using your SD card for swapping via ram (I imagine it shortens the life of the sd card at least to some extent), but the fact that it seems to be helping my distribution substantially suggests it's a ram issue

Perhaps the newer 5.x.x controller software is more RAM intensive than previous versions.

Here are resource use stats while running the docker container, and nothing else

And, if useful, here are resource stats while running the Omada controller as installed from scratch, not using the container (though a small amount of resources still seem to be allocated to Docker, like containers)

[fender324]

@drew-hill The container has been running for the past 11 days and is still reporting healthy. I haven't made any changes that would have improved the up time, but I don't think I've used the web interface since the container was restarted. I also hadn't noticed any slow downs with the web interface or app. For me, it either works or doesn't work. I'll try using the web interface some to see if usage triggers the problem.

[fender324]

The container finally went unhealthy. I was still running the v5.1.7 version and was using the default command with the -Xmx1024m option.

The top output while the container is unhealthy.

top - 16:20:15 up 36 days, 14:56,  3 users,  load average: 0.06, 0.08, 0.09
Tasks: 184 total,   1 running, 183 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.7 us,  2.3 sy,  0.0 ni, 93.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    923.2 total,     82.4 free,    406.0 used,    434.8 buff/cache
MiB Swap:    100.0 total,      0.4 free,     99.6 used.    409.0 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
20452 pi        20   0   10292   2888   2488 R  10.5   0.3   0:00.07 top
    1 root      20   0   33868   5076   3708 S   0.0   0.5  39:48.85 systemd
    2 root      20   0       0      0      0 S   0.0   0.0   0:10.76 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
    8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
    9 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
   10 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
   11 root      20   0       0      0      0 S   0.0   0.0  11:08.21 ksoftirqd/0
   12 root      20   0       0      0      0 I   0.0   0.0  25:04.44 rcu_sched
   13 root      rt   0       0      0      0 S   0.0   0.0   0:01.29 migration/0
   14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
   15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
   16 root      rt   0       0      0      0 S   0.0   0.0   0:01.17 migration/1
   17 root      20   0       0      0      0 S   0.0   0.0   4:29.84 ksoftirqd/1
   20 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/2
   21 root      rt   0       0      0      0 S   0.0   0.0   0:01.10 migration/2
   22 root      20   0       0      0      0 S   0.0   0.0   3:15.71 ksoftirqd/2
   25 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/3
   26 root      rt   0       0      0      0 S   0.0   0.0   0:00.98 migration/3
   27 root      20   0       0      0      0 S   0.0   0.0   4:16.55 ksoftirqd/3
   30 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kdevtmpfs
   31 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns

Docker stats output

CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O   PIDS
35f187898715   omada-controller   0.00%     72KiB / 923.2MiB    0.01%     138MB / 72.4MB   0B / 0B     0

When trying to run netstat in the container

cannot exec in a stopped state: unknown

Attached are mongod.log, server.log, and docker log for the container.

Let me know if there is anything else I can provide.

server_5_24.log
output_5_24.log
mongod_5_24.zip

[mbentley]

Do you have any approximate date/time stamp I can look for in the logs? Flipping through, I only see things like Schedule ThreadPool is full, Discarding task; nothing else to really indicate what is going on. Only thing I can recommend is to try to reduce the Xmx value at this point.

[antcilli]

Been running into this as well with controller version 5.1.7 on a RPi 3B with Debian 11.3.
Just running this with Portainer on the Pi and nothing else. I'm in the process of updating the controller to 5.3.1 and Portainer from 2.11.1 to 2.13.1. I was debating just scheduling the container to restart every 2 days, and then I ran across this open issue. I'll report back if there's any improvement with the later versions.