container exits randomly with Code 137 (but without OOM problems)

[svenXY]

Hi,

I'm using the latest mariadb container with the DB directory in a volume in ~20 docker-hosts. It works flawlessly, except that very randomly, but sooner or later (several days and even weeks), containers exits with Exit Code 137. On the same systems are 3 docker containers with tomcat that do not exit and keep running.

So far, no memory reservations or limits were made for any of the containers, but the tomcat contaiers have decent Xmx and Xms settings that should leave more than enough memory for mariadb. Them being mostly dev and testing systems, they do not have a lot of traffic currently.
Futhermore, there are no memory or cpu spikes visible in monitoring.

I have already checked journalctl and the docker log for errors (especially since many posts mention OOM errors), but wasn't able to find anything that could give me any hint as to what goes wrong here.

Nothing in the docker logs and nothing in the systemd logs.

I'm using docker-compose v2.1 with a healthcheck. docker inspect on the exited container says "unhealthy", but the last 5 healthcheck all exited with 0, so this does not seem to be the problem either.

I'm out of ideas - do you have any suggestions what to change or how to get more information?

Thanks,
Sven

[tianon]

From moby/moby#21083 (comment), it appears the 137 is a SIGKILL -- are you sure you don't have something trying to do docker stop or even docker kill on this container/process?

The best I can suggest is perhaps disabling your HEALTHCHECK and see if you can still reproduce (to start narrowing it down).

In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow (since this is more of a request for support rather than a reproducible issue with the image that we could actionably fix given the data provided).

[advance512]

FYI for future readers, if your healthcheck isn't responding, an external orchestrator can stop the container, and you'll see code 137.