Issue with Docker

[D-i-t-gh]

We run multiple instances of Mailpit in docker containers running 24/7 and we have been seeing no problems at all for a long time, which is great.
However, recently all these instances crashed at the same time and didn't come up again, resulting in permanent restarting of the Mailpit containers. Checking the logs, I could see the crash happened after the sqlite database was initialized. We use a database with persistent storage (MP_DATABASE), so I checked the docker volume. I checked file permissions, also free space on the corresponding partition and system RAM usage, which were perfectly fine. Other containers on the same system were running fine. So at this point I was a little lost and didn't know how to fix it.

[axllent]

This sounds to me like either an OS or a Docker issue, and not a Mailpit issue. Please read through the following questions - not all may apply, but you may see one or more that apply to your situation.

1. Any idea why did all the Docker containers initially crashed at the same time?
2. When these containers are restarting, what do the Mailpit logs say, and what does dmesg and/or system logs say? There must be some indication of an issue.
3. I assume when you say that you use a database with persistent storage in a volume, you mean a separate database/volume per container, right? Multiple Mailpit instances should never be sharing the same database.
4. Assuming each container has its own database, have you tried to get a copy of one of these databases out of the volume to inspect it?

[D-i-t-gh]

Thanks for your answer!

On the docker host some cleanup job had a leak, resulting in a full /tmp directory. After cleaning up /tmp, all Mailpit containers (all have separate database volumes) started up again. It seems SQLite needs /tmp for operations like "file locking", and it will fail if /tmp is full. Maybe there should be new option "MP_DATABASE_TMP"? Or at least a hint in the documentation that SQLite needs to be able to write files under /tmp in order to work.

[axllent]

Thanks for the feedback. I do not think that SQLite uses /tmp at all, unless of course the database is located in the /tmp directory. SQLite/Mailpit needs read/write access to the database itself, and alongside the database it creates a WAL file (related to the database) in the same folder. If Mailpit was not able to write to the database or WAL, then it should have produced an error in the Mailpit/Docker logs. In addition to this, if you're using Docker then the /tmp folder in the container is not the same as the /tmp on the host, but rather a directory most likely somewhere in /var/lib/docker/.

So my assumption here is that when your /tmp got full, there were other issues on your host which impacted Docker from starting. When harddrives get full all kinds of bad things happen, including (most likely) Docker itself failing to run as intended (which I think if far more likely).

[D-i-t-gh]

Thanks, you're right. In fact, Mailpit works perfectly in a read-only container. I tested it with this command:
docker run --rm --name=mailpit --read-only -v /mailpit-data:/data -e MP_DATABASE=/data/mailpit.db -e MP_VERBOSE=true -p 8025:8025 -p 1025:1025 axllent/mailpit:latest
So this means SQlite only needs to be able to write to the database volume.

I tried to fill up /tmp again on a test system, however, I couldn't reproduce the problem this time. I also now think it was a docker issue. Thanks for pointing me into the right direction.

[axllent]

Glad to help clarify where the problem was( or rather what it wasn't in this case), and thank you for the feedback too :)