Stalwart keeps hanging, all of the listeners are unresponsive #2600
Locked
Turab
started this conversation in
Community Discussions
Replies: 10 comments 4 replies
|
Which entries appear in the logs before the process hangs. It will really help to see a trace of a hung process: $ gdb -p <PID>
set pagination off
set print thread-events off
thread apply all btSince the Stalwart binary does not include debugging symbols, it might be necessary to compile Stalwart locally if the trace you get does not provide enough information. |
0 replies
|
As I said, there's actually nothing relatable. Before the hanging, the logs were like this: All of them are my own IP address. Seems nothing unusual. And the journal was like: Strangely, after about 5 hours of struggling and opening this thread, it hanged two more times, then it never did again. I wrote a cron script to check the hanging status at every five minutes and log it. Since then, it has been up without hanging for hours now. I will try to debug with gdb per your instructions if I ever see this happen again. I am planning to make this a production setup for a handful of domains, so I hope it won't let me down. Thank you for your time and effort. |
0 replies
|
Funnily enough, just a few seconds after I posted this reply, it hanged again :D But this time, ports were answering and there were active connections, that's really weird. Here is the debugging output, which doesn't look promising: And here is the port test: within the machine itself They both stay like this like forever. Is there a honeypot in action or something? But even in that case, the service should have had shutdown gracefully; whereas it doesn't. It needed killing again, before it fell to timeout. Edit: I am connecting to SSH over OpenVPN, because this machine is behind a firewall (Endian). It uses 192.168.220.0 pool and 192.168.220.10 is the local IP of Stalwart instance. It had banned its own local IP in this particular case. But it still doesn't explain the hanging. Because: 1- Service did not respond to graceful shutdown, it had to be killed. |
1 reply
|
Is the debug going to be clearer if you use the musl version that doesn't dynamically link to your system libraries? |
|
Thanks for trace. Looking at your GDB backtrace, there are multiple tokio worker threads (threads 20-23) are all stuck in When multiple threads are blocked trying to close file descriptors, this could be due to file descriptor exhaustion. The process may have hit its file descriptor limit. Please check: # Check the process's file descriptor limit
cat /proc/16476/limits | grep "open files"
# Check current file descriptor usage
ls /proc/16476/fd | wc -l
# See what types of file descriptors are open
ls -la /proc/16476/fdIf you're close to the limit, try increasing it in your systemd service file or startup configuration. Also, since I can see RocksDB threads in your trace, and you're running on a system with potentially slower storage, this is the more likely culprit. RocksDB has known performance issues on slow disks. To verify this, try switching to PostgreSQL as your data store instead of RocksDB. PostgreSQL generally performs much better on rotational disks. If the problem persists even with PostgreSQL, we'll need a full stack trace with debugging symbols to diagnose further. To get a proper trace, you'll need to compile Stalwart locally with debug symbols. |
0 replies
|
I can't think of a reason why would I be close to the limits with a fresh installation. Could you help me understand? Checked the storage, this VPS is using a HDD. Nonetheless, I had already switched to SQLite after the first hanging, as I told in the first post. I don't know why I am still seeing RocksDB action. All DB options are set to SQLite on the web interface. Data, blob, search and internal accounts, are all selected to use SQLite. Here are the command results you asked: |
0 replies
Please run those commands again once it hangs. I suggested moving to PostgreSQL because other users reported that it had fixed the issue for them. If it doesn't then we'll need the full GDB trace with a locally compiled binary. |
0 replies
|
Can't move to PostgreSQL because I am having this error: I moved to SQLite from RocksDB by exporting/importing but I can't move to pgsql with the same method. I can't find information about this anywhere. I had 3 storage backends setup: RocksDB, SQLite, PostgreSQL, sqlite being the default for everything. I thought maybe it needed only one to be present to determine "which one to export". So I deleted pqsql and RocksDB, but the error did not resolve or even change. Edit: I ran it with |
0 replies
|
At the time of hanging: And this time there are some DB errors in the log: I believe some are references to bad actors. I also see some references to SQLite database, but I have fully migrated to PostgreSQL and I restarted a few times after that, so how and why? Yesterday it worked for hours, today I had to restart it 5-6 times. Sometimes it hanged seconds after the last restart. Something I still don't understand is that the ports are not unresponsive anymore when it hangs; but they act like a honeypot or something. Like it answers but never responds, stalling the client. (See the telnet logs in my previous post.) |
2 replies
|
According to the |
|
PgSQL log doesn't have any errors or warnings except the one caused by me, when I tried to run a manual query to see if the tables were populated after data import. I was sure that I already switched all stores to PgSQL but nonetheless I double checked, there isn't one I forgot. (There are 4 stores that could the selected, ain't it?) Speaking of which, I think the export/import procedure (and its documentation) must be clearer. Because multiple storage backends can be added; but it's not clear into which one the import procedure will do the import. For example all my stores were set to SQLite, then I made an export which I suppose exported everything from SQLite DB. Then I created a PgSQL store. Ran an import, it said it was successful. But I don't know into which store it actually imported. As seen in the logs, I was seeing connections to both SQLite and PgSQL. So I ran a query on the DB and saw that it wasn't imported into PgSQL. tried to guess a procedure on my own: (I think this is applicable only when switching all the stores at once.) This is the only way I could make it "really" migrate to PgSQL. The case of network or hardware: I think hardware should be fine because there are 9 VPSs on the same machine which have been running without a problem for a very long time. Could this be a network error? It could but I don't understand why would a network problem hang a process? Please also check the latest comment I made. |
|
There's something I am suspecting here: Looks like the server is becoming unresponsive when I connect to it from home. It never got unresponsive when I connect from work. I don't know if there is a correlation, I will test more. But this is too weird. Why would the application hang because of the client's connection parameters? I suspected this before; is there a default honeypot Stalwart is employing? It is the only reason comes to mind, which would keep me connected to the port (for example 25) without responding. But this doesn't still explain why the process would hang. |
1 reply
|
This error you are experiencing is highly unusual. If your system is not under attack, then this could be related to hardware/system issues. |
|
I think this might be related to how Stalwart handles rate-limiting. Now I am seeing in the logs that Stalwart applies rate-limiting to its own local IP. Two things to consider here:
I am mentioning this because there are other VDSs on the same dedicated server; all of which are communicating with each other with their local IPs. So I usually whitelist /24 subnet on each machine so that they wouldn't block or rate-limit each other or the firewall; to prevent a lock out. I did the same on Stalwart too; whitelisted this subnet which its own IP belonged. But something interesting here: why would it see its own local IP as "remote IP"? It would see other local IPs in the subnet as remote, but why its own too? I don't know why only some of the connections are logged with the local IP of Stalwart while most are logged with real client IPs. This rate-limiting would be the "honeypot" I supposed. The port is open; but never responding. Last night at home, I lost the connection to the web UI again. The process was also hung because debugging showed a handle was in closing state (just like before). But is it really hanging if the ports are still listening? This morning when I got to work, now the web UI is responsive again, without having the service restarted. On the other hand, if it's not hanging, then why wouldn't it gracefully shutdown when I restart it with systemctl? At normal times, systemctl restart is graceful; but at these times I think it hanged, it doesn't respond to graceful shutdown and systemd has to kill it after a timeout.
|
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I installed Stalwart a couple of hours ago on a fresh install Debian 13.
I setup the first domain, dns and everything. It worked for about one hour, then it hang. I needed to stop the service, which was also unsuccessful to a point where systemd had to kill it. Then I restarted and everything went back to normal, only for a couple of minutes. And after that, each restart helped it work only for one minute or similar.
I heard that it could be because of RocksDB in some cases so I successfully switch to SQLite. Then it worked for about 2 hours. Now it has hung again.
I set log level to trace, but I can't see anything relatable in the log and according to the log file, my IP is the only client. Resource is enough and it's not even consumed. journalct is empty, other than reporting the starting and stopping of the service.
So I can't even diagnose the problem. How to know what's happening here?
All reactions