-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
error sending message in campaign X: subscriber Y: timed out waiting for free conn in pool #734
Comments
If you're sending a large number of e-mails, your max. connections settings should most likely be in the 10s, or even 100s, depending on the capacity of your upstream SMTP. |
(Beyond configuration listmonk seems to have issues recovering from this kind of failure as it is unable to resume sending even after the SMTP Server has no load whatsoever as explained in my message). |
Your settings look fine. This behaviour seems to be at the network level then. Firewall/proxy/iptables/Docker network? listmonk just opens standard TCP connections to the SMTP server. If those connections are timing out where each message takes ~5 seconds, constantly keeping the pool locked so that it's never cleared (connections are cleared when they're idle for ~15 seconds), then it's something to do with the network. With slow connections, you should immediately see the max allowed 25 connections all attempting to send messages. Here, outgoing connections aren't even being established. |
This was my thinking also but as explained in my post there is no problem at the network level. I have spent a lot of time inspecting network traffic on various hosts, checking the internal state of firewalls, and so on... This is not where the problem is coming from. Moreover the problem has happened again and I can now 100% confirm that it is fixed by restarting the listmonk system process. So there is somehow an issue with connection management in listmonk. Apparently listmonk is unable to recover after there have been transient "timed out waiting for free conn in pool" errors. |
Version:
Description of the bug and steps to reproduce:
After a number of emails are sent (about 100,000 or a couple of 100,000s), while the campaign continues to run and the emails sent counter continues to increase, no email is actually sent and logs show the error message:
The default behavior of listmonk was working fine: because of this the campaign used to stop because too many such errors were produced but we have intentionally disabled this behavior because we have large subscriber lists and transient errors used to hit the threshold and suspend our campaigns too often.
I initially thought that the problem came from the SMTP Server and that maybe it exhausted its connection pool, however it turns out that when the problem happens the SMTP Server has no load and no incoming connection related to SMTP: listmonk's failure seems to happen before even contacting the SMTP Server.
I've therefore looked into increasing the size of Postgres connection pool assuming that the error could actually come from the database. However Postgres shows that it has no active connections from listmonk (sometimes 1 or 2 related to database requests made by the pages of listmonk's dashboard) so this is unlikely to be the issue.
The problem seem to go away on its own after waiting and trying to restart the campaign a few times. To clarify pausing and restarting the campaign does not help, but it looks like as some point, (a couple minutes - but this can be way more than 10 minutes I feel) the campaign works again when it is restarted.
For example, when I decided to seriously look into solving this recurring problem yesterday over the course of maybe 30 minutes to hour, maybe even more, I've tried to restart the campaign without success (the campaign does restart on listmonk but just produces the aforementioned error message in the log). So it can actually last for quite a while. Letting the campaign run does not make the problem disappear on its own either (when it happens this error is not transient).
Restarting listmonk through the dashboard (by changing some settings and clicking the restart button) does not solve the issue. However as I was typing this message, I noticed that this action does not actually restart the main listmonk process (I could see it had an uptime of 31 days days), so I just tried restarting the system process and the campaign is now working again.
I will need to confirm that this fix works by trying to restart the system process if the problem happens again. The issue does not seem to come from the environment since all systems indicators are fine (number of connections etc...). The environment was my main assumption since listmonk is in a container behind a nat, and the database is on a different virtual host etc... but after digging I haven't been able to find any exhaustion issue in the environment so it looks like an internal problem with listmonk.
The text was updated successfully, but these errors were encountered: