Pending emails

[freescout-helpdesk]

Hi. Having the same issue as #506

postal status is OK, but messages are just marked as "pending" and not being sent for 2 days. There are no any error messages in logs.

What can be the reason and how to force postal to send them?

[freescout-helpdesk]

service postal restart did help, it took around 3 minutes to restart:

Nov 05 19:30:59 mail systemd[1]: Stopped Postal Mail Platform.
Nov 05 19:30:59 mail systemd[1]: postal.service: Unit entered failed state.
Nov 05 19:30:59 mail systemd[1]: postal.service: Failed with result 'timeout'.
Nov 05 19:30:59 mail systemd[1]: Starting Postal Mail Platform...
Nov 05 19:31:00 mail postal[32667]: Removed worker.1.pid because process was not active
Nov 05 19:31:00 mail postal[32667]: Started Procodile supervisor with PID 32680
Nov 05 19:31:00 mail systemd[1]: Started Postal Mail Platform.

Maybe there is some way to check Postal state via cron and restart it when is stucks?

Is there any way to get number of pending emails from console?

[willpower232]

Can you run postal status to verify everything is working?

Also can you monitor your server CPU and memory usage? It may be you need more of one or the other.

[jos0405]

Hey, just an idea: check if your rabbitmq is running and postal is connected.
Just happened to me, that rabbitmq connection got messed up, and had to reinstall it.
(I know 'messed up' and 'connected' are not the best words, but describes the issue.)

[hadifarnoud]

@jos0405 I think I have same issue.

root@smtp-server ~ # sudo rabbitmqctl start_app
Starting node 'rabbit@smtp-server'
Error: unable to connect to node 'rabbit@smtp-server': nodedown

DIAGNOSTICS
===========

attempted to contact: ['rabbit@smtp-server']

rabbit@smtp-server:
  * connected to epmd (port 4369) on smtp-server
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on smtp-server
  * suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-34@smtp-server'
- home dir: /var/lib/rabbitmq
- cookie hash: brz5PuVVscnxRss3AaoTRw==

[ahmedaldabbagh]

hi all,

we're experiencing a similar issue but with a twist.

our installation has an organizations with multiple servers. one of the servers is unable to send messages (messages remain with status "pending" in the queue), however, the other servers within that same organisation are running perfectly fine.

the problematic server is actually a testing server, with exactly the same IP-Pool, domain and configurations, except the auto-generated configs (e.g. DKIM key and login data).

in Postal-Status everything looks ok.

Any ideas, similar experiences or tips?

Thanks in advance for any input!

[willpower232]

@ahmedaldabbagh you might have to examine the database to see if the messages have been claimed by a now-killed process or have been stuck in some other fashion. There is another issue about this somewhere but I can't find it quickly.

[ahmedaldabbagh]

@ahmedaldabbagh you might have to examine the database to see if the messages have been claimed by a now-killed process or have been stuck in some other fashion. There is another issue about this somewhere but I can't find it quickly.

thanks for the feedback @willpower232 , I have seen "locked" mails in the past. however, this "pending" issue is occurring to all mails sent since 24. July 2019 via the problematic server. I am wondering whether each server has it's own separate queue, which could mean the queue manager for that account is down.

thanks for the input in any case.

[willpower232]

Thanks for confirming, certainly possible something bad is happening with the queue. Do open a new issue if you find anything discussion worthy!

[ahmedaldabbagh]

Thanks for confirming, certainly possible something bad is happening with the queue. Do open a new issue if you find anything discussion worthy!

will do @willpower232.
as i said, the issue is only linked to a test server, which we very rarely use at the moment, and could easily be replaced with a new server within the organisation. but i will create an issue if i discover a bug or any strange issues.
thanks for your input anyway.

[freescout-helpdesk]

It happened again. Now the reason is more obvious. First time Postal stoped sending emails when hosting closed outgoing 25 port (to which Postal was connected). This time Postal stoped sending emails when there were DNS issues at the server/hosting.

And again manual restarting Postal fixed the issue:

Nov 24 17:36:00 mail systemd[1]: Stopped Postal Mail Platform.
Nov 24 17:36:00 mail systemd[1]: postal.service: Unit entered failed state.
Nov 24 17:36:00 mail systemd[1]: postal.service: Failed with result 'timeout'.
Nov 24 17:36:00 mail systemd[1]: Starting Postal Mail Platform...
Nov 24 17:36:00 mail postal[28195]: Removed worker.1.pid because process was not active
Nov 24 17:36:00 mail postal[28195]: Started Procodile supervisor with PID 28206
Nov 24 17:36:01 mail systemd[1]: Started Postal Mail Platform.

Also noticed that before restarting worker.1 had some additional text next to it in the Postal status:

   CGroup: /system.slice/postal.service
           ├─29989 [procodile] Postal (/opt/postal/app)
           ├─29995 [postal] web.1
           ├─29997 [postal] worker.1 (r
           ├─29999 [postal] cron.1
           ├─30001 [postal] smtp.1
           └─30004 [postal] requeuer.1

After restarting it disappeared:

   CGroup: /system.slice/postal.service
           ├─28206 [procodile] Postal (/opt/postal/app)
           ├─28212 [postal] web.1
           ├─28214 [postal] worker.1
           ├─28216 [postal] cron.1
           ├─28218 [postal] smtp.1
           └─28221 [postal] requeuer.1

Apparently when there are connection issues something gets broken and Postal just keeps adding emails to the queue without sending them.

To reprodue the issue maybe try to run Postal and close outgoing port which Postal is listening or try to mess up DNS on the server somehow.

[freescout-helpdesk]

Now everything works fine, but worker.1 has some (i next to it:

├─28214 [postal] worker.1 (i

Just wondering if this has any meaning?

[willpower232]

I'm afraid I don't know that much about the depths of systemd. Is your systemd config based on https://github.com/postalhq/postal/wiki/Starting-on-boot ?

[freescout-helpdesk]

Yes. Just looking for some way to check Postal state via cron and restart it when is stucks.

[freescout-helpdesk]

Here is the bash script to monitor number of pending emails in Postal and restart it if needed:

# !/usr/bin/sh

ADMIN_EMAIL='admin@example.com'
DATE_TIME=`date +%Y-%m-%d_%H:%M:%S`

pending=`mysql -u postal --password=123 -h 127.0.0.1 -e "select count(*) as pending from messages where status = 'Pending'" --vertical postal-server-1 | grep pending | sed -e 's/pending: //g'`
echo "[$DATE_TIME] Pending: $pending"

if [ $pending -gt 10 ]; then
    echo 'Restarting...'
    status=`/usr/sbin/service postal status`
    /usr/sbin/service postal restart
    echo -e $status | mail -s "[Postal] Restart" $ADMIN_EMAIL
fi