Haraka crashes continued

[rricker1]

Describe the bug
provisioned haraka after latest update. Still experiencing smtp_forward timed out errors and continued haraka crashes and mail backups.

I set a cron job to restart haraka every 7 minutes which seems to keep mail flowing without the timeouts and mail backing up. Pretty ugly work around.

To Reproduce.
start haraka.

smtp_forward timed out errors start after 7-10 minutes.

Expected behavior
No timeouts/crashes

Server (please complete the following information):
FreeBSD 12.3-RELEASE-p10

Additional context
Nov 30 08:54:09 haraka haraka[57449]: [CRIT] [A779A3F8-7969-4E4A-B0DD-E547DB3D28C4.1] [core] Plugin queue/smtp_forward timed out on hook queue - make sure it calls the callback
Nov 30 08:54:24 haraka haraka[11612]: [CRIT] [F9EFB1AB-9FFE-426D-BD83-0F5E89F20794.1] [core] Plugin queue/smtp_forward timed out on hook queue - make sure it calls the callback

Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] Error: Resource not currently part of this pool
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at Pool.destroy (/root/Haraka/node_modules/generic-pool/lib/Pool.js:548:9)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at sockend (/root/Haraka/outbound/client_pool.js:172:37)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at pluggableStream. (/root/Haraka/outbound/client_pool.js:164:9)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at Object.onceWrapper (node:events:627:28)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at pluggableStream.emit (node:events:513:28)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at TLSSocket.emit (node:events:525:35)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at TLSSocket. (/root/Haraka/tls_socket.js:79:18)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at processTicksAndRejections (node:internal/process/task_queues:83:21)
Nov 30 09:17:48 haraka haraka[75461]: [CRIT] [-] [core] at endReadableNT (node:internal/streams/readable:1358:12)

I have also seen a few of these but do know if they are reinvent:

Nov 30 06:22:41 haraka haraka[37128]: [WARN] [020577EC-9423-4A8D-8249-86088A7B92A1.2] [queue/smtp_forward] transaction went away, releasing smtp_client
Nov 30 06:22:42 haraka haraka[37128]: [WARN] [159B6EC6-9C92-44A6-B2F1-3584725ED7B7.1] [queue/smtp_forward] transaction went away, releasing smtp_client
Nov 30 06:22:43 haraka haraka[37064]: [WARN] [833E329F-BC22-40A2-9A4B-E1B4E72B560B.1] [queue/smtp_forward] transaction went away, releasing smtp_client

FYI: I tried @johannes73's patch to revert to haraka v2.8.27 which did not fix the previous crashing issue for me.

[msimerson]

You had to have installed Haraka more than 2 days ago. Try again now, I removed the pooling entirely from smtp_client in HEAD, which is still shown in your logs.

[rricker1]

I installed Haraka 2 hours after your notification. To be sure, I rebuilt the entire mail system on 4 servers this morning. I am still seeing "smtp_forward timed out" errors. The bsusiest server of the four see repeated errors. Two if the servers have had only 1 or 2 "smtp_forward timed out" erros and one has had none. I am also seeing "Resource not currently part of this pool" on two of the servers.

I see "Error: Resource not currently part of this pool" every 5 - 10 minutes on two the the servers.

These are hybrid MT5/MT6 systems. If you think that this may be part of the issue let me know. I would be happy to open a support request via tnpi.biz.

Randy

[msimerson]

Try setting pool_concurrency_max=0 in your config/outbound.ini and see if that helps with the "Resource not currently part of this pool" error.

Nov 30 06:22:41 haraka haraka[37128]: [WARN] [020577EC-9423-4A8D-8249-86088A7B92A1.2] [queue/smtp_forward] transaction went away, releasing smtp_client

That's actually normal, we just need to reduce the severity of that message. It shouldn't be a warning.

[rricker1]

/data/haraka/config did not contail outbound.ini (not sure that this is normal). I created it and added pool_concurrency_max=0. That seems to have resolved the "Resource not currently part of this pool" error.

I am still getting "smtp_forward timed out" though.

[msimerson]

I am still getting "smtp_forward timed out" though.

Are you sure this isn't because the server you are forwarding to is inaccessible?

[rricker1]

So, this report contained two issues, haraka crashing "[CRIT] [-] [core] Error: Resource not currently part of this pool" and continued "smtp_forward timed out" error resulting in mail not being handed off for delivery.

"[CRIT] [-] [core] Error: Resource not currently part of this pool" crash was corrected by setting pool_concurrency_max=0 in your config/outbound.ini.

"smtp_forward timed out" error resulting in mail not being handed off for delivery was detailed very well by greenshrike in ticket #527. Thank you greenshrike.