PIPELINING issue with Postfix #18
Replies: 10 comments 5 replies
|
Thank you — this was reported here and on the forum at almost the same time, and it is fixed in 6.2.13, released a few hours ago: https://github.com/Progressiverobot/hmailserver/releases/tag/v6.2.13 Your diagnosis was close, and your workaround makes sense once you see what was actually happening. There were two independent defects behind that exact symptom, and both are fixed: 1. A blocking reverse-DNS lookup on the network thread (the main cause). When the first chunk of message data was flushed to disk, hMailServer generated the 2. Pipelined data in the same TCP segment was ignored. When a client batched its commands and the message data arrived in the same read as the preceding command, the binary read path asked the socket for new bytes and ignored what was already sitting in the receive buffer — so it waited for data that had already arrived. This is the part your PIPELINING workaround was really side-stepping. The read path now consumes what is already buffered. In 6.2.14 (shortly) that reverse-DNS work also moves to its own thread pool, so even a flood of connections from addresses with unresponsive reverse DNS cannot delay message acknowledgements for anyone else. What I would ask: upgrade to 6.2.13, then remove the Postfix workaround and re-enable PIPELINING for that host — it should now behave. If it does not, please reopen/reply with the hMailServer SMTP log for one failing session and I will look again immediately. Sorry you hit this, and thanks for reporting it clearly enough to be actionable. |
|
Hi Chris, Back to subject, I updated to 6.2.3 and commented out Postfix workaround config line and reloaded it, but unfortunately it didn't work out. Same symptom. Here full logs from both sides Postfix & HMS `PMG/Postfix maillog(... external transaction with PMG/Postfix...) HMS SMTP Log"SMTPD" 3360 13 "2026-08-11 22:23:05.482" "192.168.1.25" "SENT: 220 mail.local.com ESMTP" Thanks and have a nice evening |
|
Following up: 6.2.14 is out — https://github.com/Progressiverobot/hmailserver/releases/tag/v6.2.14 The reverse-DNS work that caused your stall now runs on its own thread pool, so even a burst of connections from addresses with unresponsive reverse DNS cannot delay message acknowledgements for other sessions. Together with the two fixes in 6.2.13 (the blocking PTR lookup, and pipelined data being ignored in the receive buffer) that should be the end of it. If you have not already: upgrade, then remove the Postfix workaround and re-enable PIPELINING for that host. If anything still stalls after |
|
Upgraded to 6.2.14 but sorry, nothing changes. I even full restarted Postfix after removing workaround config. (192.168.1.25 is PMG) |
|
Thank you for testing so quickly, and I'm sorry — I have now told you twice this was fixed and it isn't. The two defects I fixed were real, but they are evidently not your defect. Let me be straight about where this stands. What I did with your log. I built your exact scenario against a 6.2.14 server and could not reproduce it. Specifically, all of these complete normally here:
So the problem is not pipelining in itself, nor the shape of the data. Something specific to your server is involved, and I can't find it by guessing from here. What I've added so that the next attempt is conclusive. Between Could I ask for the following when 6.2.15 is out?
In the meantime, please put your Postfix workaround back. It works, it costs you nothing measurable, and there is no reason for you to be the one carrying broken inbound mail while we chase this. One thing that would help me enormously: if you are able to run |
|
6.2.15 is released: https://github.com/Progressiverobot/hmailserver/releases/tag/v6.2.15 I want to be straight about what that does and does not mean for this thread: your problem is not fixed. I could not reproduce it across a wide set of variants, and I would rather say so than let a release note imply otherwise. What 6.2.15 adds is the instrumentation to find it. Between Whichever of those three stops appearing tells us which stage is stuck. If you have the time: upgrade, turn on debug logging, reproduce once with your Postfix workaround removed, and post the log from Keep your workaround in place until then. It works and costs you nothing measurable. Tracking continues in #20. One thing from this release that may be relevant to your setup even though it is a different code path: while reviewing a connection-layer change, I found that a |
|
Found it. 6.2.17 is out: https://github.com/Progressiverobot/hmailserver/releases/tag/v6.2.17 I have told you twice before that something was fixed when it wasn't, so let me be precise about what I actually have this time: a reproduction, a mechanism, and a bound. Not a guess. Why the log stops where it does. Your Postfix error — "timed out while sending end of data" — means the body and the terminating dot were fully transmitted and Postfix was waiting for hMailServer's Why it only happens to you and not to me. When mail arrives from a trusted incoming relay — exactly what a Proxmox Mail Gateway front end is — hMailServer defers its entire spam battery to after end-of-data. For a direct or authenticated sender, those same tests run earlier, during Why it's intermittent. That deferred work runs on a pool of 15 threads, and it holds the thread that sends the I reproduced it deliberately, pointing SpamAssassin at an endpoint that accepts connections and never answers:
What 6.2.17 changes
What I'd ask of you. Upgrade and remove your Postfix workaround. One of two things will happen, and both are useful:
If it still hangs silently after 6.2.17, that is genuinely new information and I want to see it — but the path that produced your symptom can no longer wait forever. Thank you for staying with this. The report was accurate the whole time; I was searching the wrong part of the transaction, and the honest "not fixed" replies were the only thing keeping it findable. Tracking issue #20 is closed with this release. |
|
Sorry, nothing changes.
Debug: SMTP Log: |
|
Found it. Your log has it, and I owe you an apology for the two earlier "this is fixed" replies — this is the third cause and it is the one that matches every symptom you reported. The line that gave it away 7984 bytes received against your Postfix's The cause The test for the end of a message was, literally: buffer[size-5] == '\r' && buffer[size-4] == '\n' && buffer[size-3] == '.' &&
buffer[size-2] == '\r' && buffer[size-1] == '\n'It requires CRLF immediately before the dot. When the last line of a message body ends with a bare LF, the data arrives as The result is not a rejection. It is a permanent wait: reception never completes, no reply is ever sent, the spool file stays at zero bytes, and your Postfix eventually reports "timed out while sending end of data". That is your report, exactly, including the zero-byte files stuck in the Data folder. Why You said you had no idea whether it mattered. It is the centre of this. That setting exists precisely to accept senders that use bare LF instead of CRLF, and it was honoured everywhere in the server except in the one place where ignoring it hangs the connection instead of merely being strict. So the server was configured to tolerate your mail stream and then hung on it. It also explains why disabling PIPELINING appeared to help: it changed how the data was segmented, which changed where the bare LF fell relative to the terminator. Why I could not reproduce it Every test I wrote — and every one of the 1191 in the suite — sends correctly formed CRLF data, because they all go through the same test client. A defect in byte-level framing is invisible to a helper that only writes well-formed bytes. There are now three tests that speak raw SMTP and write the exact byte sequences, including the two-byte What I need from you The fix accepts When the build is out: upgrade, remove the Postfix workaround and re-enable PIPELINING for that host, and let me know. If it still stalls, the debug log will now be decisive either way — but I do not expect it to. Thank you for staying with this through two wrong answers and for posting a log with debug enabled. Without the |
|
6.2.18 B20 Debug Log SMTP Log |
|
You were right to give up on my "fixed" claims, and I owe you the real answer, because I finally have it — found by doing what should have been done from the start: running an actual Postfix against this server and watching the bytes. What your bug actually was. When Postfix finishes a message and has nothing more for the connection, it sends the end of the body, the terminating dot, and Why your workaround worked. Without PIPELINING, Postfix waits for the 250 before sending QUIT — so the terminator IS the last thing in the buffer, and the old check matched. You didn't change what was sent; you changed where the segment boundary fell. And whether an unmodified Postfix hits the bug is pure TCP segmentation luck — a same-host relay like your PMG wins that lottery essentially every time, while internet senders almost never do. That's why it looked like a PMG-specific incompatibility for so long. One honest correction. My 6.2.18 diagnosis said your stream carried a bare-LF terminator. The capture shows stock Postfix normalizes bare LFs to CRLF before retransmitting — that's also why your log showed 7984 bytes against The proof. I reproduced your hang on the current build with a real Postfix 3.10 (four deliveries frozen after 354, "end-of-data not yet seen" with the complete message buffered), fixed the detection to find the standard terminator anywhere in the buffer and hand the pipelined bytes back to command parsing, and re-ran the same four deliveries: all accepted in under 20 ms, the pipelined QUIT answered with 221, message content byte-identical. Regression tests now replay the exact captured wire bytes — pipelined QUIT, and a whole second transaction pipelined behind the dot — and the Postfix rig itself is committed to the repo so every future build gets checked against the real thing. A build ships at 00:00 GMT tonight (2026-08-16). No need to withdraw PIPELINING — it was never the culprit, and the thing that framed it is dead. If you ever feel like giving it one more try on that build, remove the workaround and it should just flow; but after three rounds I'm not asking you to be the test bench again — the real Postfix here does that job now. |
|
The pipelined- Two things worth flagging from the pre-release review, since this thread is where the credibility was spent: The review found that my own fix had introduced a way for a bare-LF end-of-data marker split across a buffer boundary to be misread as the standard one — which would have weakened the SMTP-smuggling protection (CVE-2023-51764) that the bare-LF handling exists to preserve. That is fixed in this release, before it ever shipped. It also found that a body line beginning with a dot could be sent unprotected when a buffer boundary fell one or two bytes before it, which corrupts the line at the far end and, for a line consisting of a single dot, truncates the message there. Also fixed here, with tests that drive the exact boundary. No workaround should be needed on your side now — PIPELINING was never the culprit. If anything still stalls after |
|
The pipelined- Two things worth flagging from the pre-release review, since this thread is where the credibility was spent: The review found that my own fix had introduced a way for a bare-LF end-of-data marker split across a buffer boundary to be misread as the standard one — which would have weakened the SMTP-smuggling protection (CVE-2023-51764) that the bare-LF handling exists to preserve. That is fixed in this release, before it ever shipped. It also found that a body line beginning with a dot could be sent unprotected when a buffer boundary fell one or two bytes before it, which corrupts the line at the far end and, for a line consisting of a single dot, truncates the message there. Also fixed here, with tests that drive the exact boundary. No workaround should be needed on your side now — PIPELINING was never the culprit. If anything still stalls after |
|
The pipelined- Two things worth flagging from the pre-release review, since this thread is where the credibility was spent: The review found that my own fix had introduced a way for a bare-LF end-of-data marker split across a buffer boundary to be misread as the standard one — which would have weakened the SMTP-smuggling protection (CVE-2023-51764) that the bare-LF handling exists to preserve. That is fixed in this release, before it ever shipped. It also found that a body line beginning with a dot could be sent unprotected when a buffer boundary fell one or two bytes before it, which corrupts the line at the far end and, for a line consisting of a single dot, truncates the message there. Also fixed here, with tests that drive the exact boundary. No workaround should be needed on your side now — PIPELINING was never the culprit. If anything still stalls after |
|
I confirm the solution with 6.2.21, logs below. |
Uh oh!
There was an error while loading. Please reload this page.
Hello,
External inbound mail is relayed by Proxmox Mail Gateway (MTA Postfix) to HMS.
Obvously there is an issue with new SMTP extensions PIPELINING. After "RECEIVED: DATA" -> "SENT: 354 OK, send." connection dies after a while. Postfix maillog tells "...timed out while sending end of data..". Eventually zero bytes saved and stuck in HMS Data folder. As a result, no external receiving is possible.
Workaround: Configuring Postfix to ignore PIPELINING in EHLO particularly for that HMS IP (howto is off topic).
Considering Postfix is a major player, I'd suggest to look into this asap.
All reactions