hMailServer 6.2.21

Inbound mail from a Postfix relay could hang forever. If mail from a front-end MTA has ever stopped arriving with nothing in the log after 354, this release is the fix.

Everything in the 6.2.20 notes is in this release too.

The Postfix hang, found by running a real Postfix

Discussion #18 reported inbound mail from a Proxmox Mail Gateway never being received: the data arrives, the log stops dead after 354 OK, send., and the sending server eventually gives up with "timed out while sending end of data", leaving a zero-byte file in the Data folder. Three previous releases claimed to fix it. The reporter tested each one and was right every time that it was not fixed.

What settled it was standing up an actual Postfix 3.10 against this server and reading the bytes on the wire. Two facts came out of that:

• When Postfix has nothing further for a connection, it sends the end of the body, the terminating dot and QUIT in one TCP segment — ...\r\n.\r\nQUIT\r\n. With more mail queued it pipelines the next MAIL FROM there instead.
• Every end-of-data check this server has ever shipped — including the one added in 6.2.18 for this very report — examined only the end of the received buffer.

With QUIT sitting behind the dot, the terminator is in the middle of the buffer, no tail check can match, and the session waits forever for something that already arrived. This also explains the two things that never added up: disabling PIPELINING "fixed" it because Postfix then waits for the 250 before sending QUIT, putting the terminator back at the end of the buffer; and whether any given sender triggers it is TCP segmentation luck, which a same-host relay wins essentially every time while internet senders almost never do.

The receive path now searches for the standard <CRLF>.<CRLF> anywhere in what has arrived and hands anything behind it back to the command parser, so a pipelined QUIT or a whole following transaction is executed rather than lost. Verified against the real Postfix: four deliveries that hung before now complete in under 20 ms with the QUIT answered and the message byte-identical.

The bare-LF spellings of the terminator, which this server accepts only when Allow incorrect line endings is on, are deliberately still recognised only at the very end of the buffer, and anything behind one is discarded — that is the CVE-2023-51764 SMTP-smuggling rule, and it is pinned by its own test.

A correction to the 6.2.18 notes: the bare-LF work that release described was real and is retained, but it was not this reporter's defect. Stock Postfix normalises bare LFs to CRLF before transmitting, so a bare-LF terminator cannot reach this server through Postfix at all — and that normalisation is also why their log showed 7,984 bytes against a declared SIZE=7975.

The Postfix test rig is now part of the repository (hmailserver/test/PostfixBench), so future builds are checked against a real MTA rather than only against a test helper that writes well-formed bytes.

SMTP dot transparency across buffer boundaries

A body line beginning with a dot is sent doubled and must be stored singled (RFC 5321 4.5.2). The test for "does this dot start a line" could not see across a buffer boundary, so a line-leading dot landing one or two bytes into a chunk was sent unprotected. The receiving server then un-stuffs a dot that was never stuffed: .X silently becomes X, and a body line consisting of a single . becomes <CRLF>.<CRLF> — the end-of-data marker — truncating the message there and handing whatever followed to the receiver's command parser. The receive path had the mirror-image defect, and after a forced mid-line split both could act on a dot that did not start a line at all.

Both directions now carry the line-start context across the boundary. Because this class of defect depends on exactly where a chunk boundary falls — something a socket cannot be made to do on demand — it is covered by in-process tests that drive the transform with exact boundaries, plus a black-box test that sends every dot shape through SMTP and reads it back over POP3.

Sieve scripts were left behind by renames and deletes

Sieve filters live in their own directory tree, separate from the mailbox. Nothing moved or removed them:

• Renaming a domain moved every mailbox and left every account's filter behind. Delivery then found no script and delivered to INBOX everything the user had arranged to file, discard or redirect — with no error and no log line.
• Renaming an account did the same for that user.
• Deleting a domain or an account left the scripts on disk. Recreating that address later — a returning customer, a rebuilt test domain — silently reactivated the previous holder's filter for the new owner's mail, including any redirect it contained.

Renames now move the Sieve tree with the mailbox, and deletes remove it.

Related, in the Control Panel: the account editor wrote the Sieve script before asking the server to accept a rename. Because that property is written to disk immediately and keyed on the address, a rename the server then refused — a duplicate address, or an installation that still needs Data Directory Synchronizer — had already overwritten (or, with an empty editor, deleted) the script belonging to whoever holds the target address. The dialog only reported that the account could not be saved. The script is now written after the save succeeds, and only when it actually changed.

Sieve filters can now read the message, not just its headers

The body test (RFC 5173) is implemented and advertised over ManageSieve, with all three transforms:

• :text (the default) — the decoded text of every text part, which is what "if the message mentions X" means. Transfer encoding and charset are decoded before matching, so a filter written against plain text keeps working when the sender's client base64-encodes the part.
• :content "type" — the decoded parts of the named MIME types, so a rule can look inside the HTML alternative without matching the plain-text one. A bare top-level type such as image matches every subtype.
• :raw — the body exactly as it arrived, no decoding, MIME structure ignored.

Seven end-to-end tests assert where a delivered message was actually filed — never what the evaluator reported — including one whose search phrase appears nowhere in the transmitted bytes and can only match if the base64 part was decoded first.

While confirming what this section could claim, six roadmap rows were found describing the opposite of the current behaviour: they said unimplemented Sieve constructs such as reject parse as valid and silently do nothing, so "the author believes mail is being refused while it is in fact being kept". That was true once and is not now — anything this server does not implement is refused when the script is uploaded, with an error naming the construct — and a sweep test now pins that refusal for every unimplemented command and test by name, so none of them can quietly slip back to being accepted-and-ignored.

An IMAP command that could hang a connection thread for good

SORT () US-ASCII ALL, and a bare SORT (REVERSE) ..., produce an empty sort-criteria list. The code that skipped to the last criterion then walked an iterator past the end of an empty vector, which in a release build is not a crash but an infinite loop: the connection thread spins at 100% of a core, never answers, and is never returned to the pool. Any authenticated user could send it, and SORT is advertised by default. The command is now rejected as the protocol error it is (RFC 5256 requires at least one criterion), with a guard inside the sort itself as well.

The message cache could never evict, and only counted upwards

Folder message-lists are cached with a 512 MB ceiling. Entries are added before their first load, so each was recorded as occupying nothing — and the eviction routine stopped at the first zero-sized entry, which was always the oldest. The ceiling therefore never evicted anything; memory was bounded only by the 30-minute expiry, so folders never revisited stayed cached for the life of the process. Separately, every removal subtracted a stale figure (or, on expiry, nothing at all), so the running total only ever grew.

Eviction now measures entries as they are, and the total is recomputed from the entries themselves before any eviction decision rather than trusted from a running tally — necessary because expunges, moves and folder deletes shrink a cached list without telling the cache. A guard was also added where the cache reads a collection's length concurrently with another session appending to it.

DNS: a malformed answer no longer ends the lookup

A user reported DKIM validation failing against a local DNS server that 5.7 handled without complaint, always on the large multi-string TXT record that carries a DKIM public key. Reproduced against a deliberately misbehaving server: exactly two malformations — a record length that runs past the end of the packet, and a compression pointer that points outside it — make Windows reject the whole response as unparseable, and both are answered correctly by the same server over TCP. Such a lookup is now retried once over TCP. A timeout is deliberately not retried.

The Control Panel's MX-query tool (#29) asked the operating system's resolver via nslookup, so on any server with a custom DNS server configured it disagreed with the server about where mail goes. It now asks the server itself, through the same resolver SMTP delivery uses, via a new Utilities.ResolveMXRecords COM method. A failed lookup is reported as a failed lookup rather than as "this domain has no mail servers" — the distinction that matters when it is your own DNS server that is unreachable.

Control Panel

Seven things the retired Administrator could do, restored: pause and resume the server engine, restore default TCP/IP ports, restore default IP ranges, delete a domain, rename a domain, rename an account, and see at a glance which domains and accounts are inactive. Also fixed: creating a duplicate domain reported the error instead of closing the window; the search boxes no longer match rows on the text of their enabled/disabled flag; the brand colour now follows the light theme; and two settings descriptions were corrected — the DNS cache option described an in-process cache that has never existed (it controls whether Windows' resolver cache may answer), and the mailer-daemon domain setting misdescribed what an empty value does.

And the Welcome page (#30): each tile centred its icon over the width of its own caption, so the six icons sat at six different offsets — none of them chosen. They are now centred on the tile, uniformly. Thanks to @helptiger for the report and the screenshot.

Known issues

• Equal-preference MX records are not randomised (RFC 5321 5.1); the server always tries them in the order the resolver returned. A comment and a COM help string previously claimed otherwise and have been corrected.
• One-click unsubscribe (RFC 8058) is not implemented. It requires a per-recipient token in the message headers, and this server stores one message file shared by every recipient, so it needs the same groundwork as list moderation. RFC 2369 List-* headers, including a List-Unsubscribe that reaches the list owner, are emitted for distribution-list postings.