Regression Test Environment

Setting up the regression environment

The regression suite (hmailserver/test/RegressionTests, 2,127 tests at 6.2.28) does not mock the server. It runs against a real hMailServer service on the same machine, over SMTP, IMAP and POP3, and through the COM API, and it creates and deletes domains, accounts and mail as it goes. That makes it worth far more than a unit suite, and it makes the machine it runs on part of the test. This document is how to build that machine.

build/preflight-tests.ps1 is the machine-readable version of everything below. Every check in it corresponds to a failure that has actually cost a run, and when a step here is missing the pre-flight names it. Run it before every run; Pre-flight passed - safe to run the suite is the state this document gets you to.

Never run the suite against a production installation. It authenticates as Administrator, binds the live ports, and wipes data.

The pieces

Piece	What it is for
The service, running from the repository's Release build	The code under test. Not an installed copy.
SQL Server Compact 4.0 and the bench database	The default backend; the suite upgrades it in place, never recreates it.
A short data folder	Long account names overrun MAX_PATH under a deep checkout.
The test domain and the four standard ports	What every fixture assumes exists at start.
ClamAV (clamd on 3310)	The live-scanner tests.
SpamAssassin as a Windows service named SpamAssassinJAM	The SpamAssassin tests, including the one that stops and restarts it.
The NUnit console runner and the built test assembly	build/build-tests.ps1 produces both.

The bench, drawn

Everything is on one machine, on loopback, and the test process is outside the server — it talks to it the way a mail client does, plus one COM channel for configuration.

flowchart LR
    subgraph runner["The test process - unelevated"]
        NU["nunit3-console.exe<br/>RegressionTests.dll, x64 Debug"]
        SIM["Client simulators:<br/>SmtpClientSimulator,<br/>IMAPClientSimulator,<br/>POP3ClientSimulator"]
        FAKE["Fakes it hosts itself:<br/>FakeDnsServer, FakeHttpEndpoint,<br/>FakeHttpProxy, FakeSigstore,<br/>FakeFilterEngine, SpamdSimulator,<br/>SMTPServerSimulator, Pop3ServerSimulator"]
        NU --- SIM
        NU --- FAKE
    end

    subgraph svc["The hMailServer service - the code under test"]
        EXE["hMailServer.exe<br/>from the x64 Release output in the<br/>checkout - never an installed copy"]
        COM["COM: hMailServer.Application<br/>Administrator / testar"]
    end

    subgraph deps["Real dependencies, really running"]
        CLAM["clamd.exe<br/>TCP 3310, at the hard-coded path"]
        SA["spamd.exe<br/>service SpamAssassinJAM, TCP 783"]
        CE[("SQL Server Compact 4.0<br/>the bench database -<br/>upgraded, never recreated")]
    end

    DATA["The data directory<br/>short on purpose: long account<br/>names overrun MAX_PATH"]

    SIM -->|"SMTP 25, 587"| EXE
    SIM -->|"POP3 110"| EXE
    SIM -->|"IMAP 143"| EXE
    NU -->|"configure, restart, assert"| COM
    COM --- EXE
    EXE --> CLAM
    EXE --> SA
    EXE --> CE
    EXE --> DATA
    EXE -->|"DNSServer = 127.0.0.1"| FAKE

Loading

Piece	Where it listens	Set by	If it is missing
SMTP	25 and the submission port	hm_tcpipports, seeded	Pre-flight check 11 fails; otherwise connection-refused failures that look like protocol bugs
POP3	110	the same	as above
IMAP	143	the same	as above
clamd	127.0.0.1:3310	clamd.conf; the path C:\clamav\clamd.exe is hard-coded in CustomAsserts	The anti-virus fixtures fail open — the server gets no verdict, so nothing is detected and nothing says why
spamd	127.0.0.1:783	the SpamAssassinJAM service wrapper	The SpamAssassin fixtures report inconclusive; the one that stops and restarts the service cannot find it
FakeDnsServer	127.0.0.1:53, inside the test process	Shared/SuiteDns.cs, a [SetUpFixture]	Every DNS-dependent test resolves live names again — the failure mode that made this suite-wide
SQL CE	no port; a file	Database.CreateInternalDatabase(), once	The service refuses its database and nothing in the failure says "database"

The one thing that is not real: DNS

Since 5 September 2026 the whole suite serves its own zone. SuiteDns binds a FakeDnsServer on 127.0.0.1:53 before the first fixture, points DNSServer at it, restarts the service once, and puts the setting back at the end. NODATA is the correct answer for almost everything the suite asks — "no such record" is what an SPF, DMARC or MX lookup for a test domain should see — and a fixture that needs a positive answer adds it and forgets it again in teardown. Two names are seeded for everyone: the SURBL project's permanent test point, and localhost.

The reason it is suite-wide rather than per-fixture is worth knowing, because it generalises. Before, fourteen fixtures each bound their own resolver, restarted the service twice, and every test outside those fourteen still resolved live names. Five consecutive full runs on 19 August 2026 each exposed a fresh set of live-DNS failures: the population of tests that resolve something is larger than any single run reveals, and pinning them one at a time costs thirty-five minutes per four.

A test that must serve DNS itself takes the port for the duration of Suspend(). A test that points the server at some other resolver must restore SuiteDns.Resolver afterwards — not null, which would silently put the rest of the run back on the system resolver.

One run, end to end

sequenceDiagram
    autonumber
    participant D as Developer
    participant P as preflight-tests.ps1
    participant R as nunit3-console
    participant S as hMailServer service

    D->>P: build/preflight-tests.ps1
    P->>S: check PathName, state, COM auth,<br/>domain count, port count, schema version
    P->>P: check ERROR log, clamd, SpamAssassinJAM,<br/>listeners, VPN adapters, test assembly, orphan .cs
    P-->>D: "Pre-flight passed - safe to run the suite"
    D->>R: build/run-tests.ps1
    R->>S: SuiteDns: bind 127.0.0.1:53,<br/>set DNSServer, restart the service
    loop every fixture
        R->>S: PerformBasicSetup - AssertNoReportedError
        R->>S: create domain, accounts, ports as needed
        R->>S: drive SMTP / IMAP / POP3 / COM
        R->>S: teardown - remove what it added,<br/>clear any error it provoked
    end
    R->>S: SuiteDns: restore DNSServer
    R-->>D: results, written only at the end of the run

Loading

NUnit writes its failure details only at the end, so a run that looks quiet for fifty minutes is normal. Do not stop it: an aborted run leaves the state listed under "When a run is interrupted" below, and the next run then fails in ways that point somewhere else entirely.

1. The service runs from the repository build

---

Build the server (build/build.ps1 -Configuration Release), then run build/post-build.ps1 -Configuration Release. It needs elevation and will ask for it. It copies the runtime DLLs beside hMailServer.exe, registers the COM server and the Windows service, and points the service at hmailserver\source\Server\hMailServer\x64\Release\hMailServer.exe in your checkout.

The pre-flight's first check reads the service's PathName and refuses anything else. The failure it guards against is a stray installer: running the installer on the development machine re-points the service at C:\Program Files\..., and from then on every "green" run tests the installed binary, not the one you just built. If that has happened, post-build.ps1 again, then re-apply the permission grant in step 7, then delete HKLM\SOFTWARE\hMailServer in both registry views (reg delete "HKLM\SOFTWARE\hMailServer" /reg:32 /f and the same with /reg:64) - a leftover InstallLocation value redirects the server's INI lookup and it comes up with empty configuration while still reporting "Running". build/make-hmailserver-writable.ps1 deals with the file permissions the build needs on the output directory.

Provision the output directory as the installed layout expects (IMPLEMENTATION-NOTES.md, "Test-environment recipe", has the full list): hMailServer.ini with a [Directories] section and an empty AdministratorPassword, the Logs, Temp and Events folders, a copy of DBScripts\ with CRLF line endings, Languages\, dh2048.pem and tlds.txt from installation\Extras, and a Bin\ folder holding 7za.exe and a second copy of hMailServer.ini.

2. SQL Server Compact and the bench database

---

Install the 4.0 x64 package from hmailserver\installation\SQLCE (it is the version SQLCEConnection binds to; the x86 package is neither needed nor installable on x64). Create the database once, through COM: Database.CreateInternalDatabase() followed by Reinitialize() on an hMailServer.Application object authenticated as Administrator.

From then on the database is upgraded, never recreated. REQUIRED_DB_VERSION in Server/Common/Application/Constants.h moves whenever a release adds a column, and a bench that nobody upgraded fails in a way that points somewhere else entirely: the service refuses the database connection, the first pre-flight check says "could not start", the ERROR-log check finds a stale error, and neither says the word "database". The pre-flight's fourth check exists to say it. Move the bench forward with build/upgrade-test-database.ps1 before building a server whose required version has moved past it.

Because the bench is always upgraded, no test here takes the fresh-install path that CreateTablesMSSQL.sql provides. build/check-db-scripts.ps1 covers that separately by creating a throwaway database from the script; RELEASE.md makes it a release step for exactly that reason.

3. A short data folder

---

Point the data directory at something short, C:\HMTest\Data is the convention the pre-flight knows. The suite creates accounts with deliberately long names, and under a deeply nested checkout path the resulting file names exceed MAX_PATH. The symptom is not a clear error but Windows error 206 cascading into hundreds of clean-log assertion failures in tests that never mention paths.

4. The test domain and the standard ports

---

The suite expects exactly one domain and exactly four TCP/IP ports at start (SMTP 25, POP3 110, IMAP 143 and the SMTP submission port). The pre-flight reads both counts from the running server. A port count of sixteen is a signature, not a mystery: the TLS fixtures register twelve extra ports and remove them in teardown, so a run killed while one of them was in flight leaves them behind. Remove the non-standard ones in the Control Panel, or re-add the four if there are fewer.

The suite authenticates over COM as Administrator with the password testar, falling back to a blank password. Set it to testar. It is a bench credential on a machine that must never hold real mail, which is why it appears in the scripts in plain text and why it is not a secret.

5. ClamAV

---

Install ClamAV (winget install Cisco.ClamAV works) and copy the installation to C:\clamav - the suite's CustomAsserts hard-codes C:\clamav\clamd.exe. A minimal clamd.conf is enough: TCPSocket 3310, TCPAddr 127.0.0.1, DatabaseDirectory C:\clamav\database. Run freshclam once, then start clamd.exe. It runs as a bare process, not a Windows service, so the pre-flight checks for a listener on 3310 rather than for a service.

Warm-up matters: clamd accepts connections before its signatures have finished loading, and hMailServer fails open when the scanner gives no verdict. Before trusting the anti-virus fixtures on a freshly started clamd, send one EICAR message and confirm "Virus detected" in the log.

6. SpamAssassin

---

The JAM Software x64 build works (https://downloads.jam-software.de/spamassassin/SpamAssassinForWindows-x64.zip, rules bundled under share\). Extract it to C:\SpamAssassin and wrap spamd.exe -i 127.0.0.1 -A 127.0.0.1 -p 783 as a Windows service named SpamAssassinJAM - WinSW does this well, and the child process keeps the name spamd, which both the process-gate check and the fixture that stops and restarts the service rely on.

7. Let the unelevated test runner control the services

---

One fixture stops and restarts SpamAssassinJAM; rebuilding the server stops and starts hMailServer. Neither should need an elevated console. Grant Authenticated Users start/stop on both services with sc sdset (the exact descriptor is in IMPLEMENTATION-NOTES.md). The grant is silently dropped when a service is re-registered, which is why "Access denied" starting the service after a post-build.ps1 is the pre-flight's own diagnosis for that failure.

8. Build the tests

---

build/build-tests.ps1 restores the NuGet packages into hmailserver\test\packages

• where the project's hint paths point, which a bare dotnet restore on the project does not do - and builds RegressionTests.dll for x64. Always x64: an AnyCPU build of the same project also runs, against stale assemblies, and passes tests it never actually executed.

RegressionTests.csproj is a legacy project that lists every source file explicitly. A new test file that is not added to it is not merely unrun, it is invisible, and a green suite says nothing about it. The pre-flight compares the directory against the project and names any orphan.

9. Run it

---

build/preflight-tests.ps1, then build/run-tests.ps1. The runner is nunit3-console; subsets take -Where "class =~ /RegressionTests.IMAP.Binary/". NUnit writes its failure details only at the end of the run, so a run that looks quiet for fifty minutes is normal.

Do not dotnet test this project. It exits 0 having run nothing.

Never build, run static analysis, or start other heavy work while the suite is running. A rebuild restarts the service under the tests, the fixtures' restart detector then fails every test that follows, and the run can neither convict nor exonerate the code - thirty minutes are simply thrown away.

On Linux

RegressionTests.csproj is net481 with a COMReference to the service and runs nowhere but a Windows bench. hmailserver/test/LinuxRegressionTests is the part of the suite that can test the Linux server: a net10.0 project that compiles 52 of the fixture files in place from RegressionTests - the ones that drive the server through SMTP, IMAP, POP3 and MIME rather than through COM - together with the socket simulators, and stands a REST-backed fixture layer under them (LinuxRegressionTests/Shims). The fixtures are not copied and not edited; a test that passes here is the same test that passes on Windows.

What the fixture layer does instead of COM: the test domain example.test is deleted and recreated before every test through POST /api/v1/domains (or, on a server built before that route existed, emptied of its accounts and lists); accounts, aliases and distribution lists go through their routes; a server setting a fixture writes goes through PUT /api/v1/settings or its anti-spam and logging groups, and the suite's own defaults are put back before every test the way the Windows PerformBasicSetup puts them back through COM; the localhost SMTP routes go through /api/v1/routes; the TLS listeners the SSL fixtures need are made from the certificate and port routes and brought up with POST /api/v1/server/reinitialize; the delivery queue is drained through /api/v1/queue; and the ERROR log is read through /api/v1/logs, judged by what was written during the test rather than by whether the file exists. Everything a test makes is removed in its teardown.

What it cannot do, it says: a fixture that needs the suite's fake DNS zone served to the server's resolver, or a COM-only call such as Utilities.EvaluateSieveScript, is skipped with that reason, from the shim, never passed with a weakened assertion. A shim that needs a route also asks the server's own OpenAPI document whether it has it, so an older server skips with the route named rather than failing. dotnet test reports those as skipped; a failure is the server's. The REST API also refuses a credential's 201st request in any ten-second window, and the fixture layer keeps under that on its own.

The server it runs against is any hMailServer with the REST listener on, on any host; the defaults are the PostgreSQL-backed tree the CI job lays out (.github/workflows/linux-build.yml, job regression-on-linux), which is also the recipe for a hand-made one: DBScripts from the checkout beside the binary, tlds.txt and dh2048.pem from installation/Extras, an INI with Type=PostgreSQL and RestApiPort=8045, --create-database, the listeners moved off the privileged ports with an UPDATE of hm_tcpipports, the auto-ban switched off with an UPDATE of hm_settings (AutoBanOnLogonFailureEnabled to 0 - the suite fails logons on purpose, and three of them ban the test host for an hour; the Windows setup switches it off over COM), and --set-admin-password from a pipe. The tests are told where it is through the environment, read once when the assembly loads:

Variable	Default
HMTEST_HOST	127.0.0.1
HMTEST_SMTP_PORT	2525
HMTEST_POP3_PORT	1110
HMTEST_IMAP_PORT	1143
HMTEST_REST_PORT	8045
HMTEST_ADMIN_PASSWORD	testar

Then, from any machine with the .NET 10 SDK that can reach those ports:

dotnet test hmailserver/test/LinuxRegressionTests -c Release --logger trx

Unlike the Windows project, this one is meant for dotnet test. Windows and Linux hosts both work - a WSL server is reachable from Windows through mirrored networking - and the CI job runs it on Linux against the binary the same workflow just built. Shared/TestPorts.cs is what lets the simulators be pointed elsewhere; the Windows suite never sets it and sees the standard ports.

When a run is interrupted

A run that is stopped part-way leaves state behind, and the next run fails in ways that point at the wrong thing. RELEASE.md step 4 says never abort a run for this reason; when it happens anyway, build/preflight-tests.ps1 -Clean removes what it can and names the rest:

• A stale ERROR log. One fixture writes a deliberate scanner error and removes it in teardown. Left behind, it fails every fixture's setup in the next run - the observed shape is 100% of tests failing before doing anything.
• Test-only settings left in hMailServer.ini. The suite points the resolver at its own fake DNS server on 127.0.0.1 for the whole run (Shared/SuiteDns.cs, a [SetUpFixture]: NODATA for every name a fixture has not added, the SURBL test point and localhost seeded) and puts DNSServer back when the run ends; fixtures that enable the quarantine, set a POP3 login delay, a password policy, PROXY-protocol trust, or an OTLP endpoint put their key back in teardown. Any of them surviving fails tests that never mention the setting: every DNS lookup waits for the query timeout; every anti-spam refusal becomes an acceptance; every connection from the trusted address is dropped before the banner. The server caches the INI at start, so after -Clean restart the service.
• Orphan domain directories in the data folder. Several persistence fixtures rename the test domain, and the rename moves its directory. Killed mid-rename, the directory survives with no domain row, and every later run of that fixture fails with "already exists" about a directory it never named.
• The twelve extra TLS ports described in step 4.

The environment traps

Each of these has produced a red run that looked like a code regression.

• Anti-virus products that proxy localhost mail traffic in-process. ESET, for one, answers an IMAP literal continuation itself with + just send it and rewrites raw SMTP framing. A failure whose text contains that string is the proxy, not the server, and it returns after a reboot. Disable mail and SSL protocol filtering for localhost.
• VPN clients that rewrite loopback. ProtonVPN and WireGuard adapters have broken address selection in fixtures that bind a specific local address. The pre-flight warns when one is up.
• Line endings. The repository checks out CRLF on purpose (.gitattributes): the database scripts are split on \r\n\r\n, the DKIM-signed .eml resources have body hashes that change with the line ending, and raw-message tests send file content verbatim and are refused for bare LF. A tool that "normalises" the tree to LF breaks all three at once.
• Mark-of-the-Web. Files downloaded rather than cloned carry a zone identifier that the service refuses to load. Strip it from the whole tree.
• "Passes alone, fails in a group." This points at shared state between fixtures - an account, a setting, a port left behind by the fixture that ran before - and not at the network, however network-shaped the failure message is. Look at what the preceding fixture in the run order changes.
• A build number that names a tree that no longer exists. Not an environment trap exactly, but the same lesson: the suite's verdict is about the binary the service is running, which is why the pre-flight checks the path and why RELEASE.md voids a gate the moment anything changes after it.