Regression Test Environment

Setting up the regression environment

Source of truth: hmailserver/docs/RegressionEnvironment.md, kept in the repository and copied here. Re-copied from master 92b8f9f63 on 26 September 2026, and brought up the same day to the one line the tag v6.3.4 (51e2dbda0), the release, changes, where the GitHub workflow it names is now in the git history. Up to v6.3.6 the repository document wrote the 7-Zip path in step 10 with a control character where \7 belongs, and this copy read installation\Extras\7za.exe, as the document does from v6.3.7. Where the document says the suite has roughly 2,000 tests, this copy gives the count of [Test] attributes at 6.3.3 and at the tag v6.3.4. The CI jobs it names ran on GitHub Actions until 18 September 2026; since 25 September 2026 their place is taken by GitLab CI's jobs on the project's own runners, which the document now names (ContinuousIntegration.md). The jobs have run since 26 September 2026: master's first full pipeline, 2886389460 on 27 September 2026 (master 6bc09509d, 6.3.4 and one harness commit), ran the whole pipeline, its gates on the project's two runners, and passed 25 of its 29 jobs: both regression suites among them - windows-gate on bench150, 3,811 tests, 3,736 passed, 0 failed, 74 skipped and 1 inconclusive, and linux-suite on linix, 2,020 tests, 1,710 passed, 0 failed, 310 skipped, each in 2 h 23 min. It left db-matrix manual and failed secret_detection, semgrep and static-analysis, each on a fault of the tree's that batch41, the branch of 6.3.5, mends (work item #21, ContinuousIntegration.md). Brought up on 29 September 2026 to the document as it then stood for 6.3.6, which counts the Linux suite's files and tests again and names the coverage schedule's run of it, and re-copied on 5 October 2026 from the document as the tag v6.3.6 (e3c448aa6) has it (RegressionEnvironment.md at v6.3.6): tlds.txt is no longer copied from installation\Extras, since the Public Suffix List is compiled in, and the libraries are OpenSSL 4.0.3 and PostgreSQL 18.6. Re-copied on 18 October 2026 from the document as the tag v6.3.7 (31d656c0c), the release, tagged and published on 17 October 2026, has it (RegressionEnvironment.md at v6.3.7): the MariaDB client's folder is libraries\mariadb-connector-c-3.4.11; and GitLab CI runs the Windows suite in two halves at once, windows-gate on bench150 and windows-gate-s1vm on S1-WINBENCH, since S1-WINBENCH took half, and the Linux suite whole on one of the project's Linux runners, as this copy said from 17 October 2026. Re-copied on 20 October 2026 from the document as master has it from batch50, 0c0b6bae7 (RegressionEnvironment.md at 0c0b6bae7): the list of what a new Windows bench needs names the release installers build/fetch-installer-fixtures.ps1 fetches for AdminUpdateTester and AdminUpdateStageTester, which the internal suite runs last, what fails without them, and the two failures of windows-gate-s1vm on 17 October 2026 before the script was run on S1-WINBENCH. Its links to the repository's other documents point at them as 0c0b6bae7 has them, or at this wiki's copy where there is one. The section The bench, drawn - its diagram of the bench, its table of where each piece listens, what sets it and what happens without it, and its parts The one thing that is not real: DNS and One run, end to end - is this wiki's own addition.

The regression suite (hmailserver/test/RegressionTests, 3,811 [Test] attributes in 6.3.4 and 2,302 in 6.3.3) does not mock the server. It runs against a real hMailServer service on the same machine, over SMTP, IMAP and POP3, and through the COM API, and it creates and deletes domains, accounts and mail as it goes. That makes it worth far more than a unit suite, and it makes the machine it runs on part of the test. This document is how to build that machine.

build/preflight-tests.ps1 is the machine-readable version of everything below. Every check in it corresponds to a failure that has actually cost a run, and when a step here is missing the pre-flight names it. Run it before every run; Pre-flight passed - safe to run the suite is the state this document gets you to.

Never run the suite against a production installation. It authenticates as Administrator, binds the live ports, and wipes data.

The pieces

Piece	What it is for
The service, running from the repository's Release build	The code under test. Not an installed copy.
SQL Server Compact 4.0 and the bench database	The default backend; the suite upgrades it in place, never recreates it.
A short data folder	Long account names overrun MAX_PATH under a deep checkout.
The test domain and the four standard ports	What every fixture assumes exists at start.
ClamAV (clamd on 3310)	The live-scanner tests.
SpamAssassin as a Windows service named SpamAssassinJAM	The SpamAssassin tests, including the one that stops and restarts it.
The NUnit console runner and the built test assembly	build/build-tests.ps1 produces both.

The bench, drawn

Everything is on one machine, on loopback, and the test process is outside the server — it talks to it the way a mail client does, plus one COM channel for configuration.

flowchart LR
    subgraph runner["The test process - unelevated"]
        NU["nunit3-console.exe<br/>RegressionTests.dll, x64 Debug"]
        SIM["Client simulators:<br/>SmtpClientSimulator,<br/>IMAPClientSimulator,<br/>POP3ClientSimulator"]
        FAKE["Fakes it hosts itself:<br/>FakeDnsServer, FakeHttpEndpoint,<br/>FakeHttpProxy, FakeSigstore,<br/>FakeFilterEngine, SpamdSimulator,<br/>SMTPServerSimulator, Pop3ServerSimulator"]
        NU --- SIM
        NU --- FAKE
    end

    subgraph svc["The hMailServer service - the code under test"]
        EXE["hMailServer.exe<br/>from the x64 Release output in the<br/>checkout - never an installed copy"]
        COM["COM: hMailServer.Application<br/>Administrator / testar"]
    end

    subgraph deps["Real dependencies, really running"]
        CLAM["clamd.exe<br/>TCP 3310, at the hard-coded path"]
        SA["spamd.exe<br/>service SpamAssassinJAM, TCP 783"]
        CE[("SQL Server Compact 4.0<br/>the bench database -<br/>upgraded, never recreated")]
    end

    DATA["The data directory<br/>short on purpose: long account<br/>names overrun MAX_PATH"]

    SIM -->|"SMTP 25, 587"| EXE
    SIM -->|"POP3 110"| EXE
    SIM -->|"IMAP 143"| EXE
    NU -->|"configure, restart, assert"| COM
    COM --- EXE
    EXE --> CLAM
    EXE --> SA
    EXE --> CE
    EXE --> DATA
    EXE -->|"DNSServer = 127.0.0.1"| FAKE

Loading

Piece	Where it listens	Set by	If it is missing
SMTP	25 and the submission port	hm_tcpipports, seeded	Pre-flight check 8 fails; otherwise connection-refused failures that look like protocol bugs
POP3	110	the same	as above
IMAP	143	the same	as above
clamd	127.0.0.1:3310	clamd.conf; the path C:\clamav\clamd.exe is hard-coded in CustomAsserts	The anti-virus fixtures fail open — the server gets no verdict, so nothing is detected and nothing says why
spamd	127.0.0.1:783	the SpamAssassinJAM service wrapper	The SpamAssassin fixtures report inconclusive; the one that stops and restarts the service cannot find it
FakeDnsServer	127.0.0.1:53, inside the test process	Shared/SuiteDns.cs, a [SetUpFixture]	Every DNS-dependent test resolves live names again — the failure mode that made this suite-wide
SQL CE	no port; a file	Database.CreateInternalDatabase(), once	The service refuses its database and nothing in the failure says "database"

The one thing that is not real: DNS

Since 5 September 2026 the whole suite serves its own zone. SuiteDns binds a FakeDnsServer on 127.0.0.1:53 before the first fixture, points DNSServer at it, restarts the service once, and puts the setting back at the end. NODATA is the correct answer for almost everything the suite asks — "no such record" is what an SPF, DMARC or MX lookup for a test domain should see — and a fixture that needs a positive answer adds it and forgets it again in teardown. Two names are seeded for everyone: the SURBL project's permanent test point, and localhost.

The reason it is suite-wide rather than per-fixture is worth knowing, because it generalises. Before, fourteen fixtures each bound their own resolver, restarted the service twice, and every test outside those fourteen still resolved live names. Five consecutive full runs on 19 August 2026 each exposed a fresh set of live-DNS failures: the population of tests that resolve something is larger than any single run reveals, and pinning them one at a time costs thirty-five minutes per four.

A test that must serve DNS itself takes the port for the duration of Suspend(). A test that points the server at some other resolver must restore SuiteDns.Resolver afterwards — not null, which would silently put the rest of the run back on the system resolver.

One run, end to end

sequenceDiagram
    autonumber
    participant D as Developer
    participant P as preflight-tests.ps1
    participant R as nunit3-console
    participant S as hMailServer service

    D->>P: build/preflight-tests.ps1
    P->>S: check PathName, state, COM auth,<br/>domain count, port count, schema version
    P->>P: check ERROR log, clamd, SpamAssassinJAM,<br/>listeners, VPN adapters, test assembly, orphan .cs
    P-->>D: "Pre-flight passed - safe to run the suite"
    D->>R: build/run-tests.ps1
    R->>S: SuiteDns: bind 127.0.0.1:53,<br/>set DNSServer, restart the service
    loop every fixture
        R->>S: PerformBasicSetup - AssertNoReportedError
        R->>S: create domain, accounts, ports as needed
        R->>S: drive SMTP / IMAP / POP3 / COM
        R->>S: teardown - remove what it added,<br/>clear any error it provoked
    end
    R->>S: SuiteDns: restore DNSServer
    R-->>D: results, written only at the end of the run

Loading

NUnit writes its failure details only at the end, so a run that looks quiet for fifty minutes is normal. Do not stop it: an aborted run leaves the state listed under "When a run is interrupted" below, and the next run then fails in ways that point somewhere else entirely.

1. The service runs from the repository build

---

Build the server (build/build.ps1 -Configuration Release), then run build/post-build.ps1 -Configuration Release. It needs elevation and will ask for it. It copies the runtime DLLs beside hMailServer.exe, registers the COM server and the Windows service, and points the service at hmailserver\source\Server\hMailServer\x64\Release\hMailServer.exe in your checkout.

The pre-flight's first check reads the service's PathName and refuses anything else. The failure it guards against is a stray installer: running the installer on the development machine re-points the service at C:\Program Files\..., and from then on every "green" run tests the installed binary, not the one you just built. If that has happened, post-build.ps1 again, then re-apply the permission grant in step 7, then delete HKLM\SOFTWARE\hMailServer in both registry views (reg delete "HKLM\SOFTWARE\hMailServer" /reg:32 /f and the same with /reg:64) - a leftover InstallLocation value redirects the server's INI lookup and it comes up with empty configuration while still reporting "Running". build/make-hmailserver-writable.ps1 deals with the file permissions the build needs on the output directory.

Provision the output directory as the installed layout expects: hMailServer.ini with a [Directories] section and an empty AdministratorPassword, the Logs, Temp and Events folders, a copy of DBScripts\ with CRLF line endings, Languages\, dh2048.pem from installation\Extras, and a Bin\ folder holding 7za.exe and a second copy of hMailServer.ini.

2. SQL Server Compact and the bench database

---

Install the 4.0 x64 package from hmailserver\installation\SQLCE (it is the version SQLCEConnection binds to; the x86 package is neither needed nor installable on x64). Create the database once, through COM: Database.CreateInternalDatabase() followed by Reinitialize() on an hMailServer.Application object authenticated as Administrator.

From then on the database is upgraded, never recreated. REQUIRED_DB_VERSION in Server/Common/Application/Constants.h moves whenever a release adds a column, and a bench that nobody upgraded fails in a way that points somewhere else entirely: the service refuses the database connection, the first pre-flight check says "could not start", the ERROR-log check finds a stale error, and neither says the word "database". The pre-flight's fourth check exists to say it. Move the bench forward with build/upgrade-test-database.ps1 before building a server whose required version has moved past it.

Because the bench is always upgraded, no test here takes the fresh-install path that CreateTablesMSSQL.sql provides. build/check-db-scripts.ps1 covers that separately by creating a throwaway database from the script; RELEASE.md makes it a release step for exactly that reason.

3. A short data folder

---

Point the data directory at something short, C:\HMTest\Data is the convention the pre-flight knows. The suite creates accounts with deliberately long names, and under a deeply nested checkout path the resulting file names exceed MAX_PATH. The symptom is not a clear error but Windows error 206 cascading into hundreds of clean-log assertion failures in tests that never mention paths.

4. The test domain and the standard ports

---

The suite expects exactly one domain and exactly four TCP/IP ports at start (SMTP 25, POP3 110, IMAP 143 and the SMTP submission port). The pre-flight reads both counts from the running server. A port count of sixteen is a signature, not a mystery: the TLS fixtures register twelve extra ports and remove them in teardown, so a run killed while one of them was in flight leaves them behind. Remove the non-standard ones in the Control Panel, or re-add the four if there are fewer.

The suite authenticates over COM as Administrator with the password testar, falling back to a blank password. Set it to testar. It is a bench credential on a machine that must never hold real mail, which is why it appears in the scripts in plain text and why it is not a secret.

5. ClamAV

---

Install ClamAV (winget install Cisco.ClamAV works) and copy the installation to C:\clamav - the suite's CustomAsserts hard-codes C:\clamav\clamd.exe. A minimal clamd.conf is enough: TCPSocket 3310, TCPAddr 127.0.0.1, DatabaseDirectory C:\clamav\database. Run freshclam once, then start clamd.exe. It runs as a bare process, not a Windows service, so the pre-flight checks for a listener on 3310 rather than for a service.

Warm-up matters: clamd accepts connections before its signatures have finished loading, and hMailServer fails open when the scanner gives no verdict. Before trusting the anti-virus fixtures on a freshly started clamd, send one EICAR message and confirm "Virus detected" in the log.

6. SpamAssassin

---

The JAM Software x64 build works (https://downloads.jam-software.de/spamassassin/SpamAssassinForWindows-x64.zip, rules bundled under share\). Extract it to C:\SpamAssassin and wrap spamd.exe -i 127.0.0.1 -A 127.0.0.1 -p 783 as a Windows service named SpamAssassinJAM - WinSW does this well, and the child process keeps the name spamd, which both the process-gate check and the fixture that stops and restarts the service rely on.

7. Let the unelevated test runner control the services

---

One fixture stops and restarts SpamAssassinJAM; rebuilding the server stops and starts hMailServer. Neither should need an elevated console. Grant Authenticated Users start/stop on both services with sc sdset (an access-allowed entry for AU carrying the start and stop rights). The grant is silently dropped when a service is re-registered, which is why "Access denied" starting the service after a post-build.ps1 is the pre-flight's own diagnosis for that failure.

8. Build the tests

---

build/build-tests.ps1 restores the NuGet packages into hmailserver\test\packages

• where the project's hint paths point, which a bare dotnet restore on the project does not do - and builds RegressionTests.dll for x64. Always x64: an AnyCPU build of the same project also runs, against stale assemblies, and passes tests it never actually executed.

RegressionTests.csproj is a legacy project that lists every source file explicitly. A new test file that is not added to it is not merely unrun, it is invisible, and a green suite says nothing about it. The pre-flight compares the directory against the project and names any orphan.

9. Run it

---

build/preflight-tests.ps1, then build/run-tests.ps1. The runner is nunit3-console; subsets take -Where "class =~ /RegressionTests.IMAP.Binary/". NUnit writes its failure details only at the end of the run, so a run that looks quiet for fifty minutes is normal.

Do not dotnet test this project. It exits 0 having run nothing.

Never build, run static analysis, or start other heavy work while the suite is running. A rebuild restarts the service under the tests, the fixtures' restart detector then fails every test that follows, and the run can neither convict nor exonerate the code - thirty minutes are simply thrown away.

10. A second bench, and the gate in halves

---

One bench runs one gate at a time, and a gate is the whole suite: about fifty minutes of tests once the build is done, and rising. Since 16 September 2026 a second Windows bench takes half of it. It is a Windows Server 2025 VM (192.168.11.150) that was provisioned entirely over SSH, and what it needed is the list of what any new bench needs beyond this document:

• Git, the .NET 10 SDK, the .NET Framework 4.8.1 developer pack and PowerShell 7 (all from winget), and the Visual Studio 2026 Build Tools with the C++ workload, ATL, the Windows 11 SDK and the MSBuild and managed-desktop workloads. The project's toolset is v145, so the 2022 Build Tools will not do; the bootstrapper is https://aka.ms/vs/18/stable/vs_BuildTools.exe (the /release/ names are dead).
• The prebuilt libraries under C:\Dev\hMailLibs - Boost 1.92 (only boost\ and stage\ are used), OpenSSL 4.0.3's out64 and PostgreSQL 18.6 - and the machine-wide hMailServerLibs variable pointing at them.
• Four things the repository does not carry, copied from a working bench: libraries\msado28, libraries\mariadb-connector-c-3.4.11, installation\Extras\7za.exe and the two Languages\*.ini files of the Release output. SQL Server Compact 4.0 from installation\SQLCE (also untracked).
• The release installers the internal suite checks an administrator's install of an update against - the x64 installers of 6.3.0, 6.3.1, 6.3.3, 6.3.4 and 6.3.5, 6.3.5's Sigstore bundle and Microsoft's .NET Desktop Runtime installer, about 475 MB, not in git - in C:\HMTest\Installers. Run build/fetch-installer-fixtures.ps1 from the checkout once, and again when its list changes: it fetches each from the project's GitLab releases (the runtime from Microsoft's download site) and keeps each installer only when its hash is the released file's; the bundle has no hash there, and is checked by the Sigstore verification of 6.3.5's installer that it is for. AdminUpdateTester and AdminUpdateStageTester, which the internal suite runs last, inside the service, look there, or in the folder a machine-wide HMAILSERVER_INSTALLER_FIXTURES names (a service may not see a new one before Windows restarts). Without them they fail, naming the script - a check whose test never runs would look like one that works - and with them the two tests that run the internal suite, Infrastructure.MainOperations.TestInternals and API.Events.AnOnErrorScriptRunsOutsideTheDatabaseTransactionItIsReportedFrom: S1-WINBENCH, set up on 13 October 2026 without them, failed windows-gate-s1vm twice on 17 October 2026 in TestInternals, the one of the two in its half of the suite, until the script was run there.
• ClamAV at C:\clamav, run as a scheduled task at start-up rather than from a shell - a process started from an SSH session dies with the session - and SpamAssassin as the SpamAssassinJAM service (the JAM build under Strawberry Perl, as on the first bench).

Then sections 1 to 8 as written: build.ps1, post-build.ps1, the layout of the output directory, the database created over COM, the password, the grants.

The two benches split the suite by namespace, chosen from the measured durations so that the halves are about equal (1,900 seconds each in the run that drew them): API, IMAP, AntiSpam, Sieve, POP3, Rules, Domains, MIME, Stress, Installation and Shared on one, Security, Infrastructure, SMTP, AntiVirus, SSL and ExternalAccounts on the other, each through run-tests.ps1 -Where with a class pattern, started together and reported together. Both halves are the gate; a run that reports one is not one. The same split, by the same rule, ran the Linux suite on GitHub as two matrix halves of the regression-on-linux job, after the whole suite grew past that job's ninety-minute limit.

Since 25 September 2026 GitLab CI runs the suite on the second bench itself for master, batch* branches and release tags: the windows-gate job, build/ci/windows-gate.ps1, ran the whole of it in one run until a third bench, S1-WINBENCH, took half. Since then windows-gate runs every test but the six namespaces above on bench150, and windows-gate-s1vm those six on S1-WINBENCH, at the same time. On each bench the job and the split gate take one lock, C:\bench\gate.lock, so that neither checks a commit out under the other's running suite (ContinuousIntegration.md). The split gate's eleven-namespace half leaves out the fixtures outside RegressionTests (ServerFreeTests), and runs the two [Explicit] stress fixtures under Stress, which a run of the whole suite skips: NUnit runs an [Explicit] fixture that a class pattern names. The job's half, the six namespaces' complement, does neither. GitLab CI runs the Linux suite whole as well, on one of the project's Linux runners.

A faster bench finds what a slower one cannot: the first split run found a fixture that waited for a restore by watching a stamp of one-second resolution and, on the new machine, missed a restore that finished inside the same second as the previous test's reinitialisation.

On Linux

RegressionTests.csproj is net481 with a COMReference to the service and runs nowhere but a Windows bench. hmailserver/test/LinuxRegressionTests is the part of the suite that can test the Linux server: a net10.0 project that compiles 339 of the suite's files in place from RegressionTests, 299 of them fixtures holding some 2,150 tests, together with the socket simulators, and stands a REST-backed fixture layer under them (LinuxRegressionTests/Shims). The fixtures are not copied and not edited; a test that passes here is the same test that passes on Windows. It began at 52 files - the ones that drive the server through SMTP, IMAP, POP3 and MIME and nothing else - and grew as the shim layer learned to answer more of what COM answers. build/ci/linux-coverage.sh runs it against a server built for coverage (ContinuousIntegration.md, The coverage schedule).

What the fixture layer does instead of COM: the test domain example.test is deleted and recreated before every test through POST /api/v1/domains (or, on a server built before that route existed, emptied of its accounts and lists); accounts, aliases and distribution lists go through their routes; a server setting a fixture writes goes through PUT /api/v1/settings or its anti-spam and logging groups, and the suite's own defaults are put back before every test the way the Windows PerformBasicSetup puts them back through COM; the localhost SMTP routes go through /api/v1/routes; the TLS listeners the SSL fixtures need are made from the certificate and port routes and brought up with POST /api/v1/server/reinitialize; the delivery queue is drained through /api/v1/queue; and the ERROR log is read through /api/v1/logs, judged by what was written during the test rather than by whether the file exists; the server's directories come from GET /api/v1/settings/directories, a [Settings] key of hMailServer.ini goes through /api/v1/settings/ini, the logon-failure list is cleared through /api/v1/settings/logon-failures/clear, and a fixture that restarts the service to have the ini read again gets POST /api/v1/server/reinitialize, which reads it again in the same process. Everything a test makes is removed in its teardown.

What it cannot do, it says: a fixture that needs the suite's fake DNS zone served to the server's resolver, or a COM-only call such as Utilities.EvaluateSieveScript, is skipped with that reason, from the shim, never passed with a weakened assertion. A shim that needs a route also asks the server's own OpenAPI document whether it has it, so an older server skips with the route named rather than failing. dotnet test reports those as skipped; a failure is the server's. The REST API also refuses a credential's 201st request in any ten-second window, and the fixture layer keeps under that on its own.

The server it runs against is any hMailServer with the REST listener on, on any host; the defaults are the PostgreSQL-backed tree the CI job lays out (build/ci/linux-suite.sh, run by the linux-suite job of .gitlab-ci.yml; before that, the GitHub workflow linux-build.yml's regression-on-linux, in the git history at v6.3.3), which is also the recipe for a hand-made one: DBScripts from the checkout beside the binary, dh2048.pem from installation/Extras, an INI with Type=PostgreSQL and RestApiPort=8045, --create-database, the SMTP, POP3 and IMAP listeners moved off the privileged ports with an UPDATE of hm_tcpipports (the submission listener keeps its stock 587 where the server may bind it - the CI job runs as root in its container, which also lets the suite serve its DNS zone on port 53; a bench run as an ordinary user moves it, to 5870 on the lab VM, and names it in HMTEST_SUBMISSION_PORT, the port API/RestApiSetupChecklist asks about), the auto-ban switched off with an UPDATE of hm_settings (AutoBanOnLogonFailureEnabled to 0 - the suite fails logons on purpose, and three of them ban the test host for an hour; the Windows setup switches it off over COM), and --set-admin-password from a pipe. The tests are told where it is through the environment, read once when the assembly loads:

Variable	Default
HMTEST_HOST	127.0.0.1
HMTEST_SMTP_PORT	2525
HMTEST_POP3_PORT	1110
HMTEST_IMAP_PORT	1143
HMTEST_SUBMISSION_PORT	587
HMTEST_REST_PORT	8045
HMTEST_ADMIN_PASSWORD	testar
HMTEST_SERVER_INI	unset; the ini is searched for
HMTEST_NO_FAKE_DNS	unset; set to 1 to run without the fake DNS zone, so the DNS fixtures skip. Not needed for WSL2 or the hosted runner: the suite serves the zone on 127.0.0.1, or on 127.0.53.53 where that address is held or does not deliver, and proves delivery with a query before it points the server there

HMTEST_SERVER_INI is the one that is not a port. A few fixtures write a setting the COM API does not carry - an account-lockout threshold, a DNS server, an anti-virus failure policy, TLS-RPT - straight into hMailServer.ini, then reinitialise. On the Windows bench they find the file by searching upwards from the test binary into the repository's own Release output, which is where the service under test runs from; against a Linux server in a tree of its own there is nothing above the tests to find, and those fixtures fail before they begin. Set it to the ini the server was started with. Then, from any machine with the .NET 10 SDK that can reach those ports:

dotnet test hmailserver/test/LinuxRegressionTests -c Release --logger trx

Unlike the Windows project, this one is meant for dotnet test. Windows and Linux hosts both work - a WSL server is reachable from Windows through mirrored networking - and the CI job runs it on Linux against the binary the same workflow just built. Shared/TestPorts.cs is what lets the simulators be pointed elsewhere; the Windows suite never sets it and sees the standard ports.

Never both suites at once, on a machine with WSL

Run wsl --shutdown before starting the Windows suite. Not "stop the Linux server" - the whole VM. They are two servers on two operating systems and it looks as though they cannot collide. On a machine where WSL2 is in mirrored networking mode they share one port space, the sharing is invisible from Windows, and - this is the part that costs a run - the reservation outlives the process that made it.

A socket bound inside WSL takes that port from Windows as well and bind returns WSAEADDRINUSE, while the socket appears in neither Get-NetTCPConnection (in any state) nor netstat -ano nor Resource Monitor, because it is not a Windows socket. Every tool you would reach for reports the port as free. Worse, killing the Linux process does not give it back: with every hmailserver and dotnet inside WSL killed and ss -ltnp showing nothing whatsoever on those ports, Windows still could not bind them, and a connection attempt was refused - so nothing was listening and nothing could bind. Only wsl --shutdown released them, and it released all twelve at once.

Measured, rather than assumed, on 9 September 2026: a listener bound to 0.0.0.0:11000 from inside WSL, then [System.Net.Sockets.TcpListener] on the same port from Windows - AddressAlreadyInUse, with the Windows TCP table empty of it.

What that looked like in a run: nine tests failed with HM4316, "Failed to bind to local port", on ports 11000 and 14300 - the numbers Infrastructure.TCPIP.TestPortOpening moves POP3 and IMAP to - spread across StlsConformance, EtrnAuthorization, TlsCertificateValidity, ManageSieveTls and SieveAccountScript, all of which bind a port of their own and restart. Nothing in the code was wrong and nothing in the Windows tooling could see why.

So: one suite at a time, and wsl --shutdown between them. The check that actually answers the question, from Windows, is to try the bind rather than to look for a listener:

foreach ($p in 25000,25001,25002,25003,11000,11001,11002,11003,14300,14301,14302,14303) {
   try { $l=[Net.Sockets.TcpListener]::new([Net.IPAddress]::Any,$p); $l.Start(); $l.Stop(); "$p free" }
   catch { "$p TAKEN" }
}

All twelve free means the bench is ready. Any of them taken with the service stopped means WSL still has them, whatever ss says inside it.

The CI runners have neither problem - each job is its own machine - so this is a bench rule, not a workflow one.

When a run is interrupted

A run that is stopped part-way leaves state behind, and the next run fails in ways that point at the wrong thing. RELEASE.md step 4 says never abort a run for this reason; when it happens anyway, build/preflight-tests.ps1 -Clean removes what it can and names the rest:

• A stale ERROR log. One fixture writes a deliberate scanner error and removes it in teardown. Left behind, it fails every fixture's setup in the next run - the observed shape is 100% of tests failing before doing anything.
• Test-only settings left in hMailServer.ini. The suite points the resolver at its own fake DNS server on a loopback address for the whole run (127.0.0.1 on the Windows bench; the Linux suite's Shims/SuiteDns.cs tries 127.0.53.53 when that one is held or does not deliver) (Shared/SuiteDns.cs, a [SetUpFixture]: NODATA for every name a fixture has not added, the SURBL test point and localhost seeded) and puts DNSServer back when the run ends; fixtures that enable the quarantine, set a POP3 login delay, a password policy, PROXY-protocol trust, or an OTLP endpoint put their key back in teardown. Any of them surviving fails tests that never mention the setting: every DNS lookup waits for the query timeout; every anti-spam refusal becomes an acceptance; every connection from the trusted address is dropped before the banner. The server caches the INI at start, so after -Clean restart the service.
• Orphan domain directories in the data folder. Several persistence fixtures rename the test domain, and the rename moves its directory. Killed mid-rename, the directory survives with no domain row, and every later run of that fixture fails with "already exists" about a directory it never named.
• The twelve extra TLS ports described in step 4.

The environment traps

Each of these has produced a red run that looked like a code regression.

• Anti-virus products that proxy localhost mail traffic in-process. ESET, for one, answers an IMAP literal continuation itself with + just send it and rewrites raw SMTP framing. A failure whose text contains that string is the proxy, not the server, and it returns after a reboot. Disable mail and SSL protocol filtering for localhost.
• VPN clients that rewrite loopback. ProtonVPN and WireGuard adapters have broken address selection in fixtures that bind a specific local address. The pre-flight warns when one is up.
• Line endings. The repository checks out CRLF on purpose (.gitattributes): the database scripts are split on \r\n\r\n, the DKIM-signed .eml resources have body hashes that change with the line ending, and raw-message tests send file content verbatim and are refused for bare LF. A tool that "normalises" the tree to LF breaks all three at once.
• Mark-of-the-Web. Files downloaded rather than cloned carry a zone identifier that the service refuses to load. Strip it from the whole tree.
• "Passes alone, fails in a group." This points at shared state between fixtures - an account, a setting, a port left behind by the fixture that ran before - and not at the network, however network-shaped the failure message is. Look at what the preceding fixture in the run order changes.
• A build number that names a tree that no longer exists. Not an environment trap exactly, but the same lesson: the suite's verdict is about the binary the service is running, which is why the pre-flight checks the path and why RELEASE.md voids a gate the moment anything changes after it.