Release Process

Release Process

Source of truth: RELEASE.md, kept in the repository and copied here. The diagrams, the gate table and the publishing section were added to this copy on 8 September 2026, drawn from RELEASE.md itself, .github/workflows/sbom.yml and sign-release.yml, and the refs/heads/master and refs/tags/v* rulesets. The release history they describe - which release shipped when, on what schema, behind what gate - is in Changes-Since-6210.

The order below is not advisory. Each rule exists because violating it has already cost a release cycle or nearly shipped a defect.

The release at a glance

Every box with a thick border is a gate: something that has stopped a release, or should have. A failure anywhere after step 7 sends you back to step 1, because the version is already stamped and the run that proves the binary is only valid for that binary.

flowchart TB
    F["1. Freeze the code"] --> REV["2. Adversarial review of the<br/>complete diff - BEFORE stamping"]
    REV --> FIX["3. Fix what survives.<br/>Every fix gets a negative-control test<br/>that FAILS on the pre-fix binary"]
    FIX --> PF["4. build/preflight-tests.ps1<br/>-Clean after an aborted run"]
    PF --> RM["5. build/check-roadmap.ps1<br/>tick boxes vs section counts vs contents"]
    RM --> DB["6. build/check-db-scripts.ps1<br/>+ build/check-schema-versions.ps1 (pwsh 7)"]
    DB --> STAMP["7. Version stamp:<br/>Version.h x3, section_setup_64.iss,<br/>seven .csproj - then git grep the old version"]
    STAMP --> BUILD["8. Build everything at the stamped version"]
    BUILD --> IDL{"hMailServer.idl<br/>changed?"}
    IDL -- yes --> REGEN["build/regenerate-interop.ps1<br/>AFTER the server build,<br/>AFTER any vtable-ordering fix,<br/>BEFORE the tools build"]
    IDL -- no --> REPRO
    REGEN --> REPRO["Second clean Release build.<br/>SHA-256 must match the first"]
    REPRO --> ASSERT["8b. Full suite on the<br/>-Asserts build. Expect no HM6364.<br/>Then rebuild plain Release"]
    ASSERT --> FUZZ["8c. Timed fuzz run on the release source<br/>30 minutes per harness"]
    FUZZ --> GATE["9. Full regression suite on the<br/>STAMPED binary. Nothing skipped.<br/>Any later change voids it"]
    GATE --> NOTES["10. Release notes - every claim<br/>checked against the diff"]
    NOTES --> INST["11. ISCC on hMailServer64.iss.<br/>Never run the installer here"]
    INST --> SHIP["12. Commit, PR, rebase-merge,<br/>signed annotated tag, draft release,<br/>SBOM, sign, publish"]
    SHIP --> LOOP["13. Close the loop on every issue"]

    style F stroke-width:3px
    style REV stroke-width:3px
    style PF stroke-width:3px
    style RM stroke-width:3px
    style DB stroke-width:3px
    style REPRO stroke-width:3px
    style ASSERT stroke-width:3px
    style FUZZ stroke-width:3px
    style GATE stroke-width:3px

Loading

Gate	Script or action	What it caught, at least once
2	Human review, per dimension, findings verified by skeptics	The 6.2.15 review found a BDAT truncation defect and a fix that turned transient file locks into permanent mail loss — after the installer was built
3	Negative-control test	One of the 6.2.15 abort tests passed on both builds until it was strengthened with session-count assertions
4	preflight-tests.ps1	A stale ERROR log from an aborted run fails 100% of the next run's tests in fixture setup
5	check-roadmap.ps1	Nine sections of Roadmap.md had drifted between three hand-edited restatements of the same 750 numbers
6	check-db-scripts.ps1	6.2.22-pre4 shipped an installer whose database could not be created at all; the only symptom was a service that started and did not listen
6	check-schema-versions.ps1	Missing until 6.2.23 — proves the upgrade chain is contiguous, forward-only and present for all four backends
8	Two clean builds, same SHA-256	A mismatch means something started embedding a timestamp or a path again despite /Brepro, /d1trimfile, /pdbaltpath and OPENSSL_NO_FILENAMES
8b	-Asserts build, HM6364	An assertion violation fails the test that provoked it and names the expression, file and line
8c	fuzz\run-fuzz.ps1	A crash, hang or violated assertion is a release blocker: minimise, fix, add the input to fuzz\regression, restart at step 1
9	The full suite on the stamped binary	The whole point: the verdict is about this binary, and any change after it voids the run

Order of operations

1. Freeze the code. No fixes land after this point without restarting at step 2.

2. Adversarial review of the complete diff — independent reviewers per dimension (delivery paths, protocol handling, UI truthfulness vs. code, docs claims vs. code), each finding verified by skeptics instructed to refute it. This happens before version stamping, not before tagging: the 6.2.15 review ran after the installer was built and both blockers it found (a BDAT truncation defect and a fix that converted transient file locks into permanent mail loss) forced rebuilds and a re-run of the full suite. Same review, an hour cheaper, if run first.

3. Fix what survives review; each defect fix gets a negative-control test. Build the pre-fix binary and confirm the new test fails against it. A test that passes on both builds proves nothing — one of the 6.2.15 abort tests did exactly that until it was strengthened with session-count assertions.

4. Pre-flight the bench: build\preflight-tests.ps1 (add -Clean to remove a stale ERROR log). Never skip this after an aborted run — an aborted run leaves a deliberate scanner error in the ERROR log and the next run fails 100% of tests in fixture setup, and it also leaves the TLS fixtures' twelve extra ports behind, which the pre-flight checks.

5. Check the roadmap against itself: build\check-roadmap.ps1. It reconciles every tick box in Roadmap.md against the per-section counts and the contents table, because three hand-edited restatements of the same 750 numbers drift — nine sections had drifted when the check was written.

6. Check the database scripts build a database: build\check-db-scripts.ps1. It creates a throwaway SQL CE database from CreateTablesMSSQL.sql using the same splitting rules the server uses, because nothing else here does - the regression bench's database is upgraded out of band, so no local test takes the CREATE path a fresh install takes. 6.2.22-pre4 shipped an installer whose database could not be created at all, and the only symptom was a service that started and did not listen.

   Run build\check-schema-versions.ps1 beside it (pwsh 7, not Windows PowerShell 5.1, where it dies in Measure-Object -Property To). It walks the registered upgrade chain and proves it is contiguous and forward-only, and that every step is present for every backend. That matters most on exactly the kind of release this checklist keeps being used for - one that moves the schema several steps past the last stable - and it was missing from this step until 6.2.23.

7. Version stamp: Version.h (version, numeric, build), section_setup_64.iss, all seven .csproj <Version> values. Verify nothing else still carries the old version: git grep <old-version>. In section_setup_64.iss the VersionInfoVersion fourth component stays 0: the build number lives in Version.h and the tag, and a build-only re-cut (the common case) must not have to touch the installer script or the seven .csproj files, whose versions are also <version>.0.

8. Build everything at the stamped version: build.ps1 -Configuration Release, build-tools.ps1 -Configuration Release, ControlPanel dotnet publish to its publish\ folder (build-tools does not cover it), build-tests.ps1. Confirm the stamped FileVersion on hMailServer.exe and publish\hMailCP.dll.

   If hMailServer.idl changed in this range, regenerate the checked-in COM wrapper after the server build and before the tools build: build\regenerate-interop.ps1. It runs TlbImp AND rewrites the wrapper's SHA-256 and size in hmailserver\docs\third-party-binaries.json - the binary-provenance workflow fails when those disagree, and regenerating by hand without the manifest did exactly that twice in one day. A stale wrapper still compiles, which is why this is easy to skip: the tools use a small stable subset of the API, so nothing fails, and the members added this release are simply invisible to them. Regenerate AFTER any interface-ordering fix, never before, or the old vtable layout is baked into the shipped wrapper permanently.

   Then prove the build is still reproducible: build Release a second time from clean (build\build.ps1 -Configuration Release -Clean) and compare the SHA-256 of hMailServer.exe with the first. They must be identical - /Brepro, /d1trimfile, /pdbaltpath and OPENSSL_NO_FILENAMES in the project make the executable a pure function of the source and the toolchain, and a mismatch means something has started embedding a timestamp or a path again. Put the hash in the release notes' verification section so anyone with the same toolchain (v145, Windows SDK 10.0.26100) and library layout can check the published binary came from the published source. Both builds restart the service, so do this before step 9, never after.

8b. Full regression suite on the assertion build, first. Build with build\build.ps1 -Configuration Release -Asserts - the same source, with every HM_ASSERT kept and a violated one reported as HM6364 in the ERROR log instead of compiled out - and run the whole suite on it. The ERROR log is checked before each test, so a violation fails the test that provoked it and names the expression, file and line. Expect none. Then rebuild plain Release (step 8) before step 9: the assertion build is the dynamic-analysis build and is never the binary that ships or is hashed.

8c. Timed fuzz run on the release source — required for every minor release. On the build from step 8: fuzz\run-fuzz.ps1 -Target mime_message_fuzzer -Minutes 30, and the same for each other harness listed in hmailserver/docs/Fuzzing.md. A crash, a hang or a violated assertion is a release blocker: minimise it, fix it, add the input to fuzz\regression, restart at step 1. Record the harnesses, the duration and the execution count in the release notes' verification line. A patch release may cite the previous minor's run when none of the fuzzed parsers changed; anything else runs again.

9. Full regression suite on the stamped binary — every test, nothing skipped. If anything changes after this run, the run is void: rebuild and re-run. Never abort a run; if one must be stopped, expect step 4 to fail and clean up before trusting any result.

10. README release notes — every claim checked against the diff. "Fixed" means reproduced-then-fixed or negative-control-tested; anything else is described as hardening or diagnostics. Unfixed known issues are named as unfixed.

11. Installer: ISCC on hMailServer64.iss. Never run the installer on the dev machine — validation is the CI smoke-test workflow (installer-smoke.yml), which installs it on a throwaway runner.

12. Commit (as chrisholloway5, no co-author trailers - history has been rewritten once to remove them, and will be again). master is protected: changes arrive by pull request, force-pushes and deletions are refused, and the maintainer's bypass exists for emergencies, not for the release flow. So push the working branch, open the PR and merge it with a rebase so the history stays linear and the commits keep their own messages:

    git push origin server-fixes-wave
    gh pr create --base master --head server-fixes-wave --fill
    gh pr merge --rebase            # self-merge is allowed; no reviewer is required
    git pull --ff-only origin master
    

    Then tag vX.Y.Z on master as an annotated, signed tag and push it (tags matching v* are protected too: no deletion, no rewrite - a tag is spent once):

    git tag -s vX.Y.Z -m "hMailServer X.Y.Z"
    git -c gpg.ssh.allowedSignersFile=.github/allowed_signers verify-tag vX.Y.Z
    git push origin vX.Y.Z
    

    Signing is SSH-based (git config gpg.format ssh and user.signingkey pointing at the public key, set once in the repository's config), and the key must be listed in .github/allowed_signers - the signing workflow below refuses to sign anything for a tag that is lightweight or that does not verify against that file, so an unsigned tag stops the release before it has any assets. Every tag up to v6.2.23-alpha1 was lightweight; there was nothing to verify and nothing stopped a v* ref being moved.

    Then publish as a draft first - this repository has immutable releases enabled, so a published release refuses every further asset upload and would be stuck with whatever it was created with:

    gh release create vX.Y.Z <installer> --draft --prerelease \
       --title "..." --notes-file <notes>
    gh workflow run "SBOM" -f release_tag=vX.Y.Z          # SPDX + CycloneDX
    gh workflow run "Sign release artefacts" -f tag=vX.Y.Z  # LAST: signs what is attached
    gh release view vX.Y.Z --json assets                   # expect installer + 2 SBOMs + bundle
    gh release edit vX.Y.Z --draft=false                   # publish, now complete
    

    The installer's name is not cosmetic. It must be exactly hMailServer-<tag without the leading v>-x64.exe, with its .cosign.bundle beside it. The update checker in every running server builds that name from the tag and matches an asset on exact equality; a release named anything else reports "a new version is available" and then "there is nothing to download" to every installation in the field. This was broken once already, at v6.2.22-pre3, whose installer was hMailServer-6.2.22-x64.exe. The signing workflow now refuses a release that does not carry the expected name, so the check happens before publication rather than after somebody notices.

    Order matters twice over: the SBOMs have to be on before signing, because cosign signs whatever is attached when it runs; and everything has to be on before publication, because nothing can be added afterwards. 6.2.22-pre4 was published straight away and ended up with an installer and no SBOM at all.

    The two gh workflow run lines are required, not belt-and-braces. Saving a draft does not start a workflow: GitHub does not deliver release events for draft releases, which was measured here rather than assumed - creating the pre5 draft produced no run at all. The release: created trigger on both workflows therefore only covers a release published without a draft, which is the path that can no longer attach anything. Dispatching them by hand is the path that works.

    And a tag is spent once: a tag that has backed an immutable release cannot back another, even after that release is deleted. There is no re-cutting a broken release under the same version - it needs the next number.

13. Close the loop: answer every issue the release resolves (and close them), update the ones it does not resolve saying so plainly.

Publishing: draft, attach, sign, publish

Step 12 is the part of the process with the least room to recover, because two GitHub behaviours combine: a published release refuses further asset uploads, and a tag that has backed an immutable release can never back another, even after that release is deleted. Get the order wrong and the fix is the next version number.

sequenceDiagram
    autonumber
    participant M as Maintainer
    participant G as GitHub
    participant S as SBOM workflow
    participant C as Sign-release workflow

    M->>G: git push origin server-fixes-wave
    M->>G: gh pr create --base master --fill
    M->>G: gh pr merge --rebase
    Note over M,G: self-merge allowed, no reviewer required -<br/>rebase keeps history linear
    M->>M: git pull --ff-only origin master
    M->>M: git tag -s vX.Y.Z -m "hMailServer X.Y.Z"
    M->>M: git -c gpg.ssh.allowedSignersFile=.github/allowed_signers verify-tag
    M->>G: git push origin vX.Y.Z
    Note over G: tags matching v* are protected:<br/>no deletion, no rewrite - a tag is spent once
    M->>G: gh release create vX.Y.Z, the installer, --draft
    Note over G: a DRAFT delivers no release event.<br/>Nothing starts on its own
    M->>S: gh workflow run "SBOM" -f release_tag=vX.Y.Z
    S->>G: attach hmailserver.spdx.json + hmailserver.cyclonedx.json
    M->>C: gh workflow run "Sign release artefacts" -f tag=vX.Y.Z
    C->>G: verify-tag against .github/allowed_signers
    C->>C: refuse unless an asset is named exactly<br/>hMailServer-X.Y.Z-x64.exe
    C->>G: cosign sign-blob --bundle for every asset
    M->>G: gh release view vX.Y.Z --json assets
    Note over M: expect installer + 2 SBOMs + 3 .cosign.bundle
    M->>G: gh release edit vX.Y.Z --draft=false
    Note over G: published - and now immutable

Loading

The order is forced twice over: the SBOMs have to be attached before signing, because cosign signs whatever is present when it runs; and everything has to be attached before publication, because nothing can be added afterwards. 6.2.22-pre4 was published straight away and ended up with an installer and no SBOM at all.

What each of the three "you cannot undo this" rules costs

Rule	Where it comes from	What it costs to get wrong
A published release refuses asset uploads (HTTP 422)	Immutable releases enabled on this repository	The release ships incomplete — an installer with no SBOM, or no signature
A tag is spent once	The same setting, plus the refs/tags/v* ruleset (deletion and non-fast-forward blocked)	There is no re-cutting a broken release under the same version. It needs the next number
A draft delivers no release event	Measured, not assumed — creating the pre5 draft produced no run at all	The two gh workflow run lines are required, not belt-and-braces

The installer's file name is load-bearing

The asset must be named exactly hMailServer-<tag without the leading v>-x64.exe, with its .cosign.bundle beside it. UpdateChecker in every running server builds that name from the tag and matches an asset on exact equality, so a release named anything else reports "a new version is available" and then "there is nothing to download" to every installation in the field. That happened at v6.2.22-pre3, whose installer was hMailServer-6.2.22-x64.exe under a -pre3 tag. The signing workflow now refuses a release that does not carry the expected name, so the check happens before publication rather than after somebody notices.

Verifying what was published

Anyone can check both halves without trusting this project's word for either:

# the tag was signed by the key in .github/allowed_signers
git -c gpg.ssh.allowedSignersFile=.github/allowed_signers verify-tag v6.2.28

# the installer is the blob the workflow signed, from this repository's OIDC identity
cosign verify-blob \
  --bundle hMailServer-6.2.28-x64.exe.cosign.bundle \
  --certificate-identity-regexp '^https://github\.com/Progressiverobot/hmailserver/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  hMailServer-6.2.28-x64.exe

Signing is keyless Sigstore through GitHub OIDC and is recorded in Rekor; there is no long-lived private key to steal. What it does not give you is a SmartScreen-quiet download: the installer is not Authenticode-signed, and Windows does not read Sigstore bundles. Azure Trusted Signing is the named route to fixing that and has not been taken yet.

Two more things a reader can check for themselves:

• The reproducible build. With the same toolchain (v145, Windows SDK 10.0.26100) and the README library layout, a clean Release build of the tagged source produces a hMailServer.exe with the SHA-256 printed in that release's verification section — 9E89C1E3ABF4DD36560B0B5F3515BDA3DAB8B96F00EF35DA5376A9B60176BD15 for 6.2.28.
• The SBOMs. SPDX 2.3 JSON and CycloneDX JSON are attached to every release; the native dependencies (OpenSSL, Boost, libpq) are merged in by build/merge-native-dependencies-into-sbom.ps1, which then fails if any of their PURLs is missing.

Standing rules

• Prefer two small releases over one large one. 6.2.15 carried IMAP semantics, delivery changes, a connection-layer rewrite and a UI expansion in one tag; its worst defect lived in the interaction between two of those themes.
• A fix to a shared layer needs a dependents audit. Before changing behaviour in something like TCPConnection, enumerate who depends on the old behaviour (the BDAT reads depended on "any error ends the session").
• The public record never overstates. If it is not reproduced and negative-control-tested, it is not "fixed" — say "hardened", "instrumented" or "still open". Credibility with reporters is the project's scarcest resource.