Message Storage

Message Storage

Written on 26 September 2026 from hmailserver/docs/MessageStore.md, EncryptionAtRest.md and ObjectStorage.md in 6.3.4 (27 September 2026, schema 6104). Everything on this page is new in 6.3.4, and all three features are off until an administrator turns them on: storage compression and attachment de-duplication is schema 6098, mailbox encryption at rest schema 6099, and object storage has no schema step. A server that turns none of them on stores every message exactly as 6.3.3 did. Brought up on 5 October 2026 to 6.3.6 (6 October 2026, schema 6108), for what 6.3.6 changes on this page. 6.3.7, the latest release (17 October 2026, schema 6108), is not yet reflected here.

Every message in a mailbox is a file under the data directory, with its folder, flags, size and UID in the database - and until 6.3.4 that file was always the message exactly as it arrived, RFC 822 byte for byte. 6.3.4 puts one door in front of every place a stored message is read, sized, copied, moved or deleted, and behind that door three things can now change how the file is kept, without any client noticing:

Feature	What it does	Turned on	Off by default
Storage compression and attachment de-duplication	A message is stored deflated, and each large attachment is kept once for every message on the server that carries the same bytes	MessageStoreCompression	Yes
Mailbox encryption at rest	Every message stored in one account's folders is encrypted to a key of that account's own, which its password opens at sign-in	Per account, after the domain's recovery key	Yes, on every account
Object storage	The mailboxes' files are kept in an S3-compatible store or Azure Blob Storage, with the database still the index	MessageStoreBackend and the store's settings	Yes: the local file system

flowchart LR
    Q["The delivery queue:<br/>always plain RFC 822"] -->|"rules, Sieve, scanners,<br/>filters have had it"| P["Placed in a mailbox"]
    P --> C{"Compression on?"}
    C -->|"yes"| Z["Deflated; large parts<br/>held once as blobs"]
    C -->|"no"| N["As it came"]
    Z --> E{"Account encrypted<br/>at rest?"}
    N --> E
    E -->|"yes"| K["Encrypted to the<br/>account's public key"]
    E -->|"no"| L["Written to the local disk"]
    K --> L
    L --> O{"Object store named<br/>and in use?"}
    O -->|"yes"| S3["The committer puts it<br/>into S3 or Azure Blob"]
    O -->|"no"| D["Stays on the local disk"]

Loading

What a client sees: nothing different. IMAP FETCH, RFC822.SIZE, BODYSTRUCTURE, SEARCH, SORT and THREAD; POP3 RETR, TOP and LIST; the webmail and the REST API; the rules, the quota (which counts the original sizes) and a DKIM signature checked against the stored copy - all read the original bytes back through the one door. The queue is never compressed, encrypted or put in the store, so the event scripts, the virus scanners, SpamAssassin, the sandbox and the filter hook read what they always read (Stopping Viruses §11.6d).

Contents

• Compression, and attachments held once
• Mailbox encryption at rest
• Object storage: S3 and Azure Blob
• Backups
• What none of this does

Compression, and attachments held once

With MessageStoreCompression on, a message delivered to a mailbox, or put there by IMAP APPEND, is stored as a container: the message compressed with deflate, and each large MIME part - an attachment of at least MessageStoreDedupMinKB once decoded, 256 KB unless changed - cut out and kept once as a blob that every message carrying the same bytes shares. A message sent to forty people on this server with a 10 MB presentation keeps the presentation once. Both are rows of hm_settings, and take effect for the next message stored.

Where	What
The Control Panel	Maintenance → Performance, the Message store tab: the switch, the smallest part held once, and what the store holds and saves as the server last measured it (The Control Panel §5.3a)
The Control Deck	The two settings on the server settings form, and a Message store section on the dashboard
REST	message_store_compression and message_store_dedup_min_kb in /api/v1/settings; GET /api/v1/store for the last measurement, POST /api/v1/store/measure to take one now
COM	Settings.MessageStoreCompression, Settings.MessageStoreDedupMinKB and Settings.MessageStoreStatus

What is already stored is not rewritten by the switch. The server reads a store of either form whatever the setting says, file by file. The command line does the rest, run as the user the server runs as:

hmailserver --measure-store  [--account <address> | --domain <name>]
hmailserver --compress-store [--account <address> | --domain <name>] [--decompress]
hmailserver --recount-store

--measure-store changes nothing and says what compression would save - it is the one verb that runs beside a running server. --compress-store rewrites the stored files in place, each written beside itself, read back and compared byte for byte before it takes the name, and --decompress puts them back as plain RFC 822. The rewrite and --recount-store need the service stopped. On Windows hMailServer.exe takes the same switches.

The link rule (since 6.3.6, item 43). Every store command - --measure-store included - and every copy to or from an object store first walks the data folder, and stops (exit 1), naming it, on a junction, a mount point or a symbolic link that the administrators do not alone control. A link owned by BUILTIN\Administrators, SYSTEM or TrustedInstaller that no one else may change or remove - a junction an administrator made to keep the data, or a domain's folder, on another drive - is gone through, and the path it holds is held, every link on it asked about, and checked; a folder only the administrators may list, such as System Volume Information at a drive's root, is skipped. A link made in the data folder inherits the service's write rights, and one made at a drive's root those of Users: it is refused, naming who may change it, until it is given permissions only the administrators may change (icacls <link> /inheritance:r /grant:r *S-1-5-32-544:(F) *S-1-5-18:(F) *S-1-5-32-545:(RX) /L, and /setowner *S-1-5-32-544 /L where its owner is an administrator's own account). A store command also stops, before it reads or changes a file, when a message row names a file outside the data folder - a full path elsewhere, or a part that is .. - and names the message: a full path below where a junction of the data folder leads is in it, a legacy full path elsewhere is moved with the Data Directory Synchronizer; otherwise delete the message. On Linux a store command run as root runs as the service's account, or on as root over a data directory root owns (Installing on Linux; The Service Account, What an administrator's command writes). The commands also write nothing to the error log since 6.3.6: what they did, and why one failed, is on their own standard output and standard error.

How much it saves has not been measured on a real mailbox yet: the repository's document says so, and this wiki claims no figure. Run --measure-store on your own store and read the answer.

Things to know before switching it on:

• Message.Filename over COM still names the stored file, and with compression on that file is a container, not RFC 822. A script or tool that opens the path and reads it reads the container. Read the message through the API instead, or run --decompress before something outside the server must read the files.
• Only delivery and APPEND write the new form. A webmail draft, a message saved over COM, an import or the IMAP mirror writes plain; --compress-store rewrites them.
• A message larger than 64 MB is stored as it came, because a container is decoded whole in memory.
• Every stored container is checked - its structure, each blob's hash, and the SHA-256 of the whole - before a byte is served; a damaged one is refused and named in the error log (HM7350), never served as something else.
• An encrypted mailbox holds no part once: a blob shared between accounts cannot be under one account's key, so every part stays inside the message's own container.

Mailbox encryption at rest

A mailbox whose messages are stored encrypted to a key of its own. The key is the account's; it opens when the person signs in with their password and closes when they have gone (MailboxKeyIdleMinutes after the last use, 15 by default); mail that arrives while nobody is signed in is encrypted all the same, because encrypting needs only the key's public half. When a password is lost, the domain's recovery key - opened with a passphrase the administrator keeps and this server never stores - opens the key again. It is Dovecot's mail-crypt with per-user keys.

What it protects: a disk taken out of the machine, a snapshot of the volume, a backup archive, the data directory copied somewhere, the database dumped - once the account has signed in with its password after the switch. What it does not: the mail from the running server, which reads it while the key is open - that is what lets IMAP, POP3, the webmail, search and the rules work as they always have - nor from whoever runs the server, who could capture a password at sign-in. A person who needs the server itself not to be able to read their mail wants a zero-access mailbox instead (Encryption for People); the two are never on one mailbox.

Until the account's first sign-in with its password after the switch, the key is pending: held under this machine's own secret - on Windows the machine's DPAPI key, on Linux the key file .hmailserver-secret-key inside the data directory - so a copy of the data directory with the database, or the disk itself, opens every pending key. An account that only receives mail and never signs in with its password stays pending; the Deck's account card says so.

Switching it on, on the Control Deck or over REST - there is no Control Panel page, no COM property and no webmail card:

1. The domain's recovery key first: the domain page's Mailbox recovery key card, or PUT /api/v1/domains/{domain}/mailbox-recovery with a passphrase of at least 12 characters and a hint. Keep the passphrase where this server is not: it is the one thing between a stolen database and every encrypted mailbox of the domain.
2. Then the account: the account page's Encryption at rest card, or PUT /api/v1/accounts/{address}/mailbox-encryption with {"enabled": true}. From that moment every message stored in its folders is encrypted. The switch ends every session the account had - a mail client sees its connection drop and signs in again with the password.
3. Then what is already there: hmailserver --encrypt-store --account <address>, with the server stopped, rewrites mail stored before the switch.

Refused, with a sentence: an account whose domain has no recovery key; a zero-access mailbox; an account that signs in against a Windows directory; and an account with a TOTP second factor, because a keyed mailbox opens only with its own password and a second factor would shut every mail client out. For the same reason app passwords are refused for such a mailbox (Connecting a Mail Client). A mailbox with a key stays in its domain: a move to another domain is refused.

Passwords: the person's own change re-wraps the key. An administrator's reset re-wraps it while the key is open or pending, and is refused while it is closed - a password set then would open nothing - with a sentence naming the recovery: the account card's Recover the mailbox, or POST /api/v1/accounts/{address}/mailbox-encryption/recover with the domain's passphrase and the new password. A lost password and a lost recovery passphrase together are the mail lost; nobody can open it. DELETE .../mailbox-encryption?confirm=forfeit then removes the key, and with it every message still encrypted to it, for ever - and is refused while a legal hold covers the account (Compliance and eDiscovery).

Switching it off stores new mail as it comes; what was stored encrypted stays encrypted until hmailserver --decrypt-store (server stopped, the domain's passphrase in the environment variable HMAILSERVER_RECOVERY_PASSPHRASE, never on the command line) puts it back.

What stays in the clear, the whole list: the queue and the copy a delivery works on (overwritten with zeros once placed - which a copy-on-write filesystem, a snapshot or an SSD may not honour); the calendar and the address book; the quarantine, the archive and journal copies, a forward; and the full-text index holds none of an encrypted mailbox's words, so search reads every message while the key is open.

Object storage: S3 and Azure Blob

The messages in the mailboxes - with the parts held once and the recover-deleted store's copies - can be kept in an S3-compatible store (AWS S3, MinIO, Wasabi, Backblaze B2's S3 API, Ceph RGW), signed with AWS Signature Version 4, or Azure Blob Storage, with Shared Key or a SAS. The database stays the index, and an object is named by the path its file had. It is how a store grows past one disk.

Nothing changes until a store is named. MessageStoreBackend is file as shipped, and a server that never names a store reads and writes exactly as before.

Every write is a local one. A message is placed on the local disk exactly as without a store, and a background committer puts it into the store afterwards and only then removes the local file. So a delivery never waits for the store and never fails for it, and every read looks at the local disk first, then at the store. When the store cannot be reached, IMAP answers NO [UNAVAILABLE] and POP3 -ERR [SYS/TEMP] for a message only the store holds, and the client tries again; what waits to be committed stays on the local disk, and HM7473 says once an hour how much. TLS to the store is always verified (MessageStoreObjectCaFile names a private CA).

Turning it on:

1. Make the bucket or container - this server makes none - with a key allowed to get, put, delete and list, and, for S3, the multipart upload calls.
2. Name it, with message_store_backend still file: MessageStoreObjectService (s3 or azure), MessageStoreObjectEndpoint, MessageStoreObjectRegion, MessageStoreObjectBucket, MessageStoreObjectAddressing, MessageStoreObjectPrefix, MessageStoreObjectAccessKey and MessageStoreObjectSecret or MessageStoreObjectSasToken - rows of hm_settings, set over REST (/api/v1/settings) or on the Control Deck's Settings > Server form. There is no COM member and no Control Panel page. The secret and the SAS are protected, never answered back, and left out of a backup.
3. Copy the store into it: hmailserver --copy-store-to-object-storage, which runs beside the server, verifies every object and goes on where it stopped when run again.
4. Switch new mail to it: message_store_backend to object, taken within five seconds.
5. Remove the local copies when ready: stop the server and run the copy again with --remove-source, which removes each local file whose object it has verified.

Going back is the same the other way: message_store_backend to file, then hmailserver --copy-store-from-object-storage, then, once it says nothing failed, message_store_object_service to empty.

Its requests were proved against MinIO and Azurite on the project's test lab. Its regression fixture needs such a store to run against, and on a bench with none it is skipped. Compression, parts held once and encryption at rest work over it unchanged.

Backups

• A backup with messages copies the data directory as always - containers, the Blobs directory and encrypted files as they are - and, with an object store, reads every message and blob that is only in the store into the archive, so the archive holds every message whichever backend holds it; a store that cannot be read fails the backup.
• A full restore rebuilds the blobs' counts from the stored files before any blob is deleted, and a mailbox restore stages the blobs its messages need.
• An encrypted mailbox's key travels with its account, and the domain's recovery key with its domain, so a restore opens the mail exactly as before - except a pending key, which on Windows opens only on the machine that wrote it and is recovered with the domain's passphrase elsewhere.
• A restore onto a server that never turned compression on reads the containers the archive brought: the reader never asks the setting.

See Backup and Restore.

What none of this does

• No figure for the saving: compression has not been measured on a real mailbox yet.
• No zstd: deflate is the one stage; zstd's id is reserved.
• No protection from the running server or its administrator: that is zero access, not encryption at rest. An account's key is never rotated; the domain's recovery key can be replaced.
• No encryption at rest for calendars, address books, the queue, the quarantine or the archive.
• No webmail card for encryption at rest, and a passkey sign-in to the webmail cannot open the mailbox: a keyed mailbox signs in with its password.
• No instance role, workload identity or Azure AD token for the object store: the key or the SAS the settings hold. One committer, one connection per request.
• Message.Filename over COM, and a script's view of it, names a path that holds a container, or - with an object store - no file at all. Read the message through the API.
• The Import Tool and the data directory synchronizer walk the local disk: copy the store back first when they must see all of it.

The full documents, with every setting, error code and what proves each claim: MessageStore.md, EncryptionAtRest.md and ObjectStorage.md.

See also: Encryption for People · Compliance and eDiscovery · Backup and Restore · The Control Deck · The COM API and Scripting · Security Hardening