Skip to content

Encryption at Rest

Jump to bottom
Sietse edited this page Sep 29, 2026 · 3 revisions

Encryption at Rest

Everything Galahad writes to disk is encrypted, with a key your own key manager holds.
Status ✅ Works, and on in every shipped library: libgalahad.so encrypts everything it writes
Verified 9.77 GiB written, restarted and read back byte exact, zero drift · load cost within measurement noise (580 MB block, Intel Xeon Gold 6342, local NVMe)
Needs a key: your key manager in production, or a key file to try it out · OpenSSL 3.0 (libcrypto.so.3)
Cipher AES-256-GCM

In plain words

Galahad saves the model's working memory to disk so it can be reused later. Those files hold whatever your users typed. On a shared machine, or a laptop that can be lost, that is a problem.

So every one of those files is encrypted, and so is the index that lists them. The key never lives in Galahad: your own key manager hands it over when Galahad starts, and Galahad forgets it when it stops.

It is exact. The text that comes back out is identical to the text that went in, byte for byte. Encryption changes what is on the disk, not what your model reads.

The everyday example

A hospital runs an assistant over patient notes. Without this, the notes sit in plain files on the server. With it, someone who steals the disk gets noise, and the key was never on that disk to begin with.

The basics

How it works

  1. Before Galahad saves anything, it encrypts it with AES-256-GCM.
  2. The key comes from your key manager when Galahad starts. It is never written to disk.
  3. When Galahad reads a block back, it decrypts it. The model gets exactly the same data.

How to use it

  1. Encryption is always on. There is nothing to switch on.
  2. Give Galahad a key before it starts: your key manager in production, or GALAHAD_KEY_FILE for a test.
  3. Start Galahad. Without a key it refuses to start, on purpose.

Giving it a key

⭐ There is nothing to turn on. The shipped libgalahad.so always encrypts. What you choose is where the key comes from, and you say so before merlin_init.

In production: your key manager

merlin_key_callbacks cb = {0};
cb.struct_size = sizeof cb;
cb.unwrap      = my_unwrap;     /* your KMS decrypts a stored key            */
cb.wrap        = my_wrap;       /* your KMS encrypts a new one               */
cb.available   = my_available;  /* 0 after you crypto-erase a key            */
cb.user        = my_context;    /* passed back to you unchanged              */
merlin_set_key_provider(&cb);
merlin_init(&cfg);     /* refuses with MERLIN_ERR_NO_KEY_PROVIDER otherwise */

Every callback returns 1 for success and 0 for failure. Your callbacks can wrap AWS KMS, HashiCorp Vault, or any key manager you use.

To try it out: a key file (32 random bytes, or 64 hex characters, chmod 600):

merlin_use_file_key_provider("/var/galahad/store.key", 1);   /* 1 = "I know this is unsafe" */

With vLLM or SGLang, set GALAHAD_KEY_FILE=/var/galahad/store.key instead. The library prints UNSAFE KEY PROVIDER ACTIVE every time: the key sits on the same machine as the data it protects.

⚠ Set the key source first. Without one, merlin_init returns MERLIN_ERR_NO_KEY_PROVIDER (reason GLH-E17) and nothing runs. Galahad never starts without encryption.

⚠ One rule for your callbacks: on failure, return 0. Never return 1 with zeroed or partial key material. Half a key looks like success and is worse than no key.

What it protects, and what it does not

✅ Protected
the saved memory encrypted
the index encrypted
the key never written to disk by Galahad
tampering a modified file is rejected before any data reaches you

⚠ Not protected
file sizes roughly follow token count, so they reveal approximate prompt length
access patterns file modification times show which entries are used most
memory this is encryption at rest; decrypted data is in RAM and VRAM while in use

⚠ If approximate prompt length is sensitive in your setting, include it in your threat model.

What it costs

Measured on a 580 MB block load, Intel Xeon Gold 6342, local NVMe:
plaintext 110.54 ms
encrypted 109.93 ms

Within measurement noise: the difference is under 1%.

Correctness

⭐ Byte exactness holds through encryption: 9.77 GiB written, restarted and read back with zero drift.

The cipher matches the NIST known-answer test vectors.

⭐ A corrupted or modified file is refused, not read. Galahad treats it as a miss and the model reads the text again.

Key rotation and erasure
rotation keys rotate without rewriting the data
erase a tenant's entries merlin_forget_tenant(tenant_id, 1, &removed) removes the tenant's entries and deletes its saved files
crypto-erase destroy the tenant's key in your key manager and let your available callback return 0 for it; the tenant's data can no longer be read

⚠ A crypto-erase makes data unreadable; the encrypted files stay on disk. If your obligation is physical destruction, delete them too (pass 1 to merlin_forget_tenant, or delete them at the storage layer).

See Tenant Isolation for tenants.

Health

uint64_t failures = 0;
merlin_key_failure_count(&failures);

Non zero means your key provider is failing. ⭐ Scrape this. A provider that starts failing halfway through a day is the failure most likely to be noticed late.

Troubleshooting

Symptom Cause Fix
MERLIN_ERR_NO_KEY_PROVIDER / GLH-E17 no key source merlin_set_key_provider (or merlin_use_file_key_provider, or GALAHAD_KEY_FILE) before merlin_init
merlin_key_failure_count climbing your KMS is refusing or timing out check the provider; Galahad will not invent a key
Vault works for a month, then fails everywhere the token expired renew it; make sure your provider renews its token

Related

Clone this wiki locally