Skip to content

DataEncryption

Jump to bottom
Valerio edited this page Apr 25, 2026 · 1 revision

Data Encryption

Harpyx is designed to process sensitive documents. A production instance should combine application authentication and authorization with encrypted data stores and backups for uploaded files, extracted text, RAG metadata, audit data, and configuration state.

This page describes the recommended minimum encryption layer for hosting a Harpyx instance. It is intentionally operational: it explains what to enable on SQL Server, object storage, disks, backups, and Harpyx configuration so that the deployment has a defensible baseline for data-at-rest protection.

Harpyx reads documents in plaintext during extraction, indexing, OCR, and chat-context generation. The goal of this layer is to protect against exposed disks, copied database files, leaked backups, stolen snapshots, and accidental access to raw storage media.

This page complements Security, which describes the application-level threat model and secret handling.

Recommended Minimum Layer

Layer Purpose Implemented where
SQL TLS Encrypt WebApp/Worker traffic to SQL Server Harpyx connection string + SQL Server certificate
SQL Server TDE Encrypt database, logs, and backups at rest SQL Server
Object-storage encryption Encrypt uploaded document blobs at rest MinIO/S3/volume/KMS
Backup encryption Keep exported SQL/object backups encrypted Backup tooling / storage provider
Application secret encryption Encrypt LLM API keys before persistence Harpyx (AesEncryptionService)

For most self-hosted installations, this is the fastest effective path:

  1. Enable encrypted SQL Server connections from Harpyx.
  2. Enable SQL Server TDE on the Harpyx database.
  3. Encrypt the MinIO/object-storage volume or bucket.
  4. Ensure SQL and object-storage backups are encrypted.
  5. Use a real random Encryption__MasterKey for Harpyx-managed LLM API keys.

Steps 2, 3, and 4 are infrastructure operations performed by the hosting operator on SQL Server, object storage, and backup systems.

1. SQL Server Connection Encryption

When Harpyx composes ConnectionStrings:Harpyx from Database:*, SQL Server connection encryption is enabled by default:

Encrypt=True

The setting is controlled by:

Database__Encrypt=true

Development environments may keep:

Database__TrustServerCertificate=true

because the local SQL Server container usually exposes a self-signed certificate.

Production environments should use a trusted SQL Server TLS certificate and set:

Database__Encrypt=true
Database__TrustServerCertificate=false

If a full connection string is supplied through ConnectionStrings__Harpyx, include equivalent TLS settings in that connection string.

2. SQL Server TDE

Apply Transparent Data Encryption on the SQL Server instance with a DBA/operator account that has sufficient privileges.

For a self-hosted SQL Server instance, the high-level flow is:

  1. Create or verify the server master key.
  2. Create a TDE certificate in master.
  3. Back up the TDE certificate and private key to secure offline storage.
  4. Create a database encryption key in the Harpyx database.
  5. Enable encryption on the Harpyx database.
  6. Monitor encryption progress.
  7. Test restore using the backed-up certificate.

Example:

USE master;
CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'replace-with-strong-password';

CREATE CERTIFICATE HarpyxTdeCert
WITH SUBJECT = 'Harpyx TDE Certificate';

BACKUP CERTIFICATE HarpyxTdeCert
TO FILE = '/var/opt/mssql/backup/HarpyxTdeCert.cer'
WITH PRIVATE KEY (
  FILE = '/var/opt/mssql/backup/HarpyxTdeCert.pvk',
  ENCRYPTION BY PASSWORD = 'replace-with-another-strong-password'
);

USE Harpyx;
CREATE DATABASE ENCRYPTION KEY
WITH ALGORITHM = AES_256
ENCRYPTION BY SERVER CERTIFICATE HarpyxTdeCert;

ALTER DATABASE Harpyx SET ENCRYPTION ON;

Check status:

SELECT
    db_name(database_id) AS database_name,
    encryption_state,
    percent_complete
FROM sys.dm_database_encryption_keys;

encryption_state = 3 means the database is encrypted. While encryption is still running, percent_complete shows progress.

Critical rule: do not enable TDE without backing up the certificate and private key. Without them, encrypted database backups may be impossible to restore.

Managed SQL services often expose TDE from the cloud portal or provider control plane. Prefer the managed option when available, and document where the key material lives.

3. MinIO and Object Storage Encryption

Harpyx stores original uploaded document binaries in MinIO-compatible object storage. The fastest baseline is encrypted storage for the MinIO data volume:

  • cloud volume encryption for managed disks;
  • BitLocker/LUKS or equivalent for self-hosted disks;
  • encrypted snapshots and backups.

This protects the object files at rest without requiring application changes.

For Docker Compose deployments, put MinIO's /data directory on an encrypted host path. Harpyx exposes this path through:

Docker__MinioDataPath=minio-data

The default value keeps using the Docker named volume minio-data. For production, point it to a host directory that lives on an encrypted disk or encrypted filesystem:

Docker__MinioDataPath=/srv/harpyx/minio-data

Recommended production flow:

  1. Provision an encrypted disk, partition, volume, or filesystem on the host.
  2. Mount it persistently, for example under /srv/harpyx.
  3. Create the MinIO data directory:
sudo mkdir -p /srv/harpyx/minio-data
  1. Set the production .env value:
Docker__MinioDataPath=/srv/harpyx/minio-data
  1. Start the production stack:
docker compose -f docker-compose.prod.yml up -d minio
  1. Confirm that MinIO is using the host path:
docker inspect harpyx-minio-1 --format '{{ range .Mounts }}{{ .Source }} -> {{ .Destination }}{{ println }}{{ end }}'

For stronger object-level protection, consider server-side encryption:

  • MinIO with KES/KMS for self-hosted deployments;
  • S3-compatible managed storage with default bucket encryption;
  • SSE-KMS where key separation and auditability matter.

The target state is that every object written to the Harpyx bucket is encrypted by default, without the WebApp or Worker having to set encryption headers manually.

4. Backups

Encryption is incomplete unless backups follow the same standard.

SQL Server:

  • TDE protects database backups created after TDE is enabled.
  • Keep the TDE certificate/private key backup separate from database backups.
  • Test restore on a separate environment before relying on the procedure.

Object storage:

  • Ensure bucket replication, snapshots, and exported backups preserve encryption.
  • If backups are copied off-host, encrypt the destination as well.
  • Store KMS/KES recovery material outside the MinIO data volume.

5. Application-Level Secret Encryption

Harpyx encrypts LLM API keys before persistence using AES-256-GCM through AesEncryptionService.

Required setting:

Encryption__MasterKey=<base64-encoded-32-byte-key>

Generate a key with:

[Convert]::ToBase64String([System.Security.Cryptography.RandomNumberGenerator]::GetBytes(32))

Rotating this key requires decrypting and re-encrypting existing encrypted API-key fields. Do not rotate it by simply changing the environment variable unless a re-encryption migration/procedure has been prepared.

What This Baseline Protects

The minimum layer protects against:

  • database files copied from disk;
  • SQL backups or snapshots copied outside the server;
  • MinIO data volumes or object backups exposed outside the host;
  • accidental leakage of raw infrastructure storage;
  • passive inspection of SQL traffic between Harpyx and SQL Server;
  • plaintext persistence of LLM provider API keys.

It does not fully protect against:

  • a compromised WebApp or Worker process;
  • a host-level attacker with access to process memory and mounted secrets;
  • an administrator who can query the live database through SQL Server;
  • plaintext document content while it is being processed for OCR, extraction, RAG indexing, or chat.

Production Checklist

  • Database__Encrypt=true
  • Database__TrustServerCertificate=false
  • SQL Server presents a trusted TLS certificate.
  • SQL Server TDE enabled for the Harpyx database.
  • TDE certificate and private key backed up securely.
  • SQL restore procedure tested with the TDE certificate.
  • MinIO/S3 data volume or bucket encryption enabled.
  • Object-storage backups and snapshots encrypted.
  • Encryption__MasterKey is a real random 32-byte key, not the template value.
  • Key material is stored in a secret manager or secure operator vault, not in Git.

Stronger Options

The minimum layer is intentionally practical. Deployments with stricter compliance or tenant-isolation requirements can add stronger controls later:

  • KMS-backed object encryption: use MinIO KES/KMS or managed S3 SSE-KMS so object encryption keys are managed and audited outside object storage.
  • Envelope encryption per tenant or document: generate data-encryption keys per tenant, workspace, or document and wrap them with a KMS-managed master key. This provides better isolation but requires application changes and key-rotation procedures.
  • Column-level encryption / Always Encrypted: useful for narrow fields such as high-sensitivity identifiers or secrets. It is not a good fit for searchable RAG chunks or document text that Harpyx must query and process.
  • Encrypted OpenSearch indices or encrypted volumes: required if OpenSearch stores chunk text or embeddings on disks outside an already encrypted volume.
  • HSM/Key Vault-backed master keys: keep master keys outside .env and container environments, with access policy, audit logs, and rotation workflows.
  • Confidential computing / hardened hosts: useful when the threat model includes host operators or compromised hypervisors, but significantly increases deployment complexity.

These options should be introduced deliberately. Each one affects backup, restore, reindexing, key rotation, incident response, and supportability.

Clone this wiki locally