Skip to content

Storage and Persistence

Jump to bottom
Valerio edited this page Apr 25, 2026 · 3 revisions

Storage and Persistence

Harpyx uses four data stores, each chosen for the shape of the data it holds:

Store Holds
SQL Server Transactional business state (tenants, users, projects, jobs…)
MinIO Uploaded document blobs (S3-compatible object storage)
Redis Idempotency tokens, short-lived caches
OpenSearch Dense + keyword vector index for RAG chunks

SQL Server

The primary system of record. Accessed via EF Core through HarpyxDbContext in Harpyx.Infrastructure.Data. The connection string is composed from Database:* by HarpyxConfigurationComposer (see Configuration).

SQL Server transport encryption is enabled in the composed connection string by default (Database:Encrypt=true). Production deployments should use a trusted SQL Server TLS certificate and disable certificate bypass with Database__TrustServerCertificate=false. Transparent Data Encryption (TDE) is configured on SQL Server itself, not through EF Core migrations; see Data Encryption.

Domain entities

Located under Harpyx.Domain.Entities:

  • Identity & tenancy — Tenant, Workspace, User, UserTenant, UserInvitation, UserApiKey
  • Projects & content — Project, ProjectPrompt, ProjectChatMessage, Document, DocumentChunk
  • Jobs & audit — Job, AuditEvent
  • Billing & governance — Subscription, PlanLimit, PlatformSettings
  • LLM catalog — LlmConnection, LlmModel, UserLlmModelPreference (hosted/admin models plus user BYO cloud credentials, AES-256-GCM encrypted)

All entities derive from BaseEntity, which carries a Guid primary key and CreatedAt/UpdatedAt offsets.

Repositories

Repository contracts live in Harpyx.Application.Interfaces.IRepositories.cs; implementations in Harpyx.Infrastructure.Repositories. The IUnitOfWork abstraction coordinates transactional boundaries so services don't need to juggle the DbContext directly.

Migrations

EF Core migrations are kept in src/Harpyx.Infrastructure/Migrations. Two ways to apply them:

Explicit (recommended for production):

dotnet ef database update \
  --project src/Harpyx.Infrastructure \
  --startup-project src/Harpyx.WebApp

Automatic on startup (default in Development, useful for dev/ephemeral environments):

"Database": {
  "ApplyMigrationsOnStartup": true
}

When Database:ApplyMigrationsOnStartup is omitted, InitializeHarpyxDatabaseAndSeedAsync enables it only for Development. Production deployments should set it to false and run migrations explicitly.

InitializeHarpyxDatabaseAndSeedAsync in the WebApp runs at bootstrap; it also seeds configured platform admins from SeedOptions:AdminUsers.

MinIO (object storage)

S3-compatible blob storage, accessed through IStorageService / MinioStorageService. Used for:

  • Original uploaded document blobs, keyed by tenant/project/document-id.
  • Any large binary artifacts produced during ingestion (e.g. extracted images queued for OCR).

Credentials come from Minio:AccessKey / Minio:SecretKey; the endpoint and TLS toggle from Minio:Endpoint / Minio:UseSsl. The MinIO web console is exposed on port 9001 in the Compose stack for operator inspection.

Object encryption is an infrastructure concern. The fastest baseline is encrypted storage volumes for MinIO data; stronger deployments should enable MinIO server-side encryption with KES/KMS or use a managed S3-compatible service with default bucket encryption. See Data Encryption.

Redis

Consumed via StackExchange.Redis through IIdempotencyService / RedisIdempotencyService. Scope:

  • Idempotency — short-TTL keys that guarantee at-most-once processing of RabbitMQ redeliveries (see Jobs and Messaging).
  • Caching — transient caches where hitting SQL Server every request would be wasteful.

Redis is intentionally not used as a primary store; losing it loses only replay protection and some caches, never business state.

OpenSearch

Stores RAG chunks for hybrid retrieval. Managed by OpenSearchChunkIndexService:

  • Index name / alias — OpenSearch:IndexName is the physical index, OpenSearch:IndexAlias is the logical alias queries target. This indirection supports zero-downtime reindexing.
  • Embedding dimension — OpenSearch:EmbeddingDimensions must match the embedding model's output. Mismatches are caught at index creation time.
  • Security — HTTPS with basic auth (OpenSearch:Username / OpenSearch:Password). OpenSearch:AllowInsecureTls exists for self-signed certs in dev; always set it to false in production.
  • Timeouts — OpenSearch:RequestTimeoutSeconds bounds individual requests to avoid stalling ingestion on a degraded cluster.

Schema per chunk

Field Type Purpose
tenantId keyword Tenant scope filter
projectId keyword Project scope filter
documentId keyword Filter to a subset of project documents
chunkIndex integer Ordering within a document
text text Chunk body (keyword scoring)
keywords keyword[] Pre-extracted signals from KeywordExtractionService
embedding knn_vector Dense vector (dimension = EmbeddingDimensions)
page / offset integer Citation metadata

Data retention

  • SQL Server — the source of truth, retained for the life of the tenant.
  • MinIO — blobs are deleted when their Document is deleted or when a project's lifetime expires and ExpiredProjectsCleanupService purges it.
  • OpenSearch — chunks are removed by document id on document deletion; Project.RagIndexVersion allows a project to be rebuilt by indexing a new version and deleting the old one.
  • Redis — purely transient; TTLs are short by design.

Clone this wiki locally