Storage and Persistence

Storage and Persistence

Harpyx uses four data stores, each chosen for the shape of the data it holds:

Store	Holds
SQL Server	Transactional business state (tenants, users, projects, jobs…)
MinIO	Uploaded document blobs (S3-compatible object storage)
Redis	Idempotency tokens, short-lived caches
OpenSearch	Dense + keyword vector index for RAG chunks

SQL Server

The primary system of record. Accessed via EF Core through HarpyxDbContext in Harpyx.Infrastructure.Data. The connection string is composed from Database:* by HarpyxConfigurationComposer (see Configuration).

Domain entities

Located under Harpyx.Domain.Entities:

• Identity & tenancy — Tenant, Workspace, User, UserTenant, UserInvitation, UserApiKey
• Projects & content — Project, ProjectPrompt, ProjectChatMessage, Document, DocumentChunk
• Jobs & audit — Job, AuditEvent
• Billing & governance — Subscription, PlanLimit, PlatformSettings
• LLM catalog — LlmConnection, LlmModel, UserLlmModelPreference (hosted/admin models plus user BYO cloud credentials, AES-256-GCM encrypted)

All entities derive from BaseEntity, which carries a Guid primary key and CreatedAt/UpdatedAt offsets.

Repositories

Repository contracts live in Harpyx.Application.Interfaces.IRepositories.cs; implementations in Harpyx.Infrastructure.Repositories. The IUnitOfWork abstraction coordinates transactional boundaries so services don't need to juggle the DbContext directly.

Migrations

EF Core migrations are kept in src/Harpyx.Infrastructure/Migrations. Two ways to apply them:

Explicit (recommended for production):

dotnet ef database update \
  --project src/Harpyx.Infrastructure \
  --startup-project src/Harpyx.WebApp

Automatic on startup (default in Development, useful for dev/ephemeral environments):

"Database": {
  "ApplyMigrationsOnStartup": true
}

When Database:ApplyMigrationsOnStartup is omitted, InitializeHarpyxDatabaseAndSeedAsync enables it only for Development. Production deployments should set it to false and run migrations explicitly.

InitializeHarpyxDatabaseAndSeedAsync in the WebApp runs at bootstrap; it also seeds configured platform admins from SeedOptions:AdminUsers.

MinIO (object storage)

S3-compatible blob storage, accessed through IStorageService / MinioStorageService. Used for:

• Original uploaded document blobs, keyed by tenant/project/document-id.
• Any large binary artifacts produced during ingestion (e.g. extracted images queued for OCR).

Credentials come from Minio:AccessKey / Minio:SecretKey; the endpoint and TLS toggle from Minio:Endpoint / Minio:UseSsl. The MinIO web console is exposed on port 9001 in the Compose stack for operator inspection.

Redis

Consumed via StackExchange.Redis through IIdempotencyService / RedisIdempotencyService. Scope:

• Idempotency — short-TTL keys that guarantee at-most-once processing of RabbitMQ redeliveries (see Jobs and Messaging).
• Caching — transient caches where hitting SQL Server every request would be wasteful.

Redis is intentionally not used as a primary store; losing it loses only replay protection and some caches, never business state.

OpenSearch

Stores RAG chunks for hybrid retrieval. Managed by OpenSearchChunkIndexService:

• Index name / alias — OpenSearch:IndexName is the physical index, OpenSearch:IndexAlias is the logical alias queries target. This indirection supports zero-downtime reindexing.
• Embedding dimension — OpenSearch:EmbeddingDimensions must match the embedding model's output. Mismatches are caught at index creation time.
• Security — HTTPS with basic auth (OpenSearch:Username / OpenSearch:Password). OpenSearch:AllowInsecureTls exists for self-signed certs in dev; always set it to false in production.
• Timeouts — OpenSearch:RequestTimeoutSeconds bounds individual requests to avoid stalling ingestion on a degraded cluster.

Schema per chunk

Field	Type	Purpose
tenantId	keyword	Tenant scope filter
projectId	keyword	Project scope filter
documentId	keyword	Filter to a subset of project documents
chunkIndex	integer	Ordering within a document
text	text	Chunk body (keyword scoring)
keywords	keyword[]	Pre-extracted signals from KeywordExtractionService
embedding	knn_vector	Dense vector (dimension = EmbeddingDimensions)
page / offset	integer	Citation metadata

Data retention

• SQL Server — the source of truth, retained for the life of the tenant.
• MinIO — blobs are deleted when their Document is deleted or when a project's lifetime expires and ExpiredProjectsCleanupService purges it.
• OpenSearch — chunks are removed by document id on document deletion; Project.RagIndexVersion allows a project to be rebuilt by indexing a new version and deleting the old one.
• Redis — purely transient; TTLs are short by design.