Pronunciation: /haɪˈdreɪd/ ("hi-DRAYD")
A structure-first data engine for workloads that scale by namespace. One Swamp per tenant, user, device, agent, or other natural unit. Reactive, gRPC-native, single binary.
HydrAIDE organises data into Swamps — independent namespaces (think "shard with its own file, lifecycle and lock domain") that live in their own files on disk and load into memory only when accessed. You give every natural unit of your domain its own Swamp: per tenant, per user, per device, per agent, per crawled domain. There is no shared global table. There is no central coordinator. Each Swamp is reached in O(1) by deterministic hashing — the client computes where the data lives without consulting metadata.
Inside a Swamp, data is stored as typed key/value Treasures. The Go SDK lets you save and load native structs directly; the wire protocol is gRPC, so any language with a protoc-generated client can use HydrAIDE without an SDK. Every write emits a real-time event over a Subscribe stream — there is no separate pub/sub layer to operate.
HydrAIDE has powered Trendizz.com since 2024 — indexing millions of European websites and serving keyword search from a single server. Concrete numbers: 100K inserts in ~46 ms, 10K updates in ~3.75 ms, ~15.4 bytes per entry on disk on a Threadripper + NVMe. See benchmarks.
HydrAIDE was built because, for one specific workload, every off-the-shelf option ran out of room — for different reasons, all at the same time.
In 2021 we were indexing 2M+ European websites for Trendizz. Every word was stored in its own shard, and every shard listed the domains where the word appears — tens of millions of independent storage units, multiple terabytes on disk, on a single server with 128 GB of RAM and a sub-3-second search budget across the entire corpus. PostgreSQL deadlocked under the concurrent writes and slowed past tens of millions of rows. MongoDB and any RAM-resident engine wanted the working set in memory. Cloud and per-CPU-licensed engines were ruled out on cost. Disk B-trees were too slow for the latency target.
The shared mismatch was structural: every engine assumed the data lives in one logical place and is reached through a query language. Our data was already split into millions of independent units by nature, and the access pattern was open the right small store, do work, close it. So we stopped combining tools and built a single one.
Full origin story — what we tried, what broke, what we built → · Personal narrative on dev.to →
Two paths, depending on what you want.
Best for learning the SDK, prototyping, or kicking the tires before committing to anything. Brings up a local HydrAIDE with auto-generated TLS certs and a working Go example, in two commands.
git clone https://github.com/hydraide/hydraide
cd hydraide/docs/sdk/go/examples
docker compose up -d
make quickstartFrom here the example tree has 9 focused recipes (TTL queues, atomic patches, subscriptions, distributed locks, server-side filters, …) plus 3 reference HTTP apps (todo-api, url-shortener, multi-tenant-saas) with ready-to-import Postman collections. Pick one, run it, copy from it.
Works on Linux, macOS and Windows. On Windows, run from inside WSL2 with Docker Desktop's WSL2 integration enabled — the Makefile and the docker compose flow are identical.
→ More: example tree README, testing your own models
Best for staging, production, or anywhere you want HydrAIDE running as a long-lived service. No config files, no Docker required.
# Install the CLI
curl -sSfL https://raw.githubusercontent.com/hydraide/hydraide/main/scripts/install-hydraidectl.sh | bash
# End-to-end install: generates TLS cert, downloads the binary, registers
# the systemd unit, starts the service, and waits until it is healthy.
sudo hydraidectl init -i <your-instance-name>→ More: install guide, install quickstart, Docker install, hydraidectl user manual
- Per-namespace isolation at scale — millions of independent Swamps on a single server, each with its own lifecycle, lock domain and disk file. Natural fit for multi-tenant SaaS, IoT fleets, per-agent state, search indexes.
- Hot/cold tiering without a cache layer — a Swamp loads into memory on first access and evicts itself after a configurable idle window. No external cache, no invalidation logic.
- Indexes built on demand, not maintained forever — internal indexes for a Swamp (e.g. ordering by creation or update time) are built when a read or filter needs them, and discarded when the Swamp evicts. No persistent index files; no disk space spent on indexes nobody is currently using.
- Native subscriptions on every write — gRPC streams deliver insert/update/delete events in FIFO order. No Kafka, no Redis pub/sub.
- Server-side filtering and queries — AND/OR filter expressions, vector similarity, geographic distance and field-level inspection are evaluated on the server and streamed back. See Query engine.
- Atomic field-level patches —
PatchTreasuresmutates individual fields inside a typed MessagePack Treasure on the server, without a read-modify-write round-trip. See Structural MessagePack patch. - Append-only single-file storage — one
.hydfile per Swamp with automatic compaction. On a Threadripper 2950X + Samsung 990 PRO: 100K inserts in ~46 ms, 10K updates in ~3.75 ms, 10K deletes in ~1.66 ms, ~15.4 bytes per entry on disk. See benchmark results and run instructions. - Per-key FIFO locking — concurrent writes on different keys run in parallel; same-key writes are queued. Deadlocks are not possible by construction.
- gRPC-native, SDK-optional — the proto file is the source of truth. The Go SDK is a convenience layer; any language with protoc support can talk to a HydrAIDE server directly.
The features list above is what HydrAIDE does. Here's what to build with it:
| You're building | Why HydrAIDE fits |
|---|---|
| Multi-tenant SaaS | One Swamp per tenant. Isolation, GDPR delete, eviction — by construction, no row-level security to write or audit. See the multi-tenant-saas reference app. |
| IoT / device fleet state | A device's data only lives in RAM while it's talking. Idle = zero CPU, zero memory. Scales to millions of devices on one server. |
| Per-agent / per-user AI state | Native subscriptions on every write feed agent loops without Kafka. One Swamp per agent gives independent lifecycle and lock domain. |
| Real-time collaborative apps | The gRPC subscribe stream replaces a separate event bus. One binary, one wire protocol. See live-subscribe. |
| Search and keyword indexes | What HydrAIDE was originally built for. Powers Trendizz across millions of European websites today. |
Honesty up front, so you don't pick the wrong tool:
- Not an OLAP engine. No columnar storage, no cross-Swamp aggregation, no analytical query planner.
- No multi-key transactions across Swamps. Atomicity is per-key (and within a Swamp via patches). If you need cross-shard ACID, use Postgres or CockroachDB.
- No SQL surface. Filtering and queries are expressed via the gRPC API, not SQL. There is no dialect-compatibility shim.
- Not a drop-in replacement for relational schemas with enforced foreign keys. Integrity lives in the application layer.
- Not a hosted service. You run the binary. There is no managed cloud offering.
The features below are grouped to follow the order you'd want to read them in: the data model first, then how you work with the data, then runtime behaviour, then how it scales, then the wire protocol.
| Resource | ||
|---|---|---|
| 🗂️ | Database engine | Sanctuary/Realm/Swamp/Treasure model |
| 🧬 | Struct-first data model | Why your Go struct is the schema |
| Resource | ||
|---|---|---|
| 🔍 | Query engine | Server-side filters, vector and geo |
| 🪄 | Structural MessagePack patch | Atomic field-level mutations on the server |
| 🔄 | Reactivity & subscriptions | Real-time events on every write |
| Resource | ||
|---|---|---|
| 🔐 | Concurrency safety | Per-key FIFO locks, deadlock-free |
| 🛡️ | Business locks | Application-level distributed locks with TTL |
| 🧠 | Swamp lifecycle | Idle eviction from RAM, empty-Swamp removal from disk |
| Resource | ||
|---|---|---|
| ⚡ | Deterministic addressing | Swamp name → folder → island → server, all O(1) |
| 💾 | V2 storage engine | Append-only .hyd format with measurements |
| Resource | ||
|---|---|---|
| 🛰️ | Pure gRPC control | Wire protocol as the contract |
| Resource | ||
|---|---|---|
| 📥 | Go SDK install + upgrade | go get, version compatibility, troubleshooting |
| 📘 | Go SDK reference | Full API with code samples |
| 🔧 | hydraidectl CLI | Instance management, monitoring, migration |
| 🔄 | Migration guide | V1→V2 format migration |
| 🚀 | Go examples (runnable) | Quickstart, recipes, and reference apps with integration tests |
| 🧪 | Testing HydrAIDE models | How to test against a live instance, not mocks |
| 🤔 | Why we built it | The workload that broke every off-the-shelf database we tried |
| 📈 | Benchmarks | Raw measurements, methodology, run scripts |
Install the HydrAIDE Claude Code plugin to get three skills (Go SDK reference, server operations, conceptual explanations) and three slash commands ready to use:
/plugin marketplace add hydraide/claude
/plugin install hydraide
Five skills activate automatically when you ask a HydrAIDE-related question: hydraide-install-and-upgrade (install / upgrade entry point), hydraidego (Go SDK application code), hydraidectl (server operations), hydraide-data-ops (ad-hoc data migration / cleanup / import CLIs), and hydraide (conceptual "how does X work" router). Slash commands: /hydraide-new-model (interactive Profile/Catalog wizard), /hydraide-review (pitfall checklist code review), /hydraide-debug (diagnostic flow). Full breakdown of what the plugin does in Claude Code-friendly notes.
| Resource | ||
|---|---|---|
| 🤖 | Claude Code-friendly notes | What the plugin does, install, skills, slash commands |
| 📒 | CLAUDE.md |
Project-level guidance auto-loaded by Claude Code (architecture, conventions, build) |
| 📥 | hydraide-install-and-upgrade skill |
First-stop installer router: server (Linux or Docker), Go SDK, upgrades, V1→V2 migration |
| 📜 | hydraidego skill |
Go SDK reference: modelling, filters, patches, locks, subscriptions |
| 🛠️ | hydraidectl skill |
Operations reference: start/stop/restart, backup/restore, observe, compact, explore |
| 🧰 | hydraide-data-ops skill |
Ad-hoc data ops CLIs: migrations, bulk imports, mass deletes, orphan cleanup, restore |
| 🧠 | hydraide skill |
Conceptual / "how does X work" router into the feature docs |
If you're picking this up and something doesn't fit — the docs miss a thing, the install hits a wall, a model design isn't clicking, or the workload just doesn't quite map — say so. Open a Discord thread, file an issue, or email directly. Real people answer.
- 💬 Discord — fastest channel; drop a question any time
- 🐙 GitHub issues — bugs and feature requests
- 📧 peter.gebri@trendizz.com — direct line
Want to contribute? Start with Contributors and the Contribution Guide.
HydrAIDE is built by Peter Gebri, founder of Trendizz.com, and used in production at Trendizz since 2024.
📧 peter.gebri@trendizz.com · 🌐 hydraide.io
Licensed under Apache 2.0