Skip to content

Grouped Files

Jump to bottom
Petrus Pradella edited this page Jul 29, 2026 · 7 revisions

Grouped Files

What this page covers: opening the grouped-file backend with Storages.createGroupedFile, the key-major on-disk layout (one file per key, holding every collection that shares that key — the inverse of Local Files' one-file-per-entity layout), why it's ideal for "everything about one entity-root in one file" workloads, the fact that — like local files — it accepts a JSON or YAML codec, how queries work as a correct-but-slow full scan (no real index, yet still validating index declarations), crash-safe atomic writes, and the two things it can't do — no transactions, no index acceleration.

📌 Note — switching to grouped files is a one-line change at construction. From storage.repository(...) onward your code is identical to every other backend. See Choosing a Backend for the full capability matrix.

The 30-second version

import br.com.finalcraft.everydatabase.*;
import br.com.finalcraft.everydatabase.codec.JacksonYamlCodec;
import br.com.finalcraft.everydatabase.modules.groupedfile.GroupedFileConfig;

import java.nio.file.Paths;
import java.util.Optional;
import java.util.UUID;

// 1. Describe the entity once — key type FIRST, entity type second.
EntityDescriptor<UUID, PlayerData> PLAYERS = EntityDescriptor.builder(UUID.class, PlayerData.class)
        .collection("PlayerData")                              // becomes a sub-node in each key file
        .keyExtractor(PlayerData::getUuid)
        .codec(new JacksonYamlCodec<>(PlayerData.class))       // YAML or JSON — both allowed here
        .build();

// 2. Open the backend — just point it at a directory.
GroupedFileStorage storage = Storages.createGroupedFile(
        new GroupedFileConfig(Paths.get("playerdata")));
storage.init().join();                                         // creates the base directory if absent

// 3. Use it — exactly like any other backend.
Repository<UUID, PlayerData> repo = storage.repository(PLAYERS);
UUID id = UUID.randomUUID();
repo.save(new PlayerData(id, "Alice", 100)).join();           // writes playerdata/<uuid>.yml
Optional<PlayerData> alice = repo.find(id).join();            // -> Optional[Alice]

storage.close().join();

createGroupedFile returns the concrete GroupedFileStorage type. The PlayerData entity is the same plain Jackson POJO used throughout the wiki (Quick Start).

📌 Note — every I/O call returns a CompletableFuture. .join() is shown for brevity; compose with thenApply / thenCompose in real code. There are no blocking variants — see The Async API.

Configuration: GroupedFileConfig

A single constructor — just the base directory where the per-key files live:

import java.nio.file.Paths;

new GroupedFileConfig(Paths.get("playerdata"));
Argument Type Meaning
baseDirectory Path root directory; each key becomes one file directly under it

There are no formatting knobs: the container format follows the codec (below), and JSON output is always indented for readability.

On disk: one file per key (key-major)

Where Local Files is collection-major (<base>/<collection>/<key>.json — one file per entity), grouped files invert the layout to key-major: one file per key, directly under the base directory, holding every collection that shares that key as a sub-node:

playerdata/
  _schema/
    layout.json                    # reserved: the format, key spaces and fan-out of this directory
    migrations.json                # reserved: applied migration versions (isolated in a sub-dir)
  5f1e8400-e29b-41d4-a716-446655440000.yml
  9a2c….yml

Each key file is a single structured document, one top-level node per collection:

# playerdata/5f1e8400-….yml
PlayerData:                        # collection 1
  uuid: "5f1e8400-e29b-41d4-a716-446655440000"
  name: "Alice"
  score: 100
AuthMe:                            # collection 2, same UUID key
  passwordHash: "…"

Collections only co-locate when they share the same key space (the same key.toString()).

🧭 When this shines — a player whose data is spread across many logical collections, loaded on join and persisted on quit as a single read/write per key. One file is the whole entity-root.

The filename is the sanitized key plus the codec's extension (.yml / .json). FileKeyNames.safeStem handles path separators, case-differing names on case-insensitive filesystems, reserved Windows device names, and over-long keys: it appends a stable hash suffix (hash-truncating an over-long stem to a short prefix plus that hash), so distinct keys stay one file per key, portable, with the file and its per-key lock always colliding-or-not together. The key contract is otherwise the usual one — a stable, unique toString() of ≤ 255 characters — see Entities, Keys & Collections. Collection names must match ^[a-zA-Z][a-zA-Z0-9_]*$.

📌 Note — writes are crash-atomic. Each save is a read-modify-write of the whole key file, guarded by a per-key write lock (shared storage-wide so two collections of the same key never lose each other's update), then published via a sibling .tmp + ATOMIC_MOVE (falling back when the filesystem can't do an atomic move). A crash mid-write never leaves a truncated file — at worst an orphan .tmp. This guards against torn files, not power loss: the write is not fsynced, so the very last save can still be lost on a power cut if the OS hasn't flushed its page cache. Deleting a key's last remaining collection removes the now-empty file.

JSON or YAML — the format follows the codec

Like Local Files, grouped files treat the payload structurally, so this backend accepts a JSON or YAML codec. There is no format option — the container format is taken from the codec on the descriptor: JacksonJsonCodec yields .json files, JacksonYamlCodec yields readable .yml files.

import br.com.finalcraft.everydatabase.codec.JacksonJsonCodec;
import br.com.finalcraft.everydatabase.codec.JacksonYamlCodec;

.codec(new JacksonYamlCodec<>(PlayerData.class))    // -> playerdata/<uuid>.yml
.codec(new JacksonJsonCodec<>(PlayerData.class))    // -> playerdata/<uuid>.json (indented)

⚠️ Gotcha — all collections sharing one base directory must agree on one format. They write the same physical files, so mixing a JSON codec and a YAML codec under the same GroupedFileConfig fails fast with IllegalStateException. A codec that is neither JSON nor YAML (an opaque/binary codec) can't be embedded into the structured aggregate document and is rejected with IllegalArgumentException. Pick one format per base directory.

Since 1.2.0 the directory remembers. The format is written to _schema/layout.json on first use, so reopening a YAML directory with a JSON codec fails instead of quietly reporting an empty collection and starting a parallel set of .json files beside the .yml ones holding the data. A directory that predates the file has its format inferred from the extensions on disk and written down; one that already holds both formats refuses to open, listing how many files of each it found — that is the fingerprint of a mismatch that already happened, and only you can say which set to keep.

Grouped files and Local Files are the only two backends that accept YAML — SQL, Mongo, and in-memory require a JSON codec (codec.isJsonCodec()). See Codecs.

Queries: a correct full scan (no real index)

Like local files, grouped files have no real index. Queries are answered by a full scan — every key file is read, this collection's sub-node is extracted, and the entity is matched in memory. It's O(total keys) and slow on large datasets (the scan reads files of unrelated collections too), but it returns the correct result, so a query that works here keeps working when you swap to SQL or Mongo.

repo.findBy("score", 100).join();
repo.query(Query.range("score", 50, null)).join();   // score >= 50 (full scan, correct result)

Crucially, the backend still validates index declarations before scanning:

⚠️ Gotcha — querying a field that was not declared as an IndexHint throws IllegalArgumentException here too — even though the scan could answer it. This is deliberate: it stops a query that works on grouped files from silently breaking when the storage is swapped for SQL/Mongo (which genuinely require the declaration). Declare the field with .index(IndexHint.<type>("...")) or @Indexed. See Indexing & Queries.

📌 Note — a scan reads each key file individually, so it is point-in-time consistent per file, not a whole-store snapshot. A key created or deleted while the scan runs may be transiently missing (or double-counted) — the scan sees each file as of the moment it reads it, never a frozen view of the whole directory. For an exact scan (e.g. a count() you'll act on), run it against a quiescent store.

A key file nobody can parse

This is the one backend where a stored file can belong to no collection. A key file aggregates every collection sharing its key, so if the document itself will not parse, nothing inside it can be attributed: skipping it under-reports the collection being read, and counting it inflates every other collection in the directory.

Because neither number is true, the reads that answer with a number or a set refuse:

GroupedFile: cannot count 'ec_accounts': key file 'a3f1c2.json' is not a readable document,
so it belongs to no collection and any answer would be a guess. Use scanAll() to list every
unreadable file, then repair or remove it.
Read On an unparseable key file
count() throws IllegalStateException, naming the file
keys(...) throws IllegalStateException, naming the file
all() / query(...) skips it, logs a WARN — they hand back entities, so omitting one they cannot read is their contract everywhere
scanAll(...) reports it as a failed ScanRow — this is the diagnosis, run it when a count fails
find(key) / exists(key) throws for that key only; other keys are unaffected

⚠️ Gotcha — do not confuse this with a poisoned row: a key file that parses fine and declares the collection, whose payload does not decode. That row is unambiguously yours, so count() counts it, all() skips it, and count() != all().count() is the intended tell-tale. Only a file broken as a document triggers the refusal above.

💡 Tip — a boot-time count() on a grouped-file store can now fail. If your startup treats the count as infallible (a guard, a health check), decide explicitly what an unreadable store means to it — usually "refuse to start" is the right answer, since the alternative is running on data you cannot fully see.

No transactions

GroupedFileStorage does not implement tx.TransactionalStorage. There is no inTransaction.

⚠️ Gotcha — a single save is atomic (the .tmp + ATOMIC_MOVE write of the whole key file), and because one key file holds every collection of that key, writing several collections for the same key in one save cycle lands atomically together. But there is no multi-key transaction — you can't commit/rollback a group of different-key writes as a unit. If you need ACID across multiple entities, use a SQL backend (MySQL & MariaDB / PostgreSQL / H2) or Mongo with a replica set. See Transactions.

Optimistic locking isn't enforced either — a versioned descriptor's versions(...) reports 0 for existing keys (like H2), so it degrades to plain upsert. See Optimistic Locking and Choosing a Backend.

Schema migrations are supported

Unlike transactions, migrations work. GroupedFileStorage implements schema.SchemaAwareStorage; applied versions are tracked in a reserved _schema/migrations.json ledger under the base directory, and migrations are forward-only. The ledger lives in its own _schema/ sub-directory (never collides with a key file) and is written atomically (.tmp + move) — a truncated ledger would read back as "nothing applied" and re-run every migration over already-migrated data. Extend GroupedFileMigration and override executeOnStorage(GroupedFileStorage):

import br.com.finalcraft.everydatabase.modules.groupedfile.GroupedFileMigration;

class V1_SeedAdmins extends GroupedFileMigration {
    @Override public String version()     { return "001"; }
    @Override public String description() { return "seed admin profiles"; }
    @Override protected void executeOnStorage(GroupedFileStorage storage) {
        // mutate via repositories obtained from `storage`
    }
}

storage.register(new V1_SeedAdmins()).migrate().join();

MigrationContext.getNativeClient(...) also exposes the GroupedFileStorage and its base Path. See Schema Migrations.

When to pick grouped files

🧭 Decision — choose grouped files when your data is naturally key-major: many logical collections that all key on the same identity (e.g. a player UUID), and you want the whole entity-root in one human-readable file loaded and saved as a unit. One file per key (YAML or indented JSON), crash-atomic, zero ops. The trade-offs are the same as local files: no transactions, no real index (full-scan queries), no optimistic-locking enforcement — fine for small or cold datasets and single-server deployments. If you'd rather have one file per entity per collection, use Local Files; for large or write-heavy data, or anything needing ACID, prefer a SQL backend or Mongo.

A common pattern: ship small deployments on grouped/local files, let operators flip to MariaDB/Mongo for large ones and move the live data with Moving Data Between Backends — no code changes, source untouched.

Capabilities this backend supports

Capability Grouped Files
Transactions ❌
Schema Migrations ✅ (tracked in _schema/migrations.json)
Indexing & Queries ⚠️ full scan (no real index)
Optimistic Locking ❌ (degrades to upsert)
Change feed ✅ (OS watch service — sees edits made outside the app)
Key-major reads/writes ✅ KeyMajorStorage
Persistence Durable (one file per key)

Key spaces and fan-out (1.2.0)

One base directory tends to accumulate collections keyed by unrelated things — player UUIDs, account UUIDs, free-form cooldown ids. They share the directory but never share a meaningful key, so every scan reads files that cannot hold what it is looking for, and an accidental key collision puts two unrelated collections in the same file, behind the same lock.

A key space gives a group of collections its own sub-directory, its own listing and its own locks:

GroupedFileConfig config = GroupedFileConfig.builder(Paths.get("playerdata"))
    .keySpace("player",  "PlayerData", "AuthMe")
    .keySpace("account", "Accounts")
    .build();
playerdata/
  _schema/layout.json
  player/5f1e8400-….yml            # PlayerData + AuthMe for that UUID
  account/2b7f….yml
  cooldowns_….yml                  # a collection that declared none stays in the base

Collections are declared grouped by key space on purpose: co-location is what a key space means, and listing the members together makes a typo look wrong instead of quietly splitting one entity's file in two. Declaring none leaves the tree byte-for-byte as it was.

Fan-out handles the key space that is genuinely large — ten thousand files in one directory already slows listing down on NTFS:

.keySpace("player", GroupedFilePartitioner.hashFanout(2), "PlayerData")
// -> playerdata/player/10/c5/5f1e8400-….yml

flat() (the default), hashFanout(levels) — SHA-1 of the key, two hex digits per level, identical on every JVM and OS — and prefix(chars) for a tree you can navigate by eye. Point reads never scan: the path is computed, not searched.

⚠️ Where a file lives is a file operation, not a config change. Placement and fan-out are recorded in _schema/layout.json, and a configuration that disagrees with the record fails to open. Move the files first, once:

GroupedFileRelayout.relayout(config);      // then open the storage

It moves entries, not files — a key file holds collections that are staying put, so it is split rather than moved — prunes the bucket directories it empties, and does nothing on a second run.

Reading and writing a whole key at once (1.2.0)

Every collection of one key already lives in one file, behind one lock, so grouped files implement the KeyMajorStorage capability. Check for it with instanceof and fall back on backends that store collections apart:

if (storage instanceof KeyMajorStorage kms) {
    KeyBundle bundle = kms.loadKey(uuid, PLAYER_DATA, ECONOMY, HOMES).join();   // one parse
    PlayerData data = bundle.get(PLAYER_DATA).orElseGet(PlayerData::new);
    …
    kms.batchKey(uuid, b -> b                                                   // one atomic move
        .put(PLAYER_DATA, data).put(ECONOMY, eco).put(HOMES, homes)).join();
} else {
    PlayerData data = storage.repository(PLAYER_DATA).find(uuid).join().orElseGet(PlayerData::new);
    …
}

Beyond the saved I/O, the batch is the only version that is atomic: N separate saves can be interrupted between two of them and leave the key half-updated.

⚠️ Atomicity is per key and nothing more. Grouped files still do not implement Transactions — there is no rollback, no isolation beyond the per-key lock, and no way to span two keys. All descriptors must share a key space; ones that don't have no file in common, so the call is refused rather than quietly doing N reads.

See also

EveryDatabase · Home · made by Petrus Pradella

EveryDatabase

🏠 Home

Getting Started

Core Concepts

Working with Data

Backends

Manager Module

Operations

Advanced

Reference

Contributing

Clone this wiki locally