Skip to content

Grouped Files

Jump to bottom
Petrus Pradella edited this page Jul 5, 2026 · 7 revisions

Grouped Files

What this page covers: opening the grouped-file backend with Storages.createGroupedFile, the key-major on-disk layout (one file per key, holding every collection that shares that key — the inverse of Local Files' one-file-per-entity layout), why it's ideal for "everything about one entity-root in one file" workloads, the fact that — like local files — it accepts a JSON or YAML codec, how queries work as a correct-but-slow full scan (no real index, yet still validating index declarations), crash-safe atomic writes, and the two things it can't do — no transactions, no index acceleration.

📌 Note — switching to grouped files is a one-line change at construction. From storage.repository(...) onward your code is identical to every other backend. See Choosing a Backend for the full capability matrix.

The 30-second version

import br.com.finalcraft.everydatabase.*;
import br.com.finalcraft.everydatabase.codec.JacksonYamlCodec;
import br.com.finalcraft.everydatabase.modules.groupedfile.GroupedFileConfig;

import java.nio.file.Paths;
import java.util.Optional;
import java.util.UUID;

// 1. Describe the entity once — key type FIRST, entity type second.
EntityDescriptor<UUID, PlayerData> PLAYERS = EntityDescriptor.builder(UUID.class, PlayerData.class)
        .collection("PlayerData")                              // becomes a sub-node in each key file
        .keyExtractor(PlayerData::getUuid)
        .codec(new JacksonYamlCodec<>(PlayerData.class))       // YAML or JSON — both allowed here
        .build();

// 2. Open the backend — just point it at a directory.
GroupedFileStorage storage = Storages.createGroupedFile(
        new GroupedFileConfig(Paths.get("playerdata")));
storage.init().join();                                         // creates the base directory if absent

// 3. Use it — exactly like any other backend.
Repository<UUID, PlayerData> repo = storage.repository(PLAYERS);
UUID id = UUID.randomUUID();
repo.save(new PlayerData(id, "Alice", 100)).join();           // writes playerdata/<uuid>.yml
Optional<PlayerData> alice = repo.find(id).join();            // -> Optional[Alice]

storage.close().join();

createGroupedFile returns the concrete GroupedFileStorage type. The PlayerData entity is the same plain Jackson POJO used throughout the wiki (Quick Start).

📌 Note — every I/O call returns a CompletableFuture. .join() is shown for brevity; compose with thenApply / thenCompose in real code. There are no blocking variants — see The Async API.

Configuration: GroupedFileConfig

A single constructor — just the base directory where the per-key files live:

import java.nio.file.Paths;

new GroupedFileConfig(Paths.get("playerdata"));
Argument Type Meaning
baseDirectory Path root directory; each key becomes one file directly under it

There are no formatting knobs: the container format follows the codec (below), and JSON output is always indented for readability.

On disk: one file per key (key-major)

Where Local Files is collection-major (<base>/<collection>/<key>.json — one file per entity), grouped files invert the layout to key-major: one file per key, directly under the base directory, holding every collection that shares that key as a sub-node:

playerdata/
  _schema/
    migrations.json                # reserved: applied migration versions (isolated in a sub-dir)
  5f1e8400-e29b-41d4-a716-446655440000.yml
  9a2c….yml

Each key file is a single structured document, one top-level node per collection:

# playerdata/5f1e8400-….yml
PlayerData:                        # collection 1
  uuid: "5f1e8400-e29b-41d4-a716-446655440000"
  name: "Alice"
  score: 100
AuthMe:                            # collection 2, same UUID key
  passwordHash: "…"

Collections only co-locate when they share the same key space (the same key.toString()).

🧭 When this shines — a player whose data is spread across many logical collections, loaded on join and persisted on quit as a single read/write per key. One file is the whole entity-root.

The filename is the sanitized key plus the codec's extension (.yml / .json). FileKeyNames.safeStem handles path separators, case-differing names on case-insensitive filesystems, reserved Windows device names, and over-long keys: it appends a stable hash suffix (hash-truncating an over-long stem to a short prefix plus that hash), so distinct keys stay one file per key, portable, with the file and its per-key lock always colliding-or-not together. The key contract is otherwise the usual one — a stable, unique toString() of ≤ 255 characters — see Entities, Keys & Collections. Collection names must match ^[a-zA-Z][a-zA-Z0-9_]*$.

📌 Note — writes are crash-atomic. Each save is a read-modify-write of the whole key file, guarded by a per-key write lock (shared storage-wide so two collections of the same key never lose each other's update), then published via a sibling .tmp + ATOMIC_MOVE (falling back when the filesystem can't do an atomic move). A crash mid-write never leaves a truncated file — at worst an orphan .tmp. This guards against torn files, not power loss: the write is not fsynced, so the very last save can still be lost on a power cut if the OS hasn't flushed its page cache. Deleting a key's last remaining collection removes the now-empty file.

JSON or YAML — the format follows the codec

Like Local Files, grouped files treat the payload structurally, so this backend accepts a JSON or YAML codec. There is no format option — the container format is taken from the codec on the descriptor: JacksonJsonCodec yields .json files, JacksonYamlCodec yields readable .yml files.

import br.com.finalcraft.everydatabase.codec.JacksonJsonCodec;
import br.com.finalcraft.everydatabase.codec.JacksonYamlCodec;

.codec(new JacksonYamlCodec<>(PlayerData.class))    // -> playerdata/<uuid>.yml
.codec(new JacksonJsonCodec<>(PlayerData.class))    // -> playerdata/<uuid>.json (indented)

⚠️ Gotcha — all collections sharing one base directory must agree on one format. They write the same physical files, so mixing a JSON codec and a YAML codec under the same GroupedFileConfig fails fast with IllegalStateException. A codec that is neither JSON nor YAML (an opaque/binary codec) can't be embedded into the structured aggregate document and is rejected with IllegalArgumentException. Pick one format per base directory.

Grouped files and Local Files are the only two backends that accept YAML — SQL, Mongo, and in-memory require a JSON codec (codec.isJsonCodec()). See Codecs.

Queries: a correct full scan (no real index)

Like local files, grouped files have no real index. Queries are answered by a full scan — every key file is read, this collection's sub-node is extracted, and the entity is matched in memory. It's O(total keys) and slow on large datasets (the scan reads files of unrelated collections too), but it returns the correct result, so a query that works here keeps working when you swap to SQL or Mongo.

repo.findBy("score", 100).join();
repo.query(Query.range("score", 50, null)).join();   // score >= 50 (full scan, correct result)

Crucially, the backend still validates index declarations before scanning:

⚠️ Gotcha — querying a field that was not declared as an IndexHint throws IllegalArgumentException here too — even though the scan could answer it. This is deliberate: it stops a query that works on grouped files from silently breaking when the storage is swapped for SQL/Mongo (which genuinely require the declaration). Declare the field with .index(IndexHint.<type>("...")) or @Indexed. See Indexing & Queries.

📌 Note — a scan reads each key file individually, so it is point-in-time consistent per file, not a whole-store snapshot. A key created or deleted while the scan runs may be transiently missing (or double-counted) — the scan sees each file as of the moment it reads it, never a frozen view of the whole directory. For an exact scan (e.g. a count() you'll act on), run it against a quiescent store.

No transactions

GroupedFileStorage does not implement tx.TransactionalStorage. There is no inTransaction.

⚠️ Gotcha — a single save is atomic (the .tmp + ATOMIC_MOVE write of the whole key file), and because one key file holds every collection of that key, writing several collections for the same key in one save cycle lands atomically together. But there is no multi-key transaction — you can't commit/rollback a group of different-key writes as a unit. If you need ACID across multiple entities, use a SQL backend (MySQL & MariaDB / PostgreSQL / H2) or Mongo with a replica set. See Transactions.

Optimistic locking isn't enforced either — a versioned descriptor's versions(...) reports 0 for existing keys (like H2), so it degrades to plain upsert. See Optimistic Locking and Choosing a Backend.

Schema migrations are supported

Unlike transactions, migrations work. GroupedFileStorage implements schema.SchemaAwareStorage; applied versions are tracked in a reserved _schema/migrations.json ledger under the base directory, and migrations are forward-only. The ledger lives in its own _schema/ sub-directory (never collides with a key file) and is written atomically (.tmp + move) — a truncated ledger would read back as "nothing applied" and re-run every migration over already-migrated data. Extend GroupedFileMigration and override executeOnStorage(GroupedFileStorage):

import br.com.finalcraft.everydatabase.modules.groupedfile.GroupedFileMigration;

class V1_SeedAdmins extends GroupedFileMigration {
    @Override public String version()     { return "001"; }
    @Override public String description() { return "seed admin profiles"; }
    @Override protected void executeOnStorage(GroupedFileStorage storage) {
        // mutate via repositories obtained from `storage`
    }
}

storage.register(new V1_SeedAdmins()).migrate().join();

MigrationContext.getNativeClient(...) also exposes the GroupedFileStorage and its base Path. See Schema Migrations.

When to pick grouped files

🧭 Decision — choose grouped files when your data is naturally key-major: many logical collections that all key on the same identity (e.g. a player UUID), and you want the whole entity-root in one human-readable file loaded and saved as a unit. One file per key (YAML or indented JSON), crash-atomic, zero ops. The trade-offs are the same as local files: no transactions, no real index (full-scan queries), no optimistic-locking enforcement — fine for small or cold datasets and single-server deployments. If you'd rather have one file per entity per collection, use Local Files; for large or write-heavy data, or anything needing ACID, prefer a SQL backend or Mongo.

A common pattern: ship small deployments on grouped/local files, let operators flip to MariaDB/Mongo for large ones and move the live data with Moving Data Between Backends — no code changes, source untouched.

Capabilities this backend supports

Capability Grouped Files
Transactions ❌
Schema Migrations ✅ (tracked in _schema/migrations.json)
Indexing & Queries ⚠️ full scan (no real index)
Optimistic Locking ❌ (degrades to upsert)
Change feed ❌ (no push feed in v1; poll)
Persistence Durable (one file per key)

See also

EveryDatabase · Home · made by Petrus Pradella

EveryDatabase

🏠 Home

Getting Started

Core Concepts

Working with Data

Backends

Manager Module

Operations

Advanced

Reference

Contributing

Clone this wiki locally