Skip to content

Payload Schema Evolution

Jump to bottom
Petrus Pradella edited this page Jul 15, 2026 · 1 revision

Payload Schema Evolution (everydatabase-manager)

What this page covers: evolving the shape of an entity's stored payload — renaming a field, splitting one into two, changing a unit — without a maintenance window and without keeping legacy fields on the POJO forever. A stored row written by an older build carries an int schemaVersion; registered steps upcast its raw JSON tree on read, and an optional boot sweep rewrites the whole collection in bulk.

📌 Note — this lives in the optional everydatabase-manager add-on (package br.com.finalcraft.everydatabase.manager.entityschema). The core is untouched. See Installation for the coordinates.

The third axis (this is the part people get wrong)

EveryDatabase has three orthogonal things called "version". They solve different problems, they compose freely, and an entity can carry any combination of all three:

Axis What it versions Where it lives Page
DDL / migrations the collection's structure (tables, columns) :core, ledger _schema_migrations Schema Migrations
Optimistic lock one row, to resolve concurrent writes :core, column lock_version Optimistic Locking
Payload schema one entity's field shape, to upcast old data :manager, field schemaVersion this page

The distinction that matters: a migration reshapes the container, an optimistic lock decides who wins a write, and a payload schema decides how to read a row written by yesterday's code. Payload evolution is per entity and lazy by default — a row nobody reads is never touched.

The 30-second version

import br.com.finalcraft.everydatabase.manager.entityschema.*;

// 1. The entity carries its own payload version — and MUST initialize it (see the gotcha below).
public class Account implements EntitySchema {
    private UUID id;
    private long balanceCents;                  // v2 renamed this from 'balance' (and changed unit)
    private int schemaVersion = EntitySchema.INITIAL_SCHEMA_VERSION;

    public int  getSchemaVersion()          { return schemaVersion; }
    public void setSchemaVersion(int v)     { this.schemaVersion = v; }
}

// 2. Register the chain once at startup: upcast v1 -> v2 on the RAW tree, before it binds.
EntitySchemaMigrations.register(Account.class, 1, node -> {
    long dollars = node.path("balance").asLong(0);
    node.remove("balance");                      // the POJO no longer has this field at all
    node.put("balanceCents", dollars * 100);
});

// 3. Wrap the codec so a decode runs the chain. The framework re-stamps schemaVersion for you.
EntityDescriptor<UUID, Account> ACCOUNTS = EntityDescriptor.builder(UUID.class, Account.class)
        .collection("accounts")
        .keyExtractor(Account::getId)
        .codec(EntitySchemaMigratingCodec.wrap(Account.class,
                new JacksonJsonCodec<>(Account.class), "id"))   // "id" = protected identity field
        .build();

A v1 row now reads back as a fully-shaped v2 Account. The step reads balance off the JSON tree, so the POJO is free to have dropped that field entirely — which is the whole point.

See it end-to-end in the tests: EntitySchemaMigratingCodecTest and EntitySchemaMigrationsTest.

Steps operate on the raw tree, not the POJO

An EntitySchemaStep is a functional interface over a Jackson ObjectNode:

@FunctionalInterface
public interface EntitySchemaStep {
    void upgrade(ObjectNode node) throws Exception;   // mutate in place; node -> fromVersion + 1
}

Each step upgrades the payload from its fromVersion to fromVersion + 1. Working on the tree before it binds is what lets you delete a legacy field from the class and still migrate it — a POJO-typed step would need the field to still exist.

What the framework owns, so a step can't break it:

  • schemaVersion — the runner re-stamps it after each step. Don't set it in a step.
  • The identity key field(s) you name in wrap(...) — snapshotted before each step, restored after. A step cannot re-key a row.
  • The optimistic-lock field, whatever it's named — added to the protected set automatically, so a step cannot unlock a row.

Both protections are proven by hostile-step tests (theIdentityFieldSurvivesAHostileStep, theCustomNamedLockFieldSurvivesAHostileStep).

⚠️ Gotcha — steps must be pure tree transforms: no I/O, no host-platform API, no shared mutable state. A step runs on decode threads, on flush/conflict-resolution threads, on cross-backend transfers, and — for an EAGER step — on a background boot sweep. A step that throws fails the decode of that one row (wrapped in EntitySchemaMigrationException) and leaves the stored row untouched, so the next read retries.

💡 Tip — not writing a field leaves it absent, so the POJO's own field initializer supplies the default at bind time. Tree steps get defaults for free.

Registering the chain

// LAZY (the default): upcast on read only.
EntitySchemaMigrations.register(Account.class, 1, step);
EntitySchemaMigrations.register(Account.class, 2, EntitySchemaMigrationMode.LAZY, step);

// EAGER: upcast on read AND make the collection sweepable at boot.
EntitySchemaMigrations.register(Account.class, 3, EntitySchemaMigrationMode.EAGER, step);

Registration is static and global to the process. The chain must be contiguous, starting at EntitySchema.INITIAL_SCHEMA_VERSION (1):

Mistake Result
A gap (register 1, then 3) IllegalStateException
The same fromVersion twice IllegalStateException
fromVersion below INITIAL_SCHEMA_VERSION IllegalArgumentException

Introspection: currentVersion(type) (the newest shape the code knows), hasChain(type), eagerTargetVersion(type), steps(type) (an immutable snapshot, not a live view — an unregistered type yields empty, never null). registerChain(type, steps) replaces a whole chain wholesale (an empty or null one is a no-op); clear() / clear(type) reset the registry.

LAZY vs EAGER, and the cascade cost

LAZY upcasts a payload only when it's read — a row nobody touches keeps its old shape forever, and that's usually fine. EAGER additionally asks for a boot-time sweep that rewrites every stale row.

⚠️ Gotcha — eager cascades backwards. A decoded payload must always reach currentVersion in one pass, so any row the sweep touches runs all its pending steps — the earlier ones and any LAZY steps registered above the last eager one. Declaring step N eager makes every step up to N effectively eager for the rows still behind it. This is a deliberate, documented cost.

wrap() — the codec seam

public static <V> Codec<V> wrap(Class<V> type, Codec<V> inner, String... protectedIdentityFields);
  • Returns inner unchanged only when type is not an EntitySchema.
  • An EntitySchema type is always decorated — even with no chain registered yet.
  • Throws IllegalArgumentException when inner doesn't expose an ObjectMapper (ObjectMapperAware); raw-tree ops need one. Works for JSON and YAML alike — the seam is the mapper, not the format.
  • encode is pure delegation: the write path is never migrated.

🧭 Decision — why always decorate, even with no chain? Because a chain registered after the descriptor was built would otherwise never run, silently leaving every row un-migrated. A wrapper with no chain costs one map lookup per decode, then delegates straight through. The alternative traded a lookup for a silent data bug. (chainRegisteredAfterTheWrapStillMigrates pins this.)

An upcast marks the entity dirty (both the IDirtyable and the @DirtyFlag flavors), so the migrated shape gets re-persisted by the next flush. A type with no dirty tracking still decodes migrated correctly — it just re-migrates on every read, since nothing ever writes the new shape back.

The uninitialized-schemaVersion trap

⚠️ Gotcha — a left-at-zero int schemaVersion is rejected, not guessed. A stored version below INITIAL_SCHEMA_VERSION (0 is exactly what an uninitialized int field persists) names no shape the chain can start from, so every later read of that row fails with EntitySchemaMigrationException rather than guess which shape it holds and corrupt it.

Initialize the field to one of exactly two things:

  • EntitySchema.INITIAL_SCHEMA_VERSION — this entity's shape is the original one; the chain runs from the start.
  • EntitySchemaMigrations.currentVersion(MyEntity.class) — for a brand-new entity whose shape is already the newest, so no step should ever run on it.

The row stays readable the moment the entity stamps its version correctly — the refusal is a fence, not corruption.

The eager sweep

EntitySchemaSweeper is not a second migration engine. Migration still happens inside the codec on decode; the sweep is a bulk read job that pulls every row through that codec (which migrates and marks it dirty) and writes the dirtied set back with WriteMode.UPDATE_ONLY — so it can never resurrect a row deleted concurrently. See CRUD Operations for both primitives.

SweepReport report = EntitySchemaSweeper.sweep(accountsManager, SweepOptions.defaults());

It is idempotent and crash-restartable (per-row version gating), single-runner (a lease, below), and advances its completion marker only when the whole collection was scanned without a failed row. It's a pure utility: you own the executor, the scheduling ("after boot", "one collection at a time"), and any freeze/kill-switch policy — wire the latter through SweepOptions.abortCheck(), polled at every batch boundary.

⚠️ Gotcha — sweeping a type without dirty tracking throws IllegalArgumentException. The sweep detects an upcast row through its dirty flag; with no flag it would rewrite nothing and still mark the collection complete. So an eager sweep requires the entity to implement IDirtyable or carry a @DirtyFlag field — see Caching Managers → Write-back. An example built on an entity without either does not run.

SweepOptions

Every knob has a default that suits an unattended boot sweep, so SweepOptions.defaults() is a valid choice.

Knob Default What it does
runnerId(String) a random UUID identifies this instance, so a lease can tell "mine, resume it" from "someone else's, stay out"
batchSize(int) 256 rows per batch; also the lease-renewal and abort-poll granularity (clamped to ≥ 1)
abortCheck(BooleanSupplier) never aborts polled per batch; true cuts the sweep short, leaving it resumable
logger(ManagerLog) ManagerLog.SILENT where progress and notices go
leaseMillis(long) 60_000 how long a claimed lease survives without a heartbeat
progressLogMillis(long) 5_000 minimum spacing between "still going" lines

SweepReport

Carries collection(), type(), targetVersion(), scanned(), rewritten(), conflicted(), skippedDirty(), skippedAhead(), failed(), markerAdvanced(), and a human-readable note(). A sweep that has nothing to do is cheap and says so via note() — "no eager step", "already at v<N>" (an O(1) boot), "contended (live lease held by another instance)", or "lease not claimed".

📌 Note — conflicted() counts rows a concurrent write beat the sweep to. Harmless: that write persisted the migrated shape anyway, and a lazy read heals whatever it didn't. These counters are telemetry; the rows on disk are correct either way.

EntitySchemaSweeper.isSweeping(storage, collection) tells you whether a sweep is scanning right now in this process. It's scoped to the storage instance, not just the collection name (two storages may hold same-named collections). It says nothing about a sweep in another process — that's what the lease is for.

The marker, and why it's only a hint

The sweep records progress in _entity_schema_sweeps, a framework-owned meta collection living in the reserved underscore namespace (like _schema_migrations) on the same backend as the data, keyed by the data collection's name. See Entities, Keys & Collections → the reserved namespace.

Its @OptimisticLock version doubles as a cross-instance CAS lease: only one instance claims the in-progress slot, and it heartbeats per batch. That works on backends that enforce the lock (MySQL/MariaDB, PostgreSQL, Mongo). On the others (H2, LocalFile, GroupedFile, InMemory) the CAS is advisory — but those are physically single-instance, so no second sweeper exists to race. See Optimistic Locking → the enforcement matrix.

⚠️ Gotcha — the marker is a hint, never authority. The lazy decode-time migration never consults it; per-row schemaVersion plus the chain remain the source of truth. The marker only lets a completed sweep skip a full re-scan on later boots. So a straggler — a row written by an old instance after the sweep finished — still heals on its next read.

Rolling deploys: the "written by a newer build" guard

While a rolling deploy has old and new instances live at once, an old instance can read a row written by a new one. Its decode already dropped the newer fields (the codec ignores unknown properties), so flushing it would permanently erase them while keeping the newer version stamp.

The write-back engine refuses that write (refuseAheadWrite, warning once per type): the entity stays dirty and cached, and that process is effectively read-only for that row until it's updated. See Write-Back & Conflict Resolution. EntitySchemaMigrations.isAhead(entity) / isBehind(entity) expose the same question directly. An entity that doesn't implement EntitySchema is never refused.

Logging: ManagerLog

The add-on's maintenance utilities (sweeps, write-back flush) log through a minimal seam — a functional interface, deliberately not the core's structured per-storage event log:

package br.com.finalcraft.everydatabase.manager.log;

@FunctionalInterface
public interface ManagerLog {
    void log(Level level, String message);
    ManagerLog SILENT = (level, message) -> { };   // the default everywhere
}

Route it to SLF4J, JUL, a plugin logger, or nowhere. The default is SILENT, matching the library's silent-by-default posture (Logging & Diagnostics covers the core's own event log). Implementations must be thread-safe — a sweep logs from its own worker thread.

SweepOptions opts = SweepOptions.builder()
        .logger((level, message) -> myPluginLogger.log(level, message))
        .build();

See also

EveryDatabase · Home · made by Petrus Pradella

EveryDatabase

🏠 Home

Getting Started

Core Concepts

Working with Data

Backends

Manager Module

Operations

Advanced

Reference

Contributing

Clone this wiki locally