Repository navigation
Releases: cboxdk/sync
Release list
v0.9.0
Added
-
Outbox::refused()keeps a write the server processed and refused as abandoned under the server's status, so a refused create goes on holding back its children and the refusal outlives the push that received it.dismiss()takes a dismissed create's dependants along asparent_abandoned, andrequeue()refuses areceipt_prunedorprotocol_violationwrite - one that may already be on the server - unless toldevenIfItMayHaveLanded. -
Engine::submit()answers likeprocess()and says whether this call wrote the mutation, so a host that writes its own table for a landed write does not repeat that work for a replay that raced it. -
Views\CurrentStateView, for a rule that judges a record as it is now - a host permission check on its own row. A change such a rule does not show now is delivered as a removal, the id and nothing else, whenever the underlying window spans either side of it. Judging history by the current row sent the content of a row created and deleted since the cursor, and never told a previous owner that a row had left. -
A writer can decide its own conflicts.
Engine::process()takes an optionalOnConflict. WithOnConflict::Pull, a field the resolver would have preserved is refused instead: statuspull_required, and nothing is stored - no record change, no conflict group, no receipt, no acknowledgement, no commit.MutationResult::$conflictsnames the contested fields and$recordVersionthe version that carries them, and the writer sends the same mutation again, rebased on that version. Decisions the resolver makes itself - client wins, server wins, reject - are untouched, so a writer cannot use this to get around them.Mutation::rebased(),Outbox::rebase(). -
The outbox gives a device what it was missing:
requeue()anddismiss()for an abandoned write,nameOf()for what a created record was called, and a rewrite of the fields the application names as references when a created record is named, so a child created offline reaches the server pointing at its parent's real id. -
Contracts\OutboxStoregainsreplace(),resetAcknowledged(),nameOf(),dismiss()andqueued(), andhead()takes a space. -
Engine::recordTrusted()for a host's own writes - a model save - deciding create or update, the base version and the stream position inside the space lock. Deciding them before the lock made concurrent saves race for a position.
Upgrading
- Run
migrate()- orPdoSchema::forConnection($pdo)->install($pdo)from your own migration - once after upgrading. It adds the new receipt and stream columns and indexes, gives receipts from earlier releases their stream position, and on MySQL retypes identity columns toutf8mb4_0900_bin, which copies each table: run it in a maintenance window on a large installation. Until it has run, writes fail. - A device's
PdoOutboxStore::migrate()adds its new columns in place on first use - restart every process that uses the outbox after it has run, since one still running the old code writes rows the new one does not see; writes queued before the upgrade count as sent once. Back the file up first; downgrading is not supported. Outbox::queue()refuses a create whose handle the device already uses for a record in another space. An application that gave records in different tenants the same local id has to give each its own - a UUID.Outbox::abandon()runs in its own outbox transaction now; do not call it inside one of yours on the same connection.- A device keeps sending writes queued before the upgrade on the stream they were queued on, so their retries and
depends_onstill match.OutboxStoreimplementations outside this package need the new methods, and the PDO outbox gains anentity_idcolumn in place. - Format-2 payloads cannot be read by 0.8.x; a downgrade after writing is refused by name.
Fixed
- A duplicate delivery inside a host's transaction is answered from its receipt. On MySQL at REPEATABLE READ the host's snapshot predates the space lock, so a retry whose first copy had just committed was answered
sequence_behind; the device renumbered a write that had landed and then abandoned it. A position already used is now looked up again with a read that sees the latest commit, and so is a dependency. - An echo is the writer's own knowledge.
recordTrusted(..., echoOf:)records what the host's table made of a device's write and folds its versions into that write's answer, so the device's next edit does not conflict with its own write.asCreate:writes every field when the log has never held the record - a row that existed before it was synced. AnIf-Matchon a record the log does not hold fails the precondition instead of creating it. - A restored device is not stuck behind the pruned range.
settledUnknown()takes the server's acknowledged sequence; when the server is further along than the device, every write still queued on that stream is settled together and new writes go on after the server. It used to burn one position per write, so the device could write nothing new until it had crawled through the whole range. - A failed rollback no longer hides the failure that caused it (a deadlock ends the transaction on MySQL, and the rollback after it failed too); SQLite's busy and PostgreSQL's lock timeout count as contention; a space name is checked before its row is created, and a missing space row fails instead of locking nothing.
- Handing out a write and rewriting it for a parent's new name lock the row on MySQL and PostgreSQL device stores, so neither overwrites the other with the copy it read before.
- Installing the schema gives receipts from earlier releases their stream position (in
PdoSchema::install(), so hosts whose migrations call it get it too, and safe inside a migration's transaction), and a position that already has an answer - a stream and its receipts that disagree after a partial restore - is refused as that, not as a retryable duplicate. LedgergainsamendReceipt()andreceiptAt(), which reads a stream position's answer as the latest committed state by an exact lookup on a new unique(space, replica_id, sequence)index - so inside a host transaction it locks that one row. A replayed duplicate and a dependency on the device's last few writes are looked up this way when the host's snapshot predates them.- An echo moves the writer's answer to its own version only when nothing came between the write and its echo, and adds only the fields it actually wrote - otherwise the device's next edit, based on that answer, overwrote another writer's change unseen.
- The outbox knows which writes may already be on the server: one refused as
receipt_prunedorprotocol_violation, and any write with a sending that got no answer at all (OutboxStore::countSend(),countAnswer(),clearSends(),unanswered();Outbox::answered()for an answer that leaves the write queued). A sending answered in a way that proves it did not land (Outbox::answered(), for "sign in again") is accounted for, and a gap, apull_requiredor a processed refusal proves none did; a "busy" answer proves nothing, since a gateway's timeout can look the same.abandon()takesanswered: falsefor a write given up on without an answer, and now runs in its own outbox transaction. A write queued before this release counts as sent once, since nothing recorded whether it had been.requeue()refuses a may-have-landed write withoutevenIfItMayHaveLanded;dismiss()of such a create abandons the writes that need it asparent_unknownrather thanparent_abandoned- this device never learned the record's name either way.dismiss()looks up the one row (OutboxStore::abandonedOne()) and returns how many dependants it took along. A dismissed create is kept, unreported (adismissedflag on the outbox row), because it is what still says its handle was never named - until a new create for that record is queued or accepted: a write that needs a record whose create was abandoned or dismissed and never named is requeued only once that create is requeued or the record is named withOutbox::found($handle, $name)- every such record, not just one.found()accepts only a create this device abandoned, not queued again and not named, so it cannot move another tenant's writes.orphanReason()gives a push the right reason for a child (parent_unknownwhen its parent may have landed), dismissing such a parent relabels children already abandoned, andmayHaveLanded()answers for any abandoned write.OutboxStoregainssetReason(),abandonedCreate()(with an optional space),forget(),markDismissed(),forgetDismissedCreates(),createFor()(a record's create wherever it sits in the queue, by an index on a newkindcolumn) andrecordName();rekey()takes$creates. A record accepted under its own id is recorded under it, so nothing waits on an earlier refused create for it; checks about a record are made in its own space. A handle names one record on a device:Outbox::queue()refuses a create whose handle the device already uses for a record in another space (OutboxStore::handleSpaces()), because a reference carries no space and could be pointed at either; nor may a handle equal a server id - use UUIDs. A scope's rename moves abandoned and dismissed writes along with the queued ones, so a handle never ends up in two spaces that way.createFor(),abandonedCreate(),inFlight(),rekey()and the name lookups are served by indexes on every driver, and an abandoned write leaves the in-flight range, so draining stays linear; a scope's rename moves the names recorded under it too.Outbox::relatedBy()holds the application's references for calls not given them. - A restored stream is settled across every type on it (
OutboxStore::queuedOn()), and only by the answer that is still current - a late copy of it no...
v0.8.0
Added
Contracts\CommitObserver — the engine tells a listener that a space advanced, so devices no longer have to ask to find out. Polling alone makes the interval a straight trade between how stale the data may be and how much load every idle device puts on the server.
It carries the watermark and nothing else, on purpose. The log is per space, but authorization is per principal and per view: putting the changes in the signal would hand every listener everything written in that space, including the rows and fields a given reader may not see. The signal says there is something new, up to here; the reader then asks through the endpoint that knows who it is.
Called after the transaction commits, never inside it — a rollback must not announce a write that did not happen, and a replay or a refusal appends no commit so it signals nothing. A throw cannot unmake a commit that already happened, so the engine does not let one reach the caller either: failing a push for a durably stored mutation would only make the client retry, meet its own receipt, and be told the same thing again.
Delivery is at-most-once and unordered by contract, which is why a reader still polls on a slow timer. The signal makes sync prompt; the cursor is what makes it correct.
cboxdk/laravel-sync turns it into a SpaceAdvanced event, with webhook delivery over cboxdk/laravel-ssrf and cboxdk/laravel-webhook-signature.
v0.7.0
Fixed (breaking)
A record could be dropped from a bootstrap page. PHP compares two numeric strings numerically, so the in-memory store put '9' after '10' where a database puts it before — and compared '1e2' and '100' equal, which made the keyset filter treat one of two distinct records as already passed and drop it, while the page still reported itself finished. Entity ids are client input, so this was reachable on purpose. Identifiers are compared by bytes now.
putReceipt() reported every database error as "mutation identity already recorded." A deadlock, a dropped connection and a missing table all arrived as "already processed" — and a caller that trusts that answer drops the write and reports success.
Breaking: Store::prune() is on the contract. The commit log is the only thing here that grows without bound, and a host given the contract had no way to reach the pruning both adapters already implemented.
Breaking: beforeCommit() takes the space on both adapters. They differed before, so the fault-injecting store could only extend the in-memory one — and the rollback test had never run against a database on any driver. It now runs on all of them.
Breaking: commitDraft() without a draft, and OutboxStore::append() with a duplicate identity, are refused the same way on both adapters instead of being silent on one and a raw PDOException on the other.
Performance
A selective bootstrap page costs the page, not the space. The field index was unreachable: the query drove from the record scan and probed sync_fields once per record. Measured on a 32,000-record space: 331ms → 0.2ms.
The outbox is indexed. On a 5,000-deep backlog: queueing 4,955ms → 583ms, draining ~32,000ms → 957ms. That cost lands on the device.
FrozenBootstrapSessions no longer grows without bound.
Tests
The parity suite pages a hostile identifier set through both adapters one record at a time. It used a friendly alphabet before, which is why the ordering defect could hide in it.
v0.6.0
Added (breaking)
OutboxStore::rekey() renames the entity every queued mutation refers to.
A create carries a handle the device made up for itself, and the server answers with the name it gave the record. Everything queued behind that create still refers to the handle — left alone, each of those is a write to a record that does not exist.
Mutation identities are untouched: renaming what a write targets is not a new write, and giving it a new id would let the server apply it twice. Covered by a parity test that runs against both shipped stores.
Only the key is renamed. A field value holding the handle — a child carrying its parent's id — belongs to the application, and no store can know which of its fields are references, so Outbox::rekey() leaves that to the caller rather than doing it half way.
Breaking: an implementation of OutboxStore must provide rekey(). Both shipped stores do.
EntityKey::equals() and Mutation::withEntity() come with it — the rename needs both, and neither was expressible from outside.
v0.5.0
Fixed (breaking)
A conflicting mutation skipped the host's validator. The validator ran only for Applied, Partial and Noop, and a conflict still commits its draft — so a preserved candidate reached storage without the host ever seeing it. In the Laravel transport the validator carries the authorization re-check against the record as locked, so the one path this package exists for was also the one path where a principal whose permission changed between the outer check and the lock had their proposal preserved anyway.
Breaking: Store::commitsAfter() takes an optional $entityType. An implementation of the contract must accept it; the reference adapters and anything extending them already do.
Security
Stored payloads are decoded against a named list of the 28 classes the engine actually stores, rather than unserialize()'s default of allowing any class. Any row an attacker can write — a restored backup, the replica database on an end-user's device, SQL injection elsewhere in the host — was an object-injection chain against whatever that application had loaded.
Payloads now carry a format version, so a later change of encoding is detectable on read. Rows written before the tag still read.
Performance
A view bound to one entity type no longer pays for another type's writes. Measured on a catch-up over 2,000 commits across ten types with the view matching one: 255ms over twenty round trips, down to 7ms over two.
Added
EntityTypeView — every live record of one entity type in the space, queryable so the store pages it through an index.
migrate() reconciles an existing installation: missing columns and indexes are added, nothing is dropped or retyped. Verified on SQLite, MySQL 8 and PostgreSQL, including that a second run changes nothing.
v0.4.0
Thirteen defects from an external review, each reproduced as a failing test
before it was fixed. Several lose or strand writes; upgrade before shipping a
client.
The outbox was keyed by the wrong thing (breaking)
It was keyed by replica alone. Two consequences:
- A push naming one entity type drained the entire queue and submitted every
other type's writes as that type. At best the server refuses the fields and
the write is discarded silently; at worst the field names overlap and it
creates a wrong record. - The server keys an acknowledgement stream by space and replica, so a
device writing to a second space offered it a number that space had never
seen, was told it was a gap, and could not lower its own counter to recover.
OutboxStore::head(), acknowledged(), setAcknowledged() and pending()
changed signature; Outbox::resumeAfter() now takes the mutation whose answer
it is acting on.
The two stores disagreed
PdoStore::scanRecords() could not match a field that was never set — the SQL
asserted a sync_fields row exists, and a never-set field has none, while the
in-memory store matched it. A bootstrap that omits a record still advances its
cursor, so the delta never repaired the omission either.
The in-memory watermark was derived from retained commits, so pruning all
history rewound it to zero and the next mutation reused a sequence a client had
already consumed.
Capabilities that existed but could not be reached
Outbox::queue() had no dependency argument, so an offline write chain could
not be expressed and two edits to the same field conflicted with each other. A
bootstrap interrupted part-way could only be restarted, which the replica
correctly refuses as out of order; the pending continuation token is now
exposed so it can be resumed.
Pruning
prune() is atomic, and a read re-checks the retention horizon afterwards. A
prune landing between a reader's horizon check and its query removed commits the
reader never saw, and its cursor advanced past them as though they had been
delivered.
Verification
125 tests against the in-memory store, a store that shares no objects across
commits, SQLite, and both client states, on PHP 8.4 and 8.5; MySQL and
PostgreSQL in CI; plus the deterministic simulator and the concurrent-writer
experiment.
v0.3.0
A device can now survive being killed. Client state is durable, and a local
outbox owns the ordering and retry semantics the protocol requires.
Durable client state
Views\MultiViewClient kept everything in eight private arrays committed by
clone-and-publish, so a process that died lost all of it and had to bootstrap its
whole dataset again over whatever connection it had. Its knowledge now lives
behind Client\Contracts\ClientState: InMemoryClientState by default,
Client\Pdo\PdoClientState for real. The whole view suite runs against both from
the same fixtures.
Each page is applied in one state transaction — records, version watermarks,
tombstone watermarks, memberships and the cursor move together, because a cursor
that advanced without its records would claim progress the local data does not
have. A view reset deliberately keeps the canonical watermarks: that tombstone
watermark is what stops a stale page from resurrecting a record the client
already saw deleted.
The outbox, and the bug it exposed
Client\Outbox owns the four outcomes every client has to get right: done,
retry this exact identity, resume from here, give up on this one. Getting any
wrong is silent data loss or a queue that never moves again, so it is
implemented once rather than in every application.
A mutation's sequence is assigned when it is sent, not when it is queued.
Numbering at queue time looks harmless and is not: a write the transport refuses
has already consumed a number the server never receives, so the server waits for
it forever, every later write comes back as a gap for it, and the device is
wedged permanently with no way out. This was found by running a real client
against a real server — neither half's own tests could have shown it.
Also
FieldState's version and origin are now optional, with a documented meaning: a
record that reached a client through a projection carries neither, because the
origin names an actor the reader may not be allowed to see. The engine always
sets both on canonical state.
Breaking: MultiViewClient::__construct() takes an optional ClientState.
Verification
104 tests against the in-memory store, a store that shares no objects across
commits, SQLite, and both client states, on PHP 8.4 and 8.5; MySQL and
PostgreSQL in CI; plus the deterministic simulator and the concurrent-writer
experiment.
v0.2.0
Closes a hole where a bootstrap token was a bearer capability for the space it
named. Upgrade before putting any transport in front of this package.
A bootstrap page was served on the token's own authority
ViewSyncService::bootstrap() never validated the context, unlike
openBootstrap() and delta(). The token carries the space it reads, so the
caller's session was never consulted.
Two tenants that share a view definition — the normal case, since the space is a
separate axis and does not enter the filter signature — could read each other's
data by forwarding the token. An epoch rotation, the one tool for forcing every
client to reset, could not revoke a token already issued.
bootstrap() now takes the context the caller expects, the way delta() already
carries one in its cursor, and refuses any page whose context does not match.
Space, schema version and epoch all live inside the fingerprint, so one
comparison closes all three.
Breaking: the signature is now
bootstrap(CursorContext $context, ViewDefinition $view, BootstrapToken $token).
What a transport must bind
docs/security/threat-model.md now states what the engine cannot check. A
replica id arrives from the client and selects an acknowledgement stream, so
anyone who names another device's replica claims its sequence numbers — that
device's next push then fails terminally and its queued mutations are
unrecoverable. The same applies to client-chosen mutation ids. Bind both to a
stable account identifier, never a session id: the engine compares the stored
actor on replay, so an identifier that rotates at re-login turns every legitimate
retry into a protocol error.
It also states plainly that a view filters rows and not columns, and that each
field's origin names the actor who wrote it — so a record serialized without an
explicit field whitelist discloses both values and authorship.
Fixed
PdoStore resolves its PDO handle per call through an overridable
connection() rather than capturing it. A host whose framework reconnects
underneath a long-lived store would otherwise open the transaction on the new
connection while the writes went to the dead one, with the rollback rolling back
nothing — partial persistence, no error raised.
Verification
91 tests against the in-memory store, a store that shares no objects across
commits, and SQLite, on PHP 8.4 and 8.5; MySQL and PostgreSQL in CI; plus the
deterministic simulator and the concurrent-writer experiment.
v0.1.1
Fixes two defects that made MySQL and PostgreSQL unusable. v0.1.0 worked on
SQLite only — upgrade before using either.
MySQL: the migration could not run
The schema used CREATE INDEX IF NOT EXISTS, which SQLite and PostgreSQL accept
and MySQL does not, so installation failed on the first index. Indexes are now
declared inside CREATE TABLE IF NOT EXISTS on MySQL, which is idempotent as a
whole, and as separate statements elsewhere. Identity columns are bounded at 150
characters on MySQL so the widest composite index stays well inside InnoDB's key
limit.
PostgreSQL: every read came back corrupt
Stored payloads are now base64-encoded. PHP encodes private and protected
property names with NUL bytes, and a PostgreSQL text column cannot hold those.
SQLite and MySQL only worked because they are permissive about it — the kind of
accident a single-driver test suite never reveals.
This changes the on-disk payload format. v0.1.0 could only write data on
SQLite; any such database has to be rebuilt.
Also
- PHP 8.5 static analysis:
$argvis read through$GLOBALSwith a guard, and
last-element access on a list usescount() - 1rather than
array_key_last(), which is typed as possibly returning null. - The shipped test trait migrates a shared database once and empties its tables
per test, instead of dropping and recreating seven tables each time.
Verification
Verified on PHP 8.4 and 8.5 against SQLite, MySQL 8.4 and PostgreSQL 17: 88 tests
on each driver, the deterministic simulator producing identical results
everywhere, and six concurrent writer processes producing a gapless commit log
with every replica acknowledged on both real databases — the concurrency property
the change feed documentation had listed as unproven.
v0.1.0
Superseded by v0.1.1.
This release worked on SQLite only: the MySQL migration could not run, and
every PostgreSQL read came back corrupt. Do not use it.
First tagged release of the framework-independent sync engine.
What it does
One authoritative server, many offline replicas. Independent field edits merge;
competing edits to the same field are preserved as candidates rather than
silently overwritten, with full provenance, and resolved explicitly against a
version.
Durable storage
Persistence\Pdo\PdoStore runs on SQLite, MySQL 8+ and PostgreSQL, with no
runtime dependency beyond ext-pdo.
Every mutation takes its space's write lock as the first statement. That is what
makes the commit log gapless and its numbering agree with the order commits
become visible.
Scope
No HTTP transport, no wire format, no UI, no framework integration.
BUILD-STATUS.md carries the full list of limits.