OBJECT_SYNC channelSerial: clarify sequence-id opacity (RTO5a1/RTO5a6) and close the UTS coverage gap that hides a cross-SDK parsing bug

[sacOO7]

Summary

Two SDKs (ably-js and ably-java) parse the OBJECT_SYNC channelSerial with a regex that only accepts sequence ids matching [\w-]+. The spec (RTO5a1/RTO5a6) places no character-set constraint on the sequence id — the only structural requirement is the : separator. As a result, a perfectly valid serial such as seq.1:cursor is wrongly classified as malformed, which silently ends the sync early and can discard already-synced object state.

This has never fired in production because every sequence id the realtime service currently emits happens to stay within [\w-]. But the spec grants no such guarantee, the failure would be silent and fleet-wide if the server format ever changed, and — critically for this repo — the full UTS suite cannot detect the bug, so both buggy SDKs pass it.

This issue asks ably/specification to (1) make the sequence-id opacity explicit, (2) resolve the empty-sequence-id ambiguity, and (3) add UTS coverage that pins the correct behavior. Per-repo code fixes are included below so the linked SDK PRs can reference them.

What the spec says today

From specifications/objects-features.md (RTO5a family):

(RTO5a1) The channelSerial is used as the sync cursor and is a two-part identifier: <sequence id>:<cursor value>

(RTO5a3) If the sequence id matches the previously received sequence id, the client library should continue the sync process

(RTO5a4) The objects sync sequence for that sequence identifier is considered complete once the cursor is empty; that is when the channelSerial looks like <sequence id>:

(RTO5a5) An OBJECT_SYNC may also be sent with no channelSerial attribute. In this case, the sync data is entirely contained within the ProtocolMessage

(RTO5a6) If the channelSerial is present but malformed --- that is, it does not contain the : separator required by RTO5a1 and so cannot be split into a <sequence id> and a <cursor value> --- the client library must handle the OBJECT_SYNC as if the channelSerial were absent per RTO5a5, and should log a warning

The key point: RTO5a6 defines "malformed" by exactly one criterion — the absence of the : separator. There is no rule anywhere in the spec constraining the character set of the <sequence id> part. The presence/channel sync cursor at RTP18a (specifications/features.md) uses the identical <sync sequence id>:<cursor value> shape and likewise carries no charset constraint — a second, independent data point that the sequence id is meant to be an opaque server token.

The bug in ably-js and ably-java

Both SDKs parse the serial with a regex that anchors the sequence id to [\w-]+:

/^([\w-]+):(.*)$/

• ably-js: src/plugins/liveobjects/realtimeobject.ts:445
• ably-java: liveobjects/src/main/kotlin/io/ably/lib/liveobjects/ObjectsSyncTracker.kt:81 (ported from ably-js)

\w is [A-Za-z0-9_]. Any sequence id containing a character outside [\w-] (a dot, a slash, a +, etc.) fails the regex, even though the : separator is present and the value is well-formed per RTO5a1/RTO5a6.

Step-by-step failure chain for seq.1:cursor

1. The serial contains a : and splits cleanly into <sequence id> = "seq.1" and <cursor value> = "cursor". Per RTO5a6 it is not malformed; per RTO5a3 the sync should continue.
2. The regex is anchored and . is not in [\w-], so the match fails.
3. The SDK classifies the serial as malformed and, per its RTO5a6 handling, treats it as if channelSerial were absent (RTO5a5).
4. "Absent channelSerial" means "the sync data is entirely contained in this message" — i.e. the SDK behaves as though this were a complete, self-contained sync and ends the sync.
5. The sync therefore terminates before the real <sequence id>: end-marker (RTO5a4) ever arrives.

For a genuine multi-message sync the damage compounds: each partial OBJECT_SYNC with an exotic sequence id is treated as a fresh absent-serial sync, so the accumulated SyncObjectsPool is cleared and the RTO5c2 end-of-sync pruning removes objects that were legitimately delivered in earlier partials. The result is silent data loss / state corruption, not merely an early exit — accompanied by a wrong "missing : separator" warning.

Why ably-cocoa is unaffected (reference behavior)

ably-cocoa splits on the first colon with no charset constraint (LiveObjects/Sources/AblyLiveObjects/Protocol/SyncCursor.swift): it scans up to the first :, treats "no colon present" as the only malformed case, and takes everything after the first : as the cursor (empty → sync end, RTO5a4). For seq.1:cursor it yields sequenceID = "seq.1", cursorValue = "cursor" and continues the sync correctly. This matches the literal spec and is the reference implementation for the fix below.

Cross-SDK comparison

Input	Spec-correct behavior	ably-java	ably-js	ably-cocoa
Parse strategy	split on first :	regex ^([\w-]+):(.*)$	regex ^([\w-]+):(.*)$	scan up to first :
"Malformed" definition	no : present	non-[\w-] seq id or no :	non-[\w-] seq id or no :	no : present
seq.1:cursor	continue sync	ends early (BUG)	ends early (BUG)	continues (correct)
abc (no colon)	malformed → absent (RTO5a5)	correct	correct	correct
seq: (empty cursor)	sync ends (RTO5a4)	correct	correct	correct
:cursor (empty seq id)	contains : → not malformed (literal RTO5a6; spec silent on empty id)	malformed → ends (divergent)	malformed → ends (divergent)	id "", continues

Severity

Latent, not active. The regex has shipped in ably-js for over a year without incident because the realtime service currently only emits sequence ids within [\w-]. That is a fact about the server's present output, not about client correctness. The concerns are:

• The spec deliberately gives no charset guarantee, so the backend can change the serial format at any time without notifying clients.
• If it did, the failure would be silent (a misleading warning plus premature sync completion), fleet-wide across two SDKs, and hard to attribute.
• The bug invisibly constrains the backend's freedom to evolve the serial format — a client-side accident becoming a de-facto server contract.

Low urgency, but worth fixing wherever the code is being touched: the fix is two lines per SDK and it removes a hidden coupling.

Required changes

ably-js

src/plugins/liveobjects/realtimeobject.ts:445 — replace /^([\w-]+):(.*)$/ with split-on-first-colon:

• Find the index of the first :.
• No : present → malformed, handle as absent per RTO5a6.
• Otherwise: sequence id = substring before the first :; cursor value = substring after it.
• Empty cursor value → sync end per RTO5a4.

ably-java

liveobjects/src/main/kotlin/io/ably/lib/liveobjects/ObjectsSyncTracker.kt:81 — same fix, same semantics (this regex was ported from ably-js):

val idx = syncChannelSerial.indexOf(':')
if (idx < 0) {
    isMalformed = true            // RTO5a6: no separator
} else {
    syncId = syncChannelSerial.substring(0, idx)
    syncCursor = syncChannelSerial.substring(idx + 1)  // empty → RTO5a4 sync end
}

ably-cocoa

No change. SyncCursor.swift already splits on the first colon, charset-agnostically, and is the reference implementation for the above.

Spec clarification asks

These are the actionable parts for ably/specification.

1. Make the sequence-id opacity explicit (RTO5a1)

Add to RTO5a1 (or a sub-point) an explicit statement that the sequence id is an opaque server token: clients MUST NOT constrain its character set, and the only structural requirement is the : separator, which is located by its first occurrence (so a cursor value may itself contain :). RTP18a is the precedent shape that carries the same two-part identifier with no charset constraint and can be cited as such.

2. Resolve the empty-sequence-id ambiguity (:cursor)

By the literal text of RTO5a6, a serial like :cursor contains a : and can be split, so it is well-formed (empty sequence id, cursor "cursor") — and ably-cocoa continues the sync accordingly. But ably-js and ably-java currently treat it as malformed. The spec should state the intended behavior explicitly: either confirm that an empty sequence id is well-formed and the sync continues, or add an explicit rule making an empty sequence id malformed. Right now the spec is silent, so implementations diverge.

3. Close the UTS coverage gap

The only RTO5a6-derived test is objects/unit/RTO5a6/malformed-channel-serial-treated-as-absent-0 (uts/objects/unit/objects_pool.md), and its input is a no-colon serial ("malformedserialnocolon"). Every other channelSerial in the corpus uses a [\w-]-only sequence id (sync1:cursor, seq1:cursor, sync1:, …). No test uses a non-[\w-] sequence id or an empty sequence id.

Consequence: both buggy SDKs pass the entire UTS suite. The corpus cannot distinguish "colon present, exotic sequence id → must continue" from "no colon → treat as absent".

Proposed additions:

• A derived test (suggested ID objects/unit/RTO5a3/non-word-sequence-id-continues-sync-0) that feeds a serial like seq.1:cursor and asserts the sync continues (sequence id parsed as seq.1, cursor cursor, sync not ended). This test fails on the current ably-js/ably-java regex and passes on ably-cocoa and on both SDKs after the fix.
• Optionally, once ask Move the Realtime protocol documentation to this repository #2 is decided, an empty-sequence-id case (:cursor) asserting whichever behavior the spec settles on.

Verification notes

• This was found while reviewing ably-java PR #1228 (LiveObjects OBJECT_SYNC handling).
• The analysis was validated against all three SDK sources: ably-js (realtimeobject.ts), ably-java (ObjectsSyncTracker.kt), and ably-cocoa (SyncCursor.swift).
• The spec was searched for any sequence-id charset/format constraint (terms: sequence id, charset, alphanumeric, opaque, url-safe, base64) — none exists; the only base64/URL-safe rules are unrelated (objectId digest and binary value encoding).
• No existing UTS test pins the buggy behavior, so the proposed SDK fixes break no current assertion.