devshard improvement Height-sync protocol
#1206
alexanderkuprin
started this conversation in
Proposals
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Height-sync protocol
User ↔ host envelopes carry an optional
HeightSyncSectionthat attests to a mainnet(height, block_hash)pair. This section is the sole input to cross-host time alignment, timeout decisions, and theIsStrictlyConfirmed(h)predicate that downstream protocols (cPoC, finalization) gate verdicts on.block_hashis a source of determenistic randomness that is unknown in advance, used byVALIDATION_PROTOCOL_PROPOSAL.mdThis document is the canonical, single-version spec. The in-tree implementation matches this document; the test catalog (
height-sync-tests.md) lists what is already proven and what is planned.Related docs:
height-sync-tests.md(test catalog — implemented and planned),CPOC_PROTOCOL.md,FINALIZATION_COLLECTOR_PROTOCOL_PROPOSAL.md,VALIDATION_PROTOCOL_PROPOSAL.md.Table of contents
K,slots_num, forced turns)IsStrictlyConfirmed)1. Summary
User–host inference traffic carries a two-section HTTP body:
HeightSyncSection— optional mainnet attestation: a signed(mainnet_height, mainnet_block_hash)pair, plus framing andprovenance metadata.
message_body— the application payload (opaque to heightsync).
Section 1 is emitted only when needed:
Knonces, a window ofslots_numconsecutive nonces carries Anchor; in between, Omit.MsgForceHeightSyncTurnopens aslots_num-wide Anchor span at any nonce (cPoC dispute open,operator force).
|Δ| > D— when the sender's claimed height differs from thereceiver's aligned height by more than
D, the sender MUST useStrong (
LightBlock+VerifyCommit); otherwise the receiverrejects.
Hosts sign response-leg Anchors with their secp256k1 signer key; courier users carry these signed blobs forward, verifying on ingest and using them as on-demand exculpation proof. Request-leg Anchors are trusted by hosts (no inline signature) — the user proves provenance later if disputed.
A single
IsStrictlyConfirmed(h)predicate exposes a discrete{confirmed, pending, stale}answer to downstream consumers.2. Problem
At devshard we need source of mainnet height and randomness, that is unknowm in advance but is determenistic to make all hosts and user to aggree.
Each host has it's own latest
(height, block_hash)oracle (inference chain grpc), and the prove by inference chain validators signatures. But hosts should aggree that anyheightprovided to protocol is really latest. So there should be height sync protocol provided in this doc.3. High-level overview of protocol
As devshard is designed for high throughput we aim to minimize extra data transferred in messages and minimize checks of minnet signatures. Also we are minimizing gossipping that should happen only on disputes and settlement to minimize traffic (as one host could be in a lot of devshard's)
So main design decitions are made:
(height, block_hash)and originator's (host that is responding to request) signature, to proove origin of this data for possible disputes. User is carrying forward(height, block_hash)at next requests to propogate this data to other hosts.height_in_the_future - known_height = |Δ| > D) we use the full data from mainnet (block hash and validators signatures) to validate the heightheightdoesn't match the oracleblock_hashwe start the disputeAs the result we provide API at devshardd and devshardctl that gives latest height, block hash and the knowledge if majority of devshard network participants agree on this.
4. Goals
LightBlock) on async-turn schedule keeps every host's view of mainnet time within
a bounded window without per-message proof overhead.
|Δ| > Dorfinalization requires it, validator-quorum-bound proof
(
LightBlock) is mandatory.(H, hash)istraceable to the originating host signature; carriers cannot be
blamed for forwarding a malicious host's signed claim, and
carriers that strip provenance become the cryptographic source.
Fon originatortimestamp + per-recipient last-propagated bookkeeping prevents a
carrier from re-using stale or already-delivered tips.
IsStrictlyConfirmed(h) ∈ {confirmed, pending, stale}predicateso cPoC / finalization do not invent their own quorum logic.
their own can still carry signed host tips between hosts in the
round-robin and reach
(C-quorum)confirmation.5. Glossary
HeightSyncSectionproof_type = "height-anchor-v1"; carries(H, hash)without aLightBlock. Light path.proof_type = "cometbft-light-block-v1"; carriesLightBlockbytes; verified against pinned validator set with> 2/3voting power.HeightSyncSection.HhashBlockID.Hashfor blockH.KK ≥ slots_num.slots_numD|H_peer − H_local_aligned| > D⇒ Strong required. Default2.F60 s.W_confmax(256, ⌈F / block_time⌉).Q(C-quorum)threshold. Defaultceil(2/3 × N_hosts).(H, hash). Identified byOriginatorSenderIDon the wire.local_alignedIsStrictlyConfirmed(h){confirmed, pending, stale}predicate consumed by cPoC.6. Architecture overview
flowchart LR subgraph mainnet [Gonka mainnet] BC[CometBFT consensus] end BC -- "block headers + commits" --> HSD[heightsyncd / blockoracle] subgraph host [Host runtime] HSD --> SCH_H[AnchorScheduler<br/>local-oracle source] SCH_H --> TX_H[transport.Server<br/>signs response leg] TX_H -- "HeightSyncSection<br/>request inbound" --> RX_H[Receiver pipeline<br/>D-band, freshness, classify] RX_H --> AUD_H[AuditRing + ConfirmationIndex] AUD_H -- "IsStrictlyConfirmed" --> CPOC_H[cPoC consumer] end subgraph user [Courier user / devshardctl] TIPS[HeightSyncPeerTips<br/>verbatim signed blobs] --> SCH_U[AnchorScheduler<br/>peer-tip source] SCH_U --> TX_U[transport.HTTPClient<br/>Carry, strip sig on request] TX_U -- "response inbound<br/>verify + cache" --> RX_U[Verify response Anchor<br/>RecordOriginWithBlob] RX_U --> TIPS TIPS -- "IsStrictlyConfirmed" --> CPOC_U[cPoC consumer] end TX_U -- "request leg<br/>HeightSyncSection" --> RX_H TX_H -- "response leg<br/>HeightSyncSection (signed)" --> RX_UKey invariants:
heightsyncd/blockoracle);this is the canonical source of
local_aligned.local_alignedfrom the verified peer-tip cache populated bysigned host responses.
HeightSyncSectionis the only mainnet-related wire surface oninference envelopes; the receiver pipeline is single-entry.
7. Wire format
HeightSyncSectionis carried as protobuf field on the inferenceenvelope and JSON-mirrored for tooling. Field numbers are stable.
proof_typestring"height-anchor-v1"(Anchor) or"cometbft-light-block-v1"(Strong).mainnet_heightint64H. MUST NOT be set in Omit.mainnet_block_hash_hexstring(hex)BlockID.HashforH.timestamp_unix_msint64directionstring"request"or"response".originator_sender_idstring(H, hash)from its own oracle. Carrier MUST preserve.originator_timestamp_unix_msint64F.sender_signaturebytes"heightsync.origin.v1". Empty on request leg.light_blockbytesLightBlock-equivalent (blockoracle.HeaderwithCommit.Signatures).tip_stale_after_msint64StaleAfterexceeded) while a cached(H, hash)is still available. Absent when the tip is fresh or on courier carry-forward re-emits. Receivers MUST NOT treat this field as part of the cryptographic attestation; use freshness gateFon originator timestamps and(C-quorum)/IsStrictlyConfirmedfor liveness.Notes:
received a new block within
StaleAfterbutLatest()stillreturns a cached header, hosts emit a normal Anchor (fields 1–8) plus
field 10. This avoids sync-turn response Omit during long
inter-block gaps; consensus across hosts still corrects a minority
with an outdated tip. Omit remains mandatory when there is no
cached tip (feed never started),
Latest()fails (feed unavailable),or the courier peer-tip cache is empty.
only.
Carry()clears field 8 before sending on the request leg;inbound request validation does not require an inline signature.
CanonicalOriginBytes(sec)="heightsync.origin.v1" || proto.Marshal(fields 1..7). Field 8is not part of the signing input.
origin_attestation(embeddedoriginator blob) is reserved for future inline-embed deployments;
current protocol uses the asymmetric model (response signed,
request trusted, on-demand exculpation).
JSON mirror:
{ "height_sync": { "proof_type": "height-anchor-v1", "mainnet_height": 42, "mainnet_block_hash_hex": "abc...", "timestamp_unix_ms": 1700000000000, "direction": "response", "originator_sender_id": "gonka1host...", "originator_timestamp_unix_ms": 1700000000000, "sender_signature": "base64...", "light_block": "base64...", "tip_stale_after_ms": 12000 } }(
tip_stale_after_msomitted when the cached tip is fresh.)8. Sync modes (Omit / Anchor / Strong)
stateDiagram-v2 direction LR [*] --> Omit Omit --> Anchor: nonce in sync turn / forced turn / lazy carry Anchor --> Omit: next nonce outside window Anchor --> Strong: \|H − local_aligned\| > D OR forced (StrongRequired) Strong --> Anchor: peer realigned, within D again Anchor --> Anchor: cadence next turn Strong --> Strong: still > Dlight_block)StaleAftersince last block) and a cached tip exists — see field 10.|Δ| > D, finalization-grade, or forced turn withStrongRequired = true.Periodic alignment uses Anchor only. Strong is not a default
cadence step — it is the disagreement / dispute path.
Quiet feed vs dead feed (hosts):
StaleAfter)StaleAfter)tip_stale_after_ms> 0)oracle_misswhen cadence required)Latest()error, e.g. height-sync stopped)9. Cadence
Sync-turn windows
For a session direction, on outgoing nonce
n:1 ≤ n ≤ slots_num→ Anchor (or Strong, see §10).i ≥ 1,i·K ≤ n ≤ i·K + slots_num − 1→ Anchor.Constraint:
K ≥ slots_numso windows never overlap.gantt title Sync-turn cadence (K=8, slots_num=4) dateFormat X axisFormat %s section Cadence Initial sync turn (Anchor) :a1, 1, 4 Omit :a2, 5, 7 Periodic sync turn 1 :a3, 8, 11 Omit :a4, 12, 15 Periodic sync turn 2 :a5, 16, 19Forced sync turn
MsgForceHeightSyncTurn(trigger_nonce, slots_num, reason, strong_required?)opens anActiveForcedTurn{start, end}span:[start, end]. Omit inside a forced turn is INVALID.strong_required = trueupgrades the window to Strong.it (no double-Anchor on boundary).
n > end, cadence resumes from the standard rule.Lazy carry-forward (courier deployments)
Outside any sync-turn window, the courier user MAY emit Anchor on a
request leg iff:
(
MaxFresh(now, F)returns non-nil).cached_max_height > last_propagated[recipient].The receiver classifies this as
VALID_LAZY_ANCHOR(audit taglazy); it does not open a sync-turn obligation.10. Producer rules
Hosts (have own oracle)
turn or forced turn applies, emit Anchor with
OriginatorSenderID = host_address,OriginatorTimestampMs = now, and sign the section (field 8).If the oracle is quiet (no new block within
StaleAfter) but acached tip exists, still emit that Anchor and set
tip_stale_after_msto the age of the last ingested block (field 10is set after signing input fields 1–7). Omit only when there is
no usable cached tip or
Latest()fails.forced.StrongRequiredis set OR receiver'speer_aligned_heightdiffers from local tip by> D: produceStrong by attaching the cached
LightBlockforH(field 9).receiver pipeline (§11).
Courier user (no own oracle)
HeightSyncPeerTipskeyed byOriginatorSenderID.VerifyOrigin); on failure, dropthe tip and increment
origin_sig_invalid_total.when the cache has a tip not yet propagated to the recipient. Clear
field 8 (
sender_signature) before sending.OriginatorSenderID = user_address; that fieldreflects the host that signed the cached blob.
11. Receiver pipeline
flowchart TD A[envelope arrives] --> B{HeightSyncSection<br/>present?} B -- no --> O{nonce in sync turn /<br/>active forced turn?} O -- yes --> O1[INVALID<br/>sync_turn_anchor_missing] O -- no --> O2[VALID_OMIT] B -- yes --> C{Anchor or Strong?} C -- Anchor --> D{\|H − local_aligned\| > D?} D -- yes --> D1[INVALID<br/>strong_required] D -- no --> E{carry-forward<br/>originator set?} E -- yes --> F{originator within F?} F -- no --> F1[INVALID<br/>stale_origin] F -- yes --> G[classify cadence / lazy<br/>by nonce vs sync turn] E -- no --> G C -- Strong --> H[StrongVerifier.VerifyLightBlock] H -- ok --> I[VALID_STRONG] H -- fail --> H1[INVALID<br/>strong_proof_invalid] G --> J{block H local AND<br/>hash matches?} J -- yes/match --> K[VALID_ANCHOR or<br/>VALID_LAZY_ANCHOR] J -- no/local-missing --> L[enqueue deferred check] J -- local AND mismatch --> M{originator present?} M -- yes --> M1[DISPUTE_ORIGINATOR] M -- no --> M2[DISPUTE_CARRIER]Normative steps for a non-Omit envelope:
ActiveForcedTurn[start..end]isset and
start ≤ nonce ≤ end, the envelope MUST be Anchor (orStrong when
StrongRequired). Omit ⇒ INVALID.Dband. Ifproof_type == "height-anchor-v1"and|H − local_aligned| > D: INVALID (strong_required).proof_type == "cometbft-light-block-v1": runStrongVerifier.VerifyLightBlock(chain id, header vs claims,validators_hash, optional epoch-bound Step 3b,BlockID,commit
> 2/3); failure ⇒ INVALID (strong_proof_invalid).OriginatorSenderID != "":now_ms − OriginatorTimestampMs > F⇒ INVALID(
stale_origin); audit trust =TrustDisputeCarrier.VALID_ANCHOR(tag
cadence).VALID_LAZY_ANCHOR(tag
lazy).self-attestation;
VALID_ANCHOR.His local andhashmatches → confirmedimmediately; feed
ConfirmationIndex.His not yet local → enqueue deferred check by(originator, H, hash); do not advanceheight_seen_max.His local andhashdiffers →DISPUTE_ORIGINATOR(originator metadata present) or
DISPUTE_CARRIER(originator absent or signature failed); persist the offending
signed blob.
AnchorAttestation(withTag,Trust,OriginatorSenderID,OriginSignedBlobAvailable) to theper-peer ring; emit counters.
message_bodyif not INVALID.Result classes
VALID_OMITVALID_ANCHORVALID_LAZY_ANCHORlazy.VALID_STRONGLightBlockverified against pinned set; may advance strong anchor state.VALID_STALEDEFERRED_FAILDISPUTE_ORIGINATORDISPUTE_CARRIERINVALID(reason)sync_turn_anchor_missing,stale_origin,strong_required,strong_proof_invalid,bad_framing.12. Trust model and signatures
The protocol is asymmetric: responses are signed, requests are
trusted, exculpation is on-demand.
sequenceDiagram participant U as User (courier) participant H as Host A participant H2 as Host B U->>H: request (request leg, no sig) H->>U: response Anchor [signed by Host A] Note over U: VerifyOrigin OK<br/>RecordOriginWithBlob(host_A, H, blob, sig) U->>H2: request Anchor (carry-forward, no inline sig) Note over H2: trusts carrier;<br/>freshness gate F applies Note over H2: later, follower advances<br/>compares hash to canonical H2-->>U: DEFERRED_FAIL? open dispute vs user U->>U: HeightSyncEvidenceFor(host_A, H) → blob + sig U-->>H2: signed_blob proves originator = Host A<br/>⇒ DISPUTE_ORIGINATOR vs Host AResponse leg (host → user)
OriginatorSenderID,OriginatorTimestampMs, buildsCanonicalOriginBytes(fields 1–7 + domainheightsync.origin.v1).propagation;
origin_sig_invalid_totalincrements.RecordOriginWithBlob(originator, sec, blob, sig).Request leg (user → host)
Carry()strips field 8 before sending.inline signature is required or verified.
Exculpation
If a host later opens a dispute against the user-carrier, the user calls
HTTPClient.HeightSyncEvidenceFor(originator, H)to produce the cached(blob, sig). A dispute verifier re-runsVerifyOrigin; success ⇒ blame shifts to the originating host (DISPUTE_ORIGINATOR); failure ⇒ blame stays on the carrier (DISPUTE_CARRIER).Strong proof
When Strong is on the wire (
light_blocknon-empty), validation is cryptographic against the pinned validator set:LightBlock-equivalent (blockoracle.Header).chain_id,height,block_hashagainst claims.validators_hashmatches Merkle root of the pinned set.BlockID == hdr.BlockID.VerifyCommit: every commit signature ecrecovers to a pinnedvalidator, no duplicates, accumulated power strictly
> 2/3oftotal.
h ≥ local_tip − max_lag_blockselseVALID_STALE.13. Carry-forward, provenance, attribution
Rules
OriginatorSenderIDorOriginatorTimestampMs.Dbound on carry-forward. A carry-forward Anchor with|H − local_aligned| > Dis INVALID; carrier MUST escalate to Strong instead. (This is a stricter form ofstrong_required.)DISPUTE_CARRIER).last_propagateddisciplineHeightSyncPeerTips.ShouldPropagateTo(recipient, h)returns true iffh > last_propagated[recipient]. On a successful send,MarkPropagated(recipient, h)advances the high-water mark.Reaching a quorum at a late host requires a strictly increasing
height ladder (production-faithful) — not three lazy carries at
the same
H.14. Confirmation API
Contract
Semantics:
confirmed—hhas cleared the configured confirmation rule.Downstream protocols MAY treat
(h, hash)as authoritative.pending— height-sync has data forhbut has not yet clearedthe rule. Downstream MUST NOT commit adversarial verdicts; cPoC
returns
Inconclusive(C6).stale—hcannot be evaluated because the oracle is stale /disconnected. Downstream treats verdicts as
Inconclusiveuntilrecovery.
Monotonicity: once
confirmed, a height staysconfirmed.pending → confirmedis the only forward transition.Confirmation rules
Configured at deployment time:
(C-quorum)≥ Qdistinct originators from the roster have attested heights≥ h, all withinF, all in[tip − W_conf, tip].(C-strong)LightBlockfor height≥ h.(C-hybrid)PoC deployments without Strong run
(C-quorum). Production-classdeployments SHOULD select
(C-hybrid)once Strong is enabled.Confirmation memory and pruning
Fnow − observed_at > F60 sW_conf[tip − W_conf, tip]countmax(256, ⌈F / block_time⌉)Qceil(2/3 × N_hosts)max_height < tip − W_conforobserved_atpastF).confirmed_heightsset sopruning never demotes a confirmed height.
Per-
Vview, not globalIsStrictlyConfirmedis computed against the caller's own auditring and clock. Two verifiers may transiently disagree (
pendingvsconfirmed); cPoC's quorum-based slashing tolerates this.15. cPoC integration — full API
The following Go APIs are the stable surface that cPoC and
finalization consume. Implementation paths in parentheses.
15.1 Discrete confirmation predicate
Both expose
ConfirmationView.IsStrictlyConfirmed(h uint64) ConfirmState.cPoC §C6 / §C14 / §Verdict step 5 call this directly.
Usage example (cPoC verdict):
15.2 Observed height (courier heartbeat)
Returns
(h, true)wherehis the highest fresh tip in thecourier peer-tip cache;
(0, false)when no fresh tip exists orheight sync is not configured. Used by cPoC C14 heartbeats — a
falsereturn means "Inconclusive — no fresh height".15.3 Exculpation evidence (dispute layer)
Returned by the courier cache (
HeightSyncPeerTips.OriginSignedBlobFor).Verifiable with
heightsync.VerifyOriginDetached(verifier, sec, blob, sig)without reaching the user.
15.4 Strong-grade evidence (Strong mode)
When the receiver's follower has advanced past a disputed
H, thedispute packet may carry both halves:
HeightSyncEvidenceFor(blame).LightBlockfromLightBlockFor(canonical pair).A mock dispute verifier returns
DISPUTE_ORIGINATORwhen both pass.15.5 Cold-start seed (optional)
Opt-in
POST /sessions/:id/height-sync: the host returns a forcedAnchor (originator-signed). The courier verifies + caches it before
issuing the first inference — useful for short-lived sessions where
the first inference is not in a sync turn.
15.6 Force a sync turn
Opens an
ActiveForcedTurnin the next diff; every envelope in[trigger, trigger + slots_num − 1]MUST be Anchor (or Strong whenstrongRequired = true).15.7 Audit and dispute consumers
Used by dispute / finalization consumers that need verbatim
attestations and per-peer history.
16. Attack model
Each row maps an adversary action to the protocol's defence and to
the test scenario that proves it (full catalog in
height-sync-tests.md).(H, hash)and signs it(C-quorum)rejects single bad vote; deferred check eventually triggers DEFERRED_FAIL;DISPUTE_ORIGINATORwith stored signed blobTestHeightSyncAnchor_E2E_MixedHeights_Confirmed,TestHeightSyncAnchor_E2E_CheatingTrailStoresBogusUserHash,TestHeightSyncAnchor_E2E_CarrierExculpation; planned:TestHeightSyncStrong_E2E_DeferredFail_StrongEvidenceFrejects withstale_origin; carrier flaggedTrustDisputeCarrierTestHeightSyncAnchor_E2E_StaleOriginRejected,TestHeightSyncAnchor_E2E_HeldOriginatorReplayRejectedDISPUTE_CARRIERon mismatch; sync-turn empty-originator audit dispute sentinelTestHeightSyncAnchor_E2E_ForcedSyncTurn_HostResponsesAnchorEvenIfUserOmitsconfirmed; eventually DEFERRED_FAILTestHeightSyncAnchor_E2E_CheatingTrailStoresBogusUserHashsender_signatureorigin_sig_invalid_totalincrements; no cache; reputation/liveness handles persistent offendersTestHeightSyncAnchor_E2E_ResponseOriginSignatureInvalidDropped,TestClient_ResponseAnchor_DropsOnInvalidSig(H, hash)|Δ| > Dahead with Anchor (no Strong)INVALID(strong_required); carrier cannot escape via originator metadataTestClassify_StrongRequiredOutsideD, S1, S8LightBlock(signatures, validators_hash, BlockID)TestVerifyLightBlock_*, S4Latest()fails)IsStrictlyConfirmed → stale; no crashes; cPoC returnsInconclusiveTestHeightSyncAnchor_E2E_HeightSyncFeedStopped_*,TestHeightSyncAnchor_E2E_StaleOracle_Inconclusivetip_stale_after_ms; courier may still ingest verified response tips;(C-quorum)reconciles height across hostsTestAnchorScheduler_StaleFeedEmitsDegradedAnchorInSyncTurn,TestDecide_LogStaleSyncTurn, containerTestContainerE2E_HeightSync_Cadence(testenvMOCKDAPI_STALE_AFTER≥ block cadence)(H, hash_A)then(H, hash_B))Vaudit ring + dispute layer cross-session checkforce_request_anchor_missingsentinel for disputeTestHeightSyncAnchor_E2E_ForcedSyncTurn_HostResponsesAnchorEvenIfUserOmitsLightBlocklocal_tip − max_lag_blocks→VALID_STALE; never advances aligned heightOut-of-scope adversaries:
> 2/3of mainnet validators —outside this protocol's defence; same as any L1 consensus
assumption.
oracle has its own pinned validator-set verifier
(
blockoracle/verifier); height sync does not re-validate.17. Defaults and configuration
K8AnchorSchedulerconstructor (NewAnchorSchedulerFromOracle(K, slots, src))slots_num4(testenv) /slots_num = escrow.HostSlots(production)D2StrongPolicy.D(planned)F60 sHeightSyncPeerTips.Freshness,ConfirmationConfig.FreshnessW_conf256heightsConfirmationConfig.WindowHeightsQceil(2/3 × N_hosts)ConfirmationConfig.Quorum(override; defaults from roster size)1024per peerNewAuditRing(capacity)64heightsBlockOracleStrongSource.K(planned)max_lag_blocks = 0)(C-quorum)ConfirmationConfig.Rule(Quorum/Strong/Hybrid)StaleAfter(block oracle client)10 sclient default; testenv:block_time + block_interval_delta + 1s(floor10s)MOCKDAPI_STALE_AFTERenv (compose / devshardd-testenv)18. Status and milestones
(C-quorum)+ lazy carry + freshness gateLightBlock+VerifyCommit+Dband +(C-strong)/(C-hybrid))height-sync-tests.md§6 (S1–S12).MsgHeightSyncEvidence+ slashing txReading order for contributors
height-sync-tests.md— every behaviour above is bound to at least one named test.Beta Was this translation helpful? Give feedback.
All reactions