Repository navigation
Direction review: reconciliation for settlement-based venues (Polymarket) #4722
Replies: 9 comments 2 replies
|
Hey @seungpyoson, Thanks for moving this into a discussion before opening another PR. The overall direction fits. I would take tiers 1 and 2 in‑tree, but I don't think each tier maps to a single PR. I would defer tier 3 until the earlier contracts are stable.
I encountered the same evidence-to-state boundary while reviewing #4728. That PR concerns market liveness rather than execution reconciliation, so I wouldn't combine the implementations. It does show why this contract matters: its market evidence is If this split matches the work on your fork, could you post the proposed tier 1 type and transition table here before opening the next PR? Once that contract is agreed, I think the first implementation can stay narrow and give tier 2 a stable evidence boundary. |
|
Thanks — that split matches how the work has actually decomposed on the fork, and deferring tier 3 makes sense to me. Here is the proposed tier 1 contract. Venue statuses ( Worth stating explicitly: this is not monotonic. The adapter-owned typestruct TradeEvidence {
// Identity - a venue trade filling N of our maker orders yields N evidence records
trade_id: TradeId, // composite: venue trade id + maker order id for maker legs
venue_order_id: VenueOrderId,
instrument_id: InstrumentId,
order_side: OrderSide,
liquidity_side: LiquiditySide,
// Economics - all required, no defaults, no fallbacks
last_qty: Quantity, // > 0 after precision conversion
last_px: Price, // strictly inside (0, 1)
commission: Money,
ts_event: UnixNanos, // parsed from the venue, never substituted
state: EvidenceState,
}
enum EvidenceState { Applied, Confirmed, Voided }Evidence that fails any identity or economics check is never constructed, so it cannot reach state. There is no Transition tableRows are current state, columns are the incoming observation.
Invariants
Shared boundaryOne constructor validates for both transports, so REST and WebSocket cannot derive different economics or different failure behaviour for the same execution. Today they diverge on timestamps — REST rejects an unparsable Settlement status stays inside the adapter as you suggested; Two questions
|
|
@seungpyoson, thanks for the type and transition table. The shape is close, and I agree that stored evidence should begin at Direct answers
The tombstone must absorb later provisional observations, otherwise a stale State application
Source correctionsCurrent With those changes, tier 1 can remain adapter‑scoped. If you’re able, could you revise the type and table around the terminal tombstone and shared registry before opening the PR? |
|
Thanks — before revising the table, I need to confirm what authority “trigger a refresh” should have. Core rejects any later To keep tier 1 adapter-scoped, I propose delaying the void:
Does “refresh” mean that this targeted REST result is authoritative and final for tier 1, so later contradictory stream observations are treated as stale? If not, the current core boundary needs a way to restore a voided fill. Once that authority rule is agreed, I can provide a complete type and transition table without leaving quarantine resolution ambiguous. |
|
@seungpyoson, thanks for catching the core constraint. Yes: for tier 1, treat a fresh authenticated targeted REST trade read as the resolution authority. A conflicting stream status triggers quarantine and refresh; it does not directly void the fill. Accept the REST result only when it matches the full account‑scoped venue trade identity, contains every owned maker leg, passes the shared validation, and has a terminal status.
With that authority rule, tier 1 does not need a core fill‑restoration path. If you are able, could you revise the type and transition table around this boundary? |
|
Thanks, that resolves the authority question. I’ll treat the authenticated targeted REST terminal result as the quarantine-resolution authority, gated by full account-scoped identity, complete owned-leg validation, and shared validation. I’ll revise the complete type and transition table around that rule and post it here before opening the tier 1 PR. |
|
Thanks — here is the revised tier 1 contract. This supersedes my earlier table. Registry identity and payloadThe adapter owns one synchronized registry keyed by the full account-scoped venue identity: struct AccountTradeKey {
account_id: AccountId,
venue_trade_id: String,
}
struct ValidatedTradeEvidence {
key: AccountTradeKey,
owned_legs: Vec<ValidatedTradeLeg>, // private constructor: complete and non-empty
}
struct ValidatedTradeLeg {
trade_id: TradeId, // core-facing; composite for maker legs
venue_order_id: VenueOrderId,
instrument_id: InstrumentId,
order_side: OrderSide,
liquidity_side: LiquiditySide,
last_qty: Quantity,
last_px: Price,
commission: Money,
ts_event: UnixNanos,
}
enum TradeState {
Applied,
StreamConfirmed,
Quarantined,
RestConfirmed,
RestFailed, // zero voided legs means a terminal no-fill tombstone
HardFault,
}One venue trade is one registry record. Core events remain per leg, but every owned maker leg is resolved and validated before any leg mutates the registry, fill tracker, or order state. An incomplete or empty owned-leg set cannot construct
Transitions
The targeted REST result is accepted only when it is authenticated for the same account and venue trade, contains every owned maker leg, passes the shared identity/economics validation, and is terminal. Missing, provisional, incomplete, or inconsistent REST results remain quarantined and retry. The first accepted terminal REST disposition is final. A later contradictory stream observation cannot reverse it; a later contradictory authenticated REST terminal result is a hard reconciliation fault. On restart, tier 1 explicitly hydrates applied/voided leg identity from retained core order events before processing live terminal observations. If that hydration is incomplete, the trade remains quarantined until the targeted REST result resolves it. This does not claim tier 3's atomic durable application contract. If this matches your expectation, I will rebuild tier 1 from current |
|
I found a gap in the restart paragraph of my revised Tier 1 contract, so I want to correct it before implementing the registry. The current proposal uses one
Those facts cannot be collapsed safely. A maker trade can have several owned legs, and core applies them individually. The adapter's current emitter is fire-and-forget, while the execution engine publishes I therefore propose this narrower Tier 1 contract. enum AdmittedTradeEvidence {
Fill(ValidatedFillEvidence), // provisional or confirmed; complete economics
Failed(ValidatedFailedEvidence), // complete participant authority; no unused economics
}
struct TradeRecord {
key: AccountTradeKey,
participants: Vec<ValidatedLegIdentity>, // private constructor: complete, non-empty, unique
fill_evidence: Option<ValidatedFillEvidence>,
settlement: SettlementDisposition,
legs: IndexMap<LegKey, LegEffect>,
}
struct LegKey {
venue_order_id: VenueOrderId,
trade_id: TradeId, // core-facing; composite for maker legs
}
enum SettlementDisposition {
Provisional,
StreamConfirmed,
Quarantined,
RestConfirmed,
RestFailed,
HardFault,
}
enum LegEffect {
Absent,
FillPending,
FillObserved,
VoidPending,
VoidObserved,
}
enum RegistryPhase {
Hydrating,
Live,
Faulted,
}Both admitted variants carry the same account-scoped trade key and the complete, unique set of owned participant identities. Fill-bearing statuses additionally require positive/exact economics, commission, and a venue timestamp.
All owned participant identities validate before any effect is attempted, and every fill-bearing leg additionally validates its economics. Repeated evidence is idempotent per leg, conflicting fill economics quarantine the complete trade, and the first accepted authenticated terminal REST disposition remains final. Cold-start boundaryI do not propose using a paginated account-wide REST read as a snapshot barrier. The public trade endpoint documents authenticated pagination and Instead:
This makes the restart guarantee explicit without inventing durable adapter state:
The resulting transition rule is:
The targeted REST result is accepted only for the exact authenticated account/trade key, with the complete owned-participant set and status-appropriate shared validation. A terminal settlement disposition may coexist with pending leg effects; it does not falsely assert that core applied every effect atomically. This is my recommended Tier 1 boundary. It remains adapter-owned and process-local, uses the existing core events as the application observation boundary, and does not claim durable exactly-once behavior. Does this session-proven provisional gate and two-axis state match the restart contract you intended? If so, I will implement this instead of the single-state table. Separately, #4831 is the narrow REST report-admission/fail-closed slice. It does not implement this settlement registry or claim to resolve Tier 1. |
|
Hey @seungpyoson, Thanks for the follow-up, and especially for identifying the gap between venue settlement, submission to core, and observed core application. I reviewed your August 24 revision before opening RFC #4876, and incorporated its two-axis settlement/application model, current-session provisional gate, core-event observation boundary, and process-local scope. The RFC adds the maintainer decisions and acceptance boundaries around that design, and is now the current proposal. I’m closing this discussion so the remaining design review continues in one authoritative thread. If you’re able, could you review the RFC and raise any differences or objections there? For future high-level design and policy proposals, we’ll use RFC issues so the proposal, decisions, and implementation boundaries remain in one place. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
@cjdsellers
Context
Polymarket fills are not final when first reported: a matched trade settles on-chain afterwards and can fail after the fact. NautilusTrader's execution reconciliation generally treats a reported fill as final, which on this venue can turn provisional or incomplete evidence into authoritative order/position state.
I worked this problem through a series of PRs (#4691, #4692, #4693, then replacements #4704/#4705/#4707) and closed them all — iterating through many open PRs was the wrong way to converge on a design this size, so following earlier feedback I moved iteration to a fork. Before filing anything again I'd like to align on direction here.
From that campaign, #4662 (maker fill ownership and surfacing dropped trades) is merged, and
OrderFillVoidedalready exists indevelopfor fill corrections.Problem
Three related gaps, all observed live on Polymarket:
Proposed direction
Three tiers, smallest first:
OrderFillVoidedcorrection model with confirmation/failure finality.Ask
Does this direction fit the project? Which tiers would you take as PRs, and in what shape and order? Happy to work through it here or on Discord — I'd rather converge on the design before opening any more PRs.
All reactions