Reducing P95 latency in a bounded multi-wallet XRPL scheduler without unsafe transaction retries

[opmonthetrack]

We run a Node.js service for autonomous XRPL order management. The architecture has three bounded wallet execution lanes, one low-frequency maintenance lane, and a separate transaction-observation worker. Per-account execution is serialized with a durable public-identity journal; an uncertain transaction blocks further signing on that account until validation or proven expiry. No live lease is evicted based on elapsed time.

A ten-minute observation window showed median wallet revisit time of about 3.5 seconds but P95 around 16.9 seconds. These include fast checks of blocked wallets; they are not equivalent to executable strategy decisions, fills, or profitable closed trading cycles. Public upstreams occasionally reported slowDown/tooBusy. The gateway was observed with 16 requests running and 13 queued. Generic READY status does not prove command-specific availability. We also found that inferring server_state support from the presence of server_info.server_state wrongly excluded two endpoints that successfully serve the command; we are correcting that capability test.

We already coalesce equal requests and cache explicit validated-ledger reads. Market quotes carry ledger identity and observation age. Wallet and quote freshness, issuer fees, funded lot depth, and outgoing commitments remain mandatory checks. Background actors use the same gateway and several classify themselves as trading consumers, so the nominal role flag does not reflect actual work priority.

Questions:

1. Which practical queue design best reserves capacity for submit/status/budget requests while providing bounded fairness to market research and accounting readers?
2. How would you maintain account/offer snapshots using validated transaction subscriptions, reconcile partial fills and reconnect gaps, and reduce polling without treating missing events as proof of no state change?
3. How should command capability and shared-provider quotas be represented when multiple endpoint URLs may share a backend?
4. What tracing/replay methodology can separate queue wait, upstream IO, strategy calculation, pending-account checks and actual ready-to-sign decisions?
5. Is a distributed limiter necessary on one host, or is a shared gateway sufficient? Which measured thresholds would justify another process?

A local RPC-admission prototype reserves background capacity and prioritizes execution/trading reads. Five offline tests pass for reservations, non-preemption of running requests, bounded concurrency, error release and bounded queue admission. It is not integrated into the production gateway and does not demonstrate that the P95 target is attained.

We seek architecture review and reproducible evidence, not trading signals or a guarantee of profit. This draft contains no wallet addresses, customer names, balance data, server address, keys or private repository source.

[IlyaKhalafi]

I would integrate admission control into the shared gateway first, while keeping your durable per-account signing barrier unchanged. A faster revisit to a blocked wallet is not a ready-to-sign decision, so those need separate latency distributions.

1. Reserve capacity by operation, not actor name. Give submit/status reconciliation, decision-critical reads, and background work separate bounded queues. Derive the class at trusted call sites so an accounting task cannot acquire execution priority by calling itself a trading consumer. Cap background in-flight requests below the global limit, leaving unborrowed capacity for critical work. Use separate market-research and accounting subqueues with positive weights and bounded admission shares, rather than unlimited strict priority. Apply both endpoint concurrency and provider-wide rate budgets before dispatch. Bound request duration and queued age; without bounded upstream duration and admitted load, fair admission does not imply bounded completion latency. Keep the reservations unborrowed when prompt critical admission matters, since already-running requests cannot be preempted. Expired queued reads should fail as stale and be reconsidered against fresh state. Bound a signed-submit queue entry by its actual LastLedgerSequence and recheck validity before dispatch. A blob known never dispatched can be dropped or re-admitted while still valid; a dispatched or ambiguous attempt requires reconciliation by hash. A scheduling timeout must not automatically clear the signing barrier or trigger a fresh signature.

2. Use subscriptions to invalidate snapshots, with explicit gap recovery. Subscribe first and buffer notifications, then take account/offer snapshots at one validated ledger L. Install the snapshot and process only buffered events with validated: true after L, deduplicating by transaction hash and preserving ledger/transaction order. An affected account or offer can be marked dirty and refreshed at a pinned validated ledger instead of implementing every balance and offer transition immediately. A vanished offer does not by itself prove a fill.

   For partial fills, track the offer by owner and sequence, and inspect its Offer entry in validated AffectedNodes. For a ModifiedNode, compare changed TakerGets and TakerPays values in PreviousFields with FinalFields to measure the reduction in remaining size, preserving currency/issuer identity and using decimal arithmetic. Omitted previous fields are unchanged, not zero. Treat a DeletedNode as a removal to classify using transaction context, rather than calling every removal a full fill. Verify settlement effects separately from the offer-size change, and persist each event by ledger number, transaction hash and offer identity. A pinned-ledger refresh confirms remaining state but cannot by itself recover execution history. Where subscription coverage is uncertain, use the dirty-state and history-reconciliation path rather than assuming every fill was observed. subscribe provides validated transaction metadata, and account_offers supports explicit ledger selection and pagination.

   On reconnect or a detected ledger gap, mark affected snapshots stale and reconcile from the last completed watermark to a new validated ledger. Use ledger notifications and a liveness timeout to detect gaps, plus periodic reconciliation; contiguous ledger notifications alone do not prove that every account notification was received. Page through account_tx with explicit ledger_index_min and ledger_index_max from the completed watermark plus one to the target validated ledger. Check the returned bounds against that requested range and require adequate history coverage. A successful response or an empty page is not proof that a missing range was searched. If a pinned snapshot read returns lgrNotFound, use a server holding that ledger or restart snapshot acquisition at an available validated ledger. Inspect the server’s advertised ledger coverage and keep unresolved history gaps explicit. Buffer new live events during recovery and advance the watermark only after reconciliation is complete. Missing events must never clear your pending-transaction barrier.

3. Track command capability separately from health and quota. Key capability by endpoint, network and command, with supported, unsupported and unknown states plus probe time and failure reason. An actual successful server_state request establishes that capability; a field in server_info does not. Treat an explicit unknown-command response as negative capability evidence with an expiry. Scope permission failures to the credentials used. Transient errors leave capability uncertain, and throttling triggers cooldown. Re-probe periodically, since load-balanced endpoints can change backends. Group URLs under an explicit provider/quota identity, including shared credentials or egress limits where applicable. Different URLs do not establish independent capacity; confirm provider limits rather than infer them from hostnames.

4. Replay the gateway trace before changing concurrency. Record enqueue, admission, dispatch and completion times with monotonic clocks, plus operation class, command, provider, wallet state and input ledger/age. Separately record strategy calculation and the outcome of each readiness gate. Report queue wait and upstream duration by class, blocked-wallet checks, stale-input rejections, and ready-to-sign decisions. Replay the same arrivals and observed service durations through FIFO and the proposed scheduler, including throttling, stalled requests and reconnect recovery. That can test queue behavior, but it cannot establish a production P95 improvement because upstream behavior may change with the schedule.

5. One host does not inherently need a distributed limiter. One gateway is sufficient only if every process, background actor and retry actually uses it and it owns the shared quota state. Separate processes need shared admission state or that same gateway. Measure event-loop delay percentiles, for example with Node’s monitorEventLoopDelay, and CPU saturation. Move CPU work to a worker when those measurements exceed its allocated share of the decision-latency budget while upstream I/O is healthy. There is no universal millisecond threshold from the information given. Another process or URL does not add provider quota.

For uncertain submissions, retain the same signed transaction identity and reconcile the authoritative outcome. Do not turn a scheduling timeout into a freshly signed replacement. The reliable submission guide requires validated results and continuous history when proving non-inclusion through LastLedgerSequence.

This is an architecture proposal against the documented protocol, not a benchmark of your gateway. Your offline admission tests are a useful start; the missing evidence is the integrated trace showing which stage accounts for the 16.9 second tail.