bridge: Significant Delays in BurnTransactionReady/RefundTransactionReady Processing Leading to Accumulating Unhandled Transactions

[sameh-farouk]

Describe the bug

The TFChain bridge is experiencing a critical operational issue where BurnTransactionReady events (internally handled as WithdrawReadyEvents by the bridge daemon) are processed with significant delays, often exceeding the 2-minute window during which their associated Stellar signatures are available on TFChain. This leads to a growing backlog of unhandled transactions, impacting the bridge's efficiency and reliability.

Additional context

Stellar Multi-signature and TFChain's Role

1. Stellar Multi-signature for TFT Burns
   When TFT is to be unlock on the Stellar network (e.g., as part of a withdrawal from TFChain), a multi-signature Stellar transaction is required.
   This transaction needs to be signed by multiple designated validators (bridge daemons) to ensure security and decentralization.

2. Signatures on TFChain
   To facilitate this, individual validators submit their signatures for a given burn transaction to TFChain. These signatures are stored on TFChain's state, making them accessible to the bridge daemon.

3. Stellar Sequence Numbers
   Every Stellar account has a sequence number, which is a monotonically increasing integer. Each new transaction originating from that account must have a sequence number exactly one greater than the account's current sequence number.
   This mechanism prevents transaction replay attacks and ensures transaction ordering. If the Stellar bridge account's sequence number advances before a transaction is submitted, any signatures based on the old sequence number become invalid.

---

The 2-Minute Signature Expiry Mechanism

TFChain incorporates a defensive mechanism within its pallet-tft-bridge to manage these Stellar signatures. Signatures associated with BurnTransactionReady events are periodically cleared from TFChain state after approximately 2 minutes.

• Why the Expiry?
  This timeout is a safety measure. Stellar's sequence numbers are critical for transaction validity. If the Stellar bridge account's sequence number advances (e.g., due to another transaction being submitted), any pending signatures for older transactions become stale and unusable.
  Clearing these old signatures prevents the accumulation of invalid data on-chain and reduces the risk of being stuck attempting to submit transactions that are guaranteed to fail.

---

How the Expiry Causes the Issue

1. Event Backlog
   The bridge daemon processes all incoming TFChain events sequentially. This includes:

   • WithdrawCreatedEvents
   • WithdrawExpiredEvents
   • RefundExpiredEvents
   • RefundReadyEvents
   • WithdrawReadyEvents (BurnTransactionReady events)

2. Processing Delays

   • High Event Volume: During periods of high activity, or after bridge outages, a large volume of events can accumulate.
   • Legacy Unhandled Transactions: The bridge also faces a backlog of transactions that previous versions were unable to process.
   • These factors cause the bridge daemon to spend significant time processing other events before reaching WithdrawReadyEvents.

3. Race Condition / Stale Signatures
   By the time the bridge daemon reaches a WithdrawReadyEvent emitted more than 2 minutes ago, the corresponding Stellar signatures have already been cleared from TFChain by the expiry mechanism.

4. Failed Processing and Accumulation
   When the bridge attempts to process such an event, it finds no valid signatures. Consequently:

   • It cannot construct and submit the stellar transaction.
   • The transaction remains in an unhandled state, contributing to the growing count of "stuck" transactions in TFChain storage.
   • This creates a continuous cycle where new BurnTransactionReady events are emitted, but processing is delayed until their signatures expire, leading to an ever-increasing backlog.

---

Proposed Solution: Batching processing TFChain events

To alleviate the processing bottleneck and prevent BurnTransactionReady events from becoming unprocessable, I propose to optimize the handling of all TFChain events.

1. Batching all TFChain calls

   • Instead of processing each event (WithdrawExpiredEvent, etc) individually, the bridge daemon will collect and process these events into a batch

2. Single Batched Transaction

   • A single Substrate extrinsic using Utility.force_all will be constructed and submitted to TFChain.
   • This batched extrinsic will contain multiple calls ( TFTBridgeModule.propose_burn_transaction_or_add_sig, proposeOrVoteMintTransactionCall, etc, each corresponding to a deposit, withdrawal, or refund operation.

3. Benefits

   • Reduced Transaction Overhead: Submitting one batched transaction instead of many individual ones reduces overhead from signing, network propagation, and block inclusion.
   • Improved Throughput: Efficiently clears the backlog of WithdrawExpiredEvents, allowing the bridge daemon to reach and process critical WithdrawReadyEvents more quickly—ideally before their 2-minute signature expiry.

---

This approach should help the bridge catch up on the existing backlog of unhandled transactions and prevent future ones from accumulating due to signature expiry.

[sameh-farouk]

I want to emphasize that withdrawals on the Stellar network occur in two steps: first, the funds are sent on the Stellar network, and then they are recorded on TFChain. This process is not properly atomic, and if you perform multiple Stellar withdrawals then subsequently submit all their TFChain confirmations at once, you increase the risk of something going wrong. This issue is not introduced by this proposal; it already needs to be addressed regardless of this proposal. It is being tracked in a separate ticket.

[LeeSmet]

I'm not really seeing how batching transactions will solve the issue with the explanation provided above. In general the bridge can ignore all transaction except those which have happened recently (less than 2 minutes ago). But actually the architecture itself is dated. This was the first bridge with multisignature operation, and the chain was used for synchronization between daemons. This introduces a bunch of overhead (since chain operations take time and permanent storage space). In an ideal situation, the architecture would be reworked to be similar to those used by other bridges, where bridge nodes communicate between eachother to exchange signatures, thus not needing any on chain operations other than the actual minting.

As for atomicity of operations, unfortunately bridging operates with 2 disjoint chains, which are themselves not linked (except through third party tools like the bridge), so a fully atomic operations is impossible. The best currently known approach would be an atomic swap accross two chains, but even there you are relying on the fact that the action on chain 2 can happen in a reasonable time, and a malicious actor could take advantage of a prolonged service outage on one of the involved chains to reclaim his funds after also receiving the funds on another chain.

[sameh-farouk]

In an ideal situation, the architecture would be reworked to be similar to those used by other bridges, where bridge nodes communicate between eachother to exchange signatures, thus not needing any on chain operations other than the actual minting.

Your points about the architecture being dated and the ideal solution involving inter-daemon communication are valid for a long-term vision.
However, batching directly addresses the immediate operational issue of transaction backlogs and expirations within the current architecture.
I agree that chain operations used for synchronization between daemons have unnecessary overhead. While batching doesn't eliminate this overhead, it is a tactical improvement that significantly boosts the bridge's throughput, directly mitigating the transaction backlog and expiration issue, even within the constraints of the current architecture.

I'm not really seeing how batching transactions will solve the issue with the explanation provided above.

Imagine TFChain Block N is finalized. This block contains:

• 20 TFTBridgeModule.BurnTransactionCreated events (WCE1 to WCE20): Users initiating TFT withdrawals.

The Bridge's Operational Flow (No batching)

Phase 1: Event Ingestion (Bridge Daemon - bridge.go)

1. Bridge Processes Block N (Time: ~0-6 seconds after Block N finalization):
   • The bridge daemon receives and processes Block N.
   • It identifies the 20 BurnTransactionCreated events (WCE1...WCE20).
   • These events are now in the bridge's internal queue of pending work.

Phase 2: Strict Sequential Processing and Individual Extrinsic Submission (The True Bottleneck)

This is where the backlog forms and the submitted signature expires. The bridge can only process one pending event at a time and submit its corresponding extrinsic.

1. Processing WCE1 (Time: ~6 seconds after Block N):

   • The bridge picks WCE1 from its internal queue.
   • It prepares and submits a propose_burn_transaction_or_add_sig extrinsic for WCE1, adding its signatures (for simplification, let's assume the bridge is running in single daemon mode where only one signature is sufficient)
   • This extrinsic is included in TFChain Block N+1.
   • The BurnTransactionReady event for WCE1 is emitted in TFChain Block N+1.
   • Crucially, this entire process (from picking WCE1 to its extrinsic being included) takes approximately 6 seconds (one block time).

2. Processing WCE2 (Time: ~12 seconds after Block N):

   • The bridge picks WCE2 from its internal queue.
   • It prepares and submits its extrinsic.
   • This extrinsic is included in TFChain Block N+2.
   • The BurnTransactionReady event for WCE2 is emitted in TFChain Block N+2.

3. ...and so on, for all 20 WCE:

   • Each subsequent WithdrawCreatedEvent consumes approximately 6 seconds of the bridge's processing time to get its extrinsic included in a block.
   • Therefore, to submit the extrinsic for WCE20, it will take the bridge approximately 20 * 6 = 120 seconds (2 minutes) of continuous, uninterrupted processing time just for these 20 events.
   • Plus more time for processing other types of events in BlockN

The Real Problem:

• Internal Queue Growth: While the bridge is busy processing WCE1 (taking 6 seconds), new events might be arriving from the subsequent block (N+1) or new Stellar deposits. These new events are added to the bridge's internal queue.
• Expiration in Queue:
  • By the time the bridge is getting to process WCE20 (after 120 seconds of processing previous events), WCE1 (which originated from Block N, and was ready since Block N+1) is now 120 seconds old. Since the expiration threshold is 2 minutes, WCE1 signatures are now expired.
  • Similarly, WCE2 is 114 seconds old, WCE3 is 108 seconds old, and so on. A significant portion of signatures added for the initial 20 WithdrawCreatedEvents from Block N will have expired while the relevant WithdrawReadyEvents are waiting in the bridge's internal queue to be processed and submitted.
• The "Catch-Up" Dilemma: The bridge, when under load, is constantly falling behind. It takes 2 minutes to process just the 20 WithdrawCreatedEvents from Block N. During those 2 minutes, new WithdrawCreatedEvents (and other events) have arrived from Block N+1 through Block N+20. The backlog isn't just from Block N; it's a continuous stream that the bridge cannot keep up with.
• Impact of Busy Blocks: Whether the bridge can "catch" WCE20 before it expires depends entirely on how busy the blocks from N+2 to N+20 are. If those blocks also contain many events, the bridge's queue will grow even faster, and more transactions will expire.

Now, back to your questions, how would batching fix this issue:

Batching directly addresses this by allowing the bridge to process many events (e.g., all 20 WCEs from Block N) within a single 6-second window, dramatically increasing its effective throughput and preventing signatures from expiring in its events internal queue.

[sameh-farouk]

We’re now at a crossroads between two options: introducing a fundamental architectural shift, which would require major development effort, new consensus mechanisms, and possibly new security considerations. This path would likely involve incorporating batching to reduce TFChain overhead when marking withdrawals as executed or processing mint operations. Alternatively, we could pursue a solution/refactor within the current architecture that directly addresses the urgent issues of transaction expiration and backlog by maximizing throughput. Both approaches have their pros and cons, and from a development perspective, I’m open to either direction. CC: @xmonader

[sameh-farouk]

As for atomicity of operations, unfortunately bridging operates with 2 disjoint chains, which are themselves not linked (except through third party tools like the bridge), so a fully atomic operations is impossible.

I proposed an idea here
#1054 (comment)

Feel free to follow the discussion regarding this matter in the relevant issue if you'd like