Syncing and the mempool

[joelreymont]

What are the tickets this depends on? Do we have all of them?

[lucafavatella]

@wagerlabs @sennui @uwiger

What are the tickets this depends on?

Tickets about the mempool and the syncing of transactions:

• Update mempool when receiving new transactions #72, that depends on Broadcast transaction #62
• Broadcast transaction #62

Tickets about the mempool and the syncing of blocks:

• For syncing blocks among nodes:
  • Describe and implement the gossip algorithm for sending out blocks #35
  • Peers utility - basic code #40
  • Send block #11
  • Receive block #12
  • Swagger endpoint to receive block #84
• For what to do on reception of block:
  • Consensus #13 (case 1) - for the simplest case of block received from network that is direct successor of local top block. The transactions in the block need to be deleted from the mempool;
  • Block with no parent #64 (case 2) - for the case of block received from network that is indirect successor of local top block. The history of the block until the local top block needs to be filled from the network, then the transactions in each block need to be deleted from the mempool;
  • Block with no parent #64 (or Handle orphaned blocks #59 ?) (case 3A) - for the case of block received from network that is not successor of local top block but has common ancestor (potentially genesis) with local top block. The history of the block until a common ancestor with the local top block needs to be filled from the network, then if the chain received from the network has more work the transactions in each received block up to the common ancestor need to be deleted from the mempool;
  • Handle orphaned blocks #59 (optimization hence optional) (case 3B) - additional action for the same case as above (the chain received from the network has common ancestor and has more work). The transactions in the orphaned blocks and not in the new longest chain need to be returned to the mempool.

Do we have all of them?

• We need to confirm that Broadcast transaction #62 is meant to implement broadcast of transactions including their reception;
  • UPDATE: Broadcast transaction #62 updated.
• We need to ensure we are on the same page on the pieces of work on "reception of block" I described above (cases 1, 2, 3A, 3B), clarify which fit in sprint 3, clarify which ticket tracks what.

[sennui]

Do you on purpose exclude ticket #68 ? On one hand its more technical and closer to implementation, on the other it is mempool and depends on choices we make in this thread.

There is fundamental question: do we sync the transactions from mempool? (transactions that are not mined)?

If we do - transaction is propagated to another nodes and included in another solution attempts in parallel. So there is chance that it will be mined sooner than if it stays on one node.

The disadvantages I see:

• more network traffic (low priority)
• potential conflicts (when transaction is included in two forks - medium priority)

I think we should sync nodes' mempools. Maybe we should add TTL to transactions to control parallelism of the mining of a transaction?

If we do sync mempools then the growth of the mempool is faster and #68 is very useful.

[lucafavatella]

@sennui

Do you on purpose exclude ticket #68 ? On one hand its more technical and closer to implementation, on the other it is mempool and depends on choices we make in this thread.

I had excluded #68 on purpose because not directly related to syncing. But yes - good to mention in order to have the full perspective.

There is fundamental question: do we sync the transactions from mempool? (transactions that are not mined)?

I understand this question impacts the scope of GH-62.

If we do - transaction is propagated to another nodes and included in another solution attempts in parallel. So there is chance that it will be mined sooner than if it stays on one node.

The disadvantages I see:

• more network traffic (low priority)
• potential conflicts (when transaction is included in two forks - medium priority)

Even without transmitting to other nodes transactions that are in mempool, the case of the same transaction being included by two concurrent distinct miners cannot be excluded.

I think we should sync nodes' mempools. Maybe we should add TTL to transactions to control parallelism of the mining of a transaction?

If we do sync mempools then the growth of the mempool is faster and #68 is very useful.

It feels a TTL would complicate things, and I would avoid that.

I am not sure yet about complexity of protocol for broadcasting transactions (that I understand in scope of GH-62). I believe a basic solution shall be implemented first as part of GH-62, and complex optional parts / fine tuning for optimal distribution deferred.

[joelreymont]

@sennui

There is fundamental question: do we sync the transactions from mempool? (transactions that are not mined)?

Yes, we do this!

Maybe we should add TTL to transactions to control parallelism of the mining of a transaction?

What does this mean exactly? Is this the time the transaction lives in the mempool before it's deleted? What is the parallelism of the mining of a transaction?

@lucafavatella Thank you, this is very comprehensive!

[joelreymont]

@sennui @lucafavatella Transactions are fungible so we'll keep them in the mempol if they already exist. There's no need to delete them just to add them back later. See #230.

Shall we close this ticket?

[lucafavatella]

For what to do on reception of block:

• (case 1) - for the simplest case of block received from network that is direct successor of local top block. The transactions in the block need to be deleted from the mempool;
• (case 2) - for the case of block received from network that is indirect successor of local top block. The history of the block until the local top block needs to be filled from the network, then the transactions in each block need to be deleted from the mempool;
• (case 3A) - for the case of block received from network that is not successor of local top block but has common ancestor (potentially genesis) with local top block. The history of the block until a common ancestor with the local top block needs to be filled from the network, then if the chain received from the network has more work the transactions in each received block up to the common ancestor need to be deleted from the mempool;

Wrt mempool, cases 1, 2 and 3A are covered in GH-232 - as I see "Delete transactions from the mempool when adding a block to our chain".

• (optimization hence optional) (case 3B) - additional action for the same case as above (the chain received from the network has common ancestor and has more work). The transactions in the orphaned blocks and not in the new longest chain need to be returned to the mempool.

Wrt mempool, I see this case 3B is covered in GH-233 - as I see "Return transactions to the mempool when discarding a block".