[Oracles] Memory pool and Transaction Broadcast Model

[belane]

This issue describes the behavior of the memory pool for Oracle transactions, their verification and its broadcast model.

Transaction Broadcast Model

(https://medium.com/neo-smart-economy/on-the-importance-of-oracles-neo-3-0-and-dbft-17c37ee35f32)

Transaction Delivery Process

Blue nodes represent common P2P nodes, red nodes represent Oracle nodes, and green nodes represent consensus nodes.

The user broadcasts an Oracle transaction transaction TxA into Neo network.

TxA will be broadcast to all nodes in network, and will be stored in their own memory pool by each node.

Oracle node will execute the network requests in TxA, and make a consensus of return value with other Oracle nodes. And then append it to the transaction as OracleAgreement, and broadcast to the network.

The consensus node will select the Oracle transaction in the memory pool that has been appended with OracleAgreement to add into block and broadcast the block to the network.

Transaction Receiving Process

When a node receives a transaction,it will determine whether the transaction is an Oracle transaction.

• If transaction is not a Oracle transaction,node continue to run according to the existing logic
• If transaction is a Oracle transaction, node will determine whether it is in a known hash list.
  • If it is in a known hash list, node will update the transaction in memory pool.
  • If it is not in a known hash list, node will determine whether itself is a Oracle node.
    • If it is an Oracle node, add the transaction to the known hash list and start the Oracle process.
    • If it is not an Oracle node, continue to run according to the existing logic.

Oracle Node Process

When executing an interop service with Oracle, the node will check if the returned data already exists in the OracleResult dict.

• If existed, load cache data in OracleResult dict.
• If not existed, access external resource to get data and filter.

The VM needs to limit the number of executions according to OracleFee when executing the Opcode. If the number of executions of the OracleFee representative is less than the actual number of executions, it directly returns fault.

Oracle TX Verification Process

[vncoelho]

@belane, it is not clear to me how to avoid spam on Oracles nodes.
I mean, if someone just broadcast transactions, in which point the peer-to-peer will stop to rebroadcast it? I am not sure if the known hash list is enough.

[realloc]

Could you please clarify how can the node link the OracleTx to the invocation that was producing this OrcaleTx?

I mean VM starts to execute SC, meets Oracle call, stops executions, issues OracleTx and waits for the result. If there is no link to the parent Tx, how would other nodes executing the parent invocation Tx know that the OracleTx already exists in mempool and there is no need to issue on more?

[belane]

Do you think @vncoelho that Know Hashes is not enough? Like any transaction when it is in know hashes it will not be broadcasted anymore.

During an oracle transaction, the only extra exchange that exists is the transfer of OracleResult between Oracle nodes (or the possible consensus) and these messages will be signed by the Oracle nodes so any node can identify fake OracleResult messages.

@realloc when an oracle transaction is received, it is dded into the mempool, but not verified.

Normal nodes will not execute the transaction until it is verified and enters in a block, at that point they will have all the necessary data to run it.

Only Oracle nodes will execute unverified Oracle transactions in order to generate OracleResult.

[realloc]

@belane I mean not only the OracleTx, but both SC invocation Tx and resulting Oracle Tx.

[belane]

I think the transaction (invocation tx) will remain the same and will be included into a block. To run this transaction a node will need the transaction itself and the verified OracleResult linked to the transaction Hash.

It is still not very clear to me how to store in the blockchain the OracleResult agreed by the consensus.

[erikzhang]

We can store the result in the blocks. There's a ConsensusData structure in blocks.

https://github.com/neo-project/neo/blob/master/src/neo/Network/P2P/Payloads/ConsensusData.cs

[realloc]

@erikzhang But result refers to a particular Oracle transaction, not the block. Transaction invoking a SC that call Oracles may have more then one Oracle invocation, hence many dependent Oracle Transactions that it would be waiting to enter the block. It means each Oracle Tx should hold the result, not the block. Otherwise it would be impossible to have both SC invocation Tx and depending Oracle TXs in one block or have enough Oracle Tx results there.

[erikzhang]

You can put multiple results in a single block, there is no problem.

[realloc]

@erikzhang You can, but it breaks the semantics. Logically Oracle call is a dependent transaction, the same as SC invocation, so there appears a logical link between transactions, not between the SC invocation and a block.