Support for all token pool creation scenarios

[awrichar]

Background

Token pools are initialized with two steps:

1. The actual pool is initialized on a blockchain smart contract
2. The pool details (namespace, name, etc) are announced as a FireFly definition broadcast

The pool creator is responsible for sending 2 only after 1 is confirmed. However, because these steps may or may not occur on the same blockchain, there are multiple possible ways for them to play out.

Scenarios

A) 1 and 2 both arrive and are guaranteed to be in that order for all nodes

This is true for the case where tokens and messages are backed by the same blockchain (such as when using Ethereum with ERC1155). Each node may receive and record the blockchain transaction, then fill in the rest of the pool details after the announcement broadcast.

B) 1 and 2 both arrive, but 2 may arrive before 1

This is possible in cases where tokens and messages are backed by different blockchains. It should be uncommon, but networking conditions (for instance) could cause a particular node to receive 2 quickly while 1 is delayed. There must be a way to save the details from 2 but "rewind" and complete the message processing once 1 is received.

C) Only 2 arrives, and 1 must be looked up on demand

This is envisioned for cases where the "token pool" is a pre-existing set of tokens (could be on mainnet or a pre-existing private chain). In this case, step 1 for the author involves simply passing details about the existing tokens, but there will be no new blockchain transaction to define the pool. Therefore, the other nodes receiving the broadcast in step 2 must also have a way to lookup and verify the pre-existing token pool.

[awrichar]

Proposed solution for this issue:

Changes to FireFly core:

• Token pools will have a "confirmed" boolean state
• FireFly will not allow mint/burn/transfer operations to be initiated on unconfirmed pools
• When a token pool creation event is received via the fftokens plugin:
  • If the pool protocol ID matches an unconfirmed pool, the pool will be marked confirmed
  • Else if the pool transaction was initiated by this node, the pool will be announced
  • Else the event will be ignored
• When a token pool announcement is received via broadcast:
  • The broadcast contains both the pool and the transaction object
  • The pool details are stored in "unconfirmed" state
  • Then a new "activate pool" operation is triggered against the tokens plugin (which will trigger the token pool creation event)

Changes to firefly-tokens-erc1155 connector:

• The existing "/pool" API will be renamed "/createpool" for clarity
• A new "/activatepool" API will be added
  • Inputs are the pool protocol ID and (optionally) the transaction/block details from the pool creation event
  • This will trigger it to create a new listening subscription for events on that pool
  • This will cause the token pool creation event to be (re)delivered, along with any transfers beginning after creation
• The connector will always deliver all "token pool creation" events, but will not deliver any transfer events until a particular pool is activated
• This implies that the ethconnect event stream will now have a dynamic number of subscriptions instead of just 2:
  • The "base" subscription will only get token pool events, and will proxy all of them
  • For each activated pool, a new subscription will be added (and the pool ID will be incorporated into the name as a means of remembering it)
  • These "per pool" subscriptions will subscribe to both pool creation and transfer events, but will be filtered at the application level and will only proxy events related to the pool they monitor (and will ignore/immediately ack other events)

This solves all scenarios above by guaranteeing that (1) will be ignored if it arrives early, each node can tentatively store the details upon receiving (2), and then can (re)trigger (1) in order to fully confirm the pool. Essentially all scenarios are coerced to behave like (B).

[peterbroadhurst]

Else if the pool transaction was initiated by this node, the pool will be announced

Just wanted to confirm my understanding here, given the order of confirmation is reversed on the submitting nodes compared to other nodes in the network.

Submitting node

• Token pool becomes confirmed when any on-chain processing from the pool creation is completed locally
• The announcement is triggered
• The announcement pinning completes - the message is confirmed immediately as the pool is already confirmed

All other nodes

• The pinned announcement arrives - the message is not confirmed, because the pool is unconfirmed
• The token pool becomes confirmed when the event comes in
• A rewind happens at this point, so the message can confirm

[peterbroadhurst]

The "base" subscription will only get token pool events, and will proxy all of them

I'm trying to understand what these events are used for. Feel like I've still got a gap in my understanding as I can't see why they are needed. I understand a receipt subscription is needed to detect a submitted transaction completes, in order to create the subscription for that pool on the submitting node. But it sounds like we have a different type of event we're always listening for, that's separate to the transaction receipt.

[awrichar]

My proposal was simpler in some regards, ie:

• the announcement message is considered confirmed when successfully received, regardless of whether the token pool itself has been confirmed
• the submitter does not confirm the pool early, but waits for the announcement to propagate like everyone else (which triggers activation and another pool created event)

Key difference is whether we need/want the announcement message to pend until the pool is confirmed. I'm assuming it's a subset of the overall transaction, and will confirm first, with the pool and the pool transaction becoming confirmed only after the whole process is complete.

[awrichar]

The "base" subscription will only get token pool events, and will proxy all of them

I'm trying to understand what these events are used for.

We have a chicken and egg problem because the "per pool" subscriptions (created by an "activatepool" action) I've proposed here are filtered at the app level by the pool ID assigned on chain. But the only way to get that pool ID in the first place is if we're already listening for pool creation events emitted by the contract.

Hence the need for the base subscription to all pool creation events - they'll therefore be delivered to all FireFly nodes all the time, but will be ignored by FireFly unless they correlate (via transaction ID) to a previously submitted request from that node.

Also noting for clarity that the "receipt" event only contains success/failure - it does not contain the assigned pool ID. So it's not sufficient to know how to create the new subscription for the pool.

[awrichar]

And to describe it in another way - the diagram I originally drew here proposed this flow:

Submitter: create, announce
Others: receive announcement, validate

But my new proposal is:

Submitter: create, announce, receive announcement, activate
Others: receive announcement, activate

I'll redraw the picture if/when this actually goes in.

[peterbroadhurst]

the submitter does not confirm the pool early, but waits for the announcement to propagate like everyone else (which triggers activation and another pool created event)

Super, this was a piece I missed.

The pool and the pool transaction becoming confirmed only after the whole process is complete.

Nice 👍

[awrichar]

I've prototyped this flow and it seems to work well. I did notice that there are now a few inconsistencies between token pools and token transfers.

Token Pools

• All nodes will ultimately receive the pool itself, a transaction, and a definition broadcast message
• The pool object references the message by ID (used only for lookup purposes)
• The definition broadcast message is confirmed once it is received with all data (although the pool itself won't yet be confirmed) - this will send a message_confirmed event (guaranteed to happen first on each node)
• The pool and the transaction are confirmed once the message and the token pool blockchain event are both received - this will send a token_pool_confirmed event (guaranteed to happen second on each node)

Token Transfers

• All nodes will receive the transfer object, and if initiated via FireFly they will all receive a transaction - but each node may or may not receive a message (of type "transfer broadcast" or "transfer private") along with the transfer
• The transfer object references the message (if any) by hash (used for both lookup and validation)
• The transaction and transfer are confirmed as soon as they are received from the tokens chain (as it's unknown whether a message will be received or not) - this will send a token_transfer_confirmed event (guaranteed to happen first on each node)
• The message (if any) will not be confirmed until the transfer has also been received and recorded - this will send a message_confirmed event (guaranteed to happen second on each node)

These are mostly pre-existing and not introduced by this new flow, but it seems like an appropriate place to reevaluate them. Each of the differences is justifiable, just want to be sure it's not too confusing overall.

[peterbroadhurst]

The transfer object references the message (if any) by hash (used for both lookup and validation)

This one worries me a bit, as a bit as we've made having a message child quite a core part of the asynchronous nature of a bunch of other REST APIs. Even if we had to have both (and one was a bit superflous), I'd advocate the ID being the thing you get on the message child there if possible.

[peterbroadhurst]

The definition broadcast message is confirmed once it is received with all data (regardless of whether the pool itself is confirmed)

This feels a bit wonky, but I can't work out if it's a serious problem or not. Specifically we need to tell applications how to all process a consistently ordered set of events. Now here we're saying an application cannot be sure whether the messages are going to arrive after the pool is created, so if they had logic that wanted to be triggered that used the pool, they'd need to coordinate those two events themselves. That's a bit of a pain.

This was where I was thinking when I wrote #315 (comment) we'd have a rewind on the pool confirmed event, to go back and complete the message (so the message would stay unconfirmed until the associated pool creation is done).

[awrichar]

I edited my notes above slightly to note when events are fired. Additional responses below.

I'd advocate the ID being the thing you get on the message child there if possible

There are two aspects - first is the thing we write to the tokens chain alongside the transfer, to attest that it matches a given message and to allow other nodes to lookup that message. I believe this must be the hash (although it could be both the ID and hash).

Second is the thing we write to the database and return in API responses. It's probably more consistent for this to be the ID (although it could be both the ID and hash).

we're saying an application cannot be sure whether the messages are going to arrive after the pool is created

Not really - I'm actually guaranteeing the order for both pools and transfers (annotations added to my notes above). The only odd thing is that the order is flipped. For pools, the definition message confirms right away and the pool/transaction confirms afterward. For transfers, the transfer/transaction confirms right away and the message (if any) confirms afterward.

I could coerce pools to behave in the opposite order, if we think the extra effort is justified. One detail to note: there is no situation where we receive a pool announcement and already have sufficient data to confirm the pool - even if we receive the pool protocol ID first (which we often will), we have nowhere to stash it on its own, so we drop it. Therefore, when we receive the announcement, we poke the tokens connector so the pool protocol ID is (re)sent, and that's the final bit of data that allows the pool to confirm. If we want to hold the message in unconfirmed state, it would require a rewind at this point for 100% of pool announcements.

[awrichar]

Pushed #317 to conform with the solution I laid out above.

Opened #318 to track the remaining inconsistencies noted between pools and transfers.