FLIP proposal to add Access Node event streaming API

[franklywatson]

Initial draft proposal for the Access Node event streaming API FLIP

[turbolent]

Nice! This is going to be very useful!

While the high-level API is described, the low-level details are still unclear to me.
The FLIP mentions the existing gRPC and REST APIs, but the examples usages are Go code.

Will this API be available for gRPC and the REST API?

For gRPC supports streaming, so I assume the endpoints will use that functionality?

If the REST API will support event streaming, how will this be implemented? HTTP long polling? Server-Sent Events?

[nvdtf]

Can we also filter events by individual fields in the event object? For example all TokensWithdrawn events with a specific from address.
This also helps filtering events by associated dapp in common contracts like NFTStorefront.

[peterargue]

Yes, I think it should be doable to add second level filtering, but I think there will be some complexity to work out, particularly with arbitrary cadence types in the event payload. I'd prefer to separate this out into its own FLIP. Would you be willing to submit a proposal? Alternatively, we could work together on it.

[nvdtf]

Yes makes sense. Happy to collaborate on the proposal!

[bluesign]

This is great I recently made a poc for this with websocket pubsub in https://events.dnz.dev ( which following state via google bucket )

I have an idea list like below, can give some inspiration. I believe adding a lot of subscription options helps a lit here.

• subscribing to latest block header
• subscribing to latest block
• subscribing to a transaction, status change
• subscribing to an account state change
• subscribing to resource movement by type ( this also includes resource creation )
• subscribing to resource change ( I tried this before , atree was a bit blocker )

Also I think adding few new AN methods to assist those would be very nice.

PS: I am totally going off topic but one more important thing ( which somehow totally ignored till now ) and would be nice to have would have some signature for all these data to prevent rouge AN providing wrong information.

[bluesign]

Another good idea would be numbering events by type, like eventIndex from the creation of the event. That way if I miss some event for some reason ( if I am using this streaming on server side ) I can go and grab missing ones easily.

[bluesign]

Oh also as we are in the subject of ExecutionData, state changes there are collection based instead of transaction based, I think if we can separate them into transactions, it can help building cool things.

[peterargue]

@turbolent

While the high-level API is described, the low-level details are still unclear to me.
The FLIP mentions the existing gRPC and REST APIs, but the examples usages are Go code.

the examples are based on the PoC gRPC implementation linked at the bottom. This flip is more geared towards alignment on API level. The technical approaches to grpc/rest based streaming are fairly well established.

Will this API be available for gRPC and the REST API?

Initially it would be implemented for gRPC, with REST to follow.

For gRPC supports streaming, so I assume the endpoints will use that functionality?

Yes, gRPC will use the native streaming support. See onflow/flow-go#3723 for a PoC

If the REST API will support event streaming, how will this be implemented? HTTP long polling? Server-Sent Events?

TBD

[peterargue]

@bluesign

This is great I recently made a poc for this with websocket pubsub in https://events.dnz.dev ( which following state via google bucket )

I have an idea list like below, can give some inspiration. I believe adding a lot of subscription options helps a lit here.

• subscribing to latest block header
• subscribing to latest block
• subscribing to a transaction, status change
• subscribing to an account state change
• subscribing to resource movement by type ( this also includes resource creation )
• subscribing to resource change ( I tried this before , atree was a bit blocker )

Also I think adding few new AN methods to assist those would be very nice.

Great point. I think once there's an established framework for streaming, it will be straight forward to add for other endpoints/data types.

PS: I am totally going off topic but one more important thing ( which somehow totally ignored till now ) and would be nice to have would have some signature for all these data to prevent rouge AN providing wrong information.

There is now an endpoint to download BlockExecutionData, which can be hashed and compared to the ID in the block's ExecutionResult. This hash covers everything in the block (not just events), but at least provides verifiability for the data.

There's a proposal in the works to break the BlockExecutionData down further so the events list is also hashable. FLIP coming soon...

[peterargue]

Another good idea would be numbering events by type, like eventIndex from the creation of the event. That way if I miss some event for some reason ( if I am using this streaming on server side ) I can go and grab missing ones easily.

Do you mean within the stream, or something else?

Oh also as we are in the subject of ExecutionData, state changes there are collection based instead of transaction based, I think if we can separate them into transactions, it can help building cool things.

Can you say more about this? If it's off-topic, we could also discuss in an issue against the flow-go repo

[bjartek]

This post is going to be a loose list of my thoughts around this. Sorry for it not beeing more coherent.

This looks like a nice feature however I think it is very important to get this right to avoid confusion. Is the main intended use case here in backend work or will this also be used in UIs?

The filtering of events on just event name is too primitive IMHO. Especially since the standard NFT/FT Event patterns allow empty to/from fields. These empty events are very often not needed. An option to filter them away would be good

One option here is to allow for more advanced filtering in this api.

Another option is to have a separate stream that is an list of events. For instance I want to fetch all Transactions that only withdraw NFT but do not Deposit. If i can use a seperate streaming method on that to group it by transaction or another field.

It would also be nice to be able to stream a sealed transaction with result as @bluesign suggests. As well as listen to state changes for a single transaction in an UI.

Another use case I can see for this feature is for instance when you open a pack in a UI. You want to see what was revealed to you (if you do packs that way). Having the ability to then subscribe to events of a particular type with a given field filter and group it by transaction would make it very usable.

I guess this feature can be composed on top of this api, but an option to actually enrich an Deposit event with data from a script would be nice. For instance add NFTView or something else to Deposit events.

[bluesign]

There is now an endpoint to download BlockExecutionData, which can be hashed and compared to the ID in the block's ExecutionResult. This hash covers everything in the block (not just events), but at least provides verifiability for the data.

Yeah I meant individual events level a bit, not sure how it can work but this is the weakest link in security. ( AN - Client communication )

Another good idea would be numbering events by type, like eventIndex from the creation of the event. That way if I miss some event for some reason ( if I am using this streaming on server side ) I can go and grab missing ones easily.
Do you mean within the stream, or something else?

Lets say 'A.0b2a3299cc857e29.TopShot.Deposit' this should have some serialNumber every time fired it should increase.

Can you say more about this? If it's off-topic, we could also discuss in an issue against the flow-go repo

basically now state changes seem to be group by Collections ( chunks ), would be nice to get them group by transaction instead.

[peterargue]

@bjartek

This looks like a nice feature however I think it is very important to get this right to avoid confusion. Is the main intended use case here in backend work or will this also be used in UIs?

This is intended to introduce a mechanism for streaming to the flow Access API, with events being the first usecase. Ideally it will be useful for both front and backend applications. It is OK if it has limited use for frontends in the initial version, as the intention is to iterate.

The filtering of events on just event name is too primitive IMHO. Especially since the standard NFT/FT Event patterns allow empty to/from fields. These empty events are very often not needed. An option to filter them away would be good

One option here is to allow for more advanced filtering in this api.

Another option is to have a separate stream that is an list of events. For instance I want to fetch all Transactions that only withdraw NFT but do not Deposit. If i can use a separate streaming method on that to group it by transaction or another field.

The vision I'm currently working with is to have a set of core APIs which provide access to all of the data in a convenient way for 80% of usecases. For the remaining 20%, open source modules can be built and shared that provide more advanced features. (80/20 not exact figures)

With that said, I think the Access API implementation should include some event payload filtering. I think that should go in a separate FLIP though since there is some additional complexity.

It would also be nice to be able to stream a sealed transaction with result as @bluesign suggests. As well as listen to state changes for a single transaction in an UI.

Absolutely. Streaming is planned for more endpoints over time.

Another use case I can see for this feature is for instance when you open a pack in a UI. You want to see what was revealed to you (if you do packs that way). Having the ability to then subscribe to events of a particular type with a given field filter and group it by transaction would make it very usable.

Can you say more about grouping by transaction. The event objects returned do include the tx ID. Is that sufficient or is something else needed for this type of usecase?

I guess this feature can be composed on top of this api, but an option to actually enrich an Deposit event with data from a script would be nice. For instance add NFTView or something else to Deposit events.

Yea, this is exactly the type of functionality that could be built into an open sourced module.

[peterargue]

@bluesign

Yeah I meant individual events level a bit, not sure how it can work but this is the weakest link in security. ( AN - Client communication )

That makes sense. This is definitely something that's needed for a fully trustless Access API. There's still some more work needed on a proof system before this is possible.

Lets say 'A.0b2a3299cc857e29.TopShot.Deposit' this should have some serialNumber every time fired it should increase.

We discussed this on discord. To capture the general ask:

• Access/Observer nodes could maintain an index of events.
• For each unique EventType, the index would maintain a sequential emitted number which counts the number of times the event was emitted.
• As an event is streamed, the specific event's emitted number could be included in the response providing a simple way for clients to track that they have received all events.

I think this is a great idea. This FLIP doesn't include anything about indexing events on nodes. That will eventually be needed so I suggest we include this idea when event indexing is added.

basically now state changes seem to be group by Collections ( chunks ), would be nice to get them group by transaction instead.

Ah, ok. This would require significant changes to the way state updates are tracked from execution nodes through the execution sync format and clients. I'd be happy to discuss the use cases more if you wanted to open an issue. It's not a quick though.

[peterargue]

I posted a summary of the discussion so far on the forums:
https://forum.onflow.org/t/flip-73-proposal-to-add-event-streaming-to-access-api/4509/2

We're planning to end the discussion phase for this FLIP on April 14, 2023.

[franklywatson]

Thanks for all the input, we're closing this FLIP as accepted. Improvements will be tabled as follow up FLIPs, per the discussion thread