-
Notifications
You must be signed in to change notification settings - Fork 202
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable contract listeners with multiple filters #1418
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1418 +/- ##
==========================================
- Coverage 99.97% 99.91% -0.06%
==========================================
Files 323 322 -1
Lines 23537 23506 -31
==========================================
- Hits 23530 23485 -45
- Misses 5 12 +7
- Partials 2 9 +7 ☔ View full report in Codecov by Sentry. |
@nguyer (fyi @EnriqueL8) - there is a really good reason why we need this (at least prior to this PR). That's because calculating the signature of a listener is not an easy thing to do - and FireFly doesn't expose any way for code outside of FireFly to do it. So if you think about the common startup reconcile in most applications, where it does a query-then-create on the listeners it needs, you need to be able to match the things you get back, to the things you need. That means you need to be able to create the signature for the things you need (not just the things you create). Currently, we avoid this by having it so that if you try and recreate the new thing with the same signature, it prevents you with a 409. In fact that's important enough that we found and fixed a bug in it recently in #1433. Now I can imaging this PR might provide a proposal for a fundamentally new way to do this, by pushing the complex reconcile logic inside of listener. But we do need a clear migration path, and to know what the impact would be on apps relying on the old behavior - because without any thought they would duplicate a new listener on every startup. |
@peterbroadhurst I'm trying to pick this back up but really not sure what direction to take it. I think part of my struggle is that I myself haven't run into the use case that is driving the need for this functionality, so I'm sort of designing a solution to an unknown problem. I could add an updated version of the constraint we used to have (and maybe do some heavy DB schema changes to support that) but what would the constraint be? Topic/Location/[SOMETHING]. Is that last element of the constraint the combined signature of each filter in the listener? Do we need to prevent someone from creating the same exact Topic/Location/[Combination of Signatures] or do we want to prevent the same Topic/Location/[Single event signature] from being used in multiple listeners even if the set of filters in each of those listeners is different. Does that question make sense? I feel like this PR probably needs another round of design discussion before it's ready for more coding. |
Signed-off-by: Nicko Guyer <nicko.guyer@kaleido.io>
Signed-off-by: Nicko Guyer <nicko.guyer@kaleido.io>
Signed-off-by: Nicko Guyer <nicko.guyer@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
New approach is that we have introduced a new API at PR needs some conflict cleaning and test coverage but test in a FF CLI setup and looking good! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @EnriqueL8 - lots of mini comments here, but it all rolls up to one big question about the relationship between:
len(filters) == 1
sosignature
is likeSomeEvent(1,2,3)
len(filters) > 1
so:- Each
filter
has asignature
like likeSomeEventA(1,2,3)
etc. is queryable in thefilter[0]
JSON - The
signature
of the listener ishash(filters[*].signature)
- Each
Then the implication of this is that signature + topic
is still a uniqueness criteria on the listener (assuming it is at 1.3.0)
db/migrations/postgres/000117_update_contractlisteners_add_filters_column.up.sql
Outdated
Show resolved
Hide resolved
|
||
filters := make([]*filter, 0) | ||
// This keeps the existing behaviour where a contract listener could only listen to one event | ||
if listener.Event != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feels odd to me that it's the blockchain/ethereum
layer responsible for this array-ification.
Why isn't this done for us in a consistent deprecated field handler, in the FF/API layer, before calling this always with an array.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If then a blockchain connector supports only len(filters) == 1
, it can return an error
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah makes sense - I think this was done before for reload of contract listeners on restart, but how not standardise that on load of listeners from DB instead of here!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to address this now
internal/contracts/manager.go
Outdated
@@ -835,6 +901,50 @@ func (cm *contractManager) AddContractListener(ctx context.Context, listener *co | |||
listener.Options.FirstEvent = cm.getDefaultContractListenerOptions().FirstEvent | |||
} | |||
|
|||
// Normalize Event/EventPath + Location to list of Listeners | |||
if len(listener.Filters) == 0 { | |||
listener.Filters = append(listener.Filters, &core.ListenerFilterInput{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is what I was expecting to see as the migration from the deprecated fields.
The fact this is in the manager
level, makes me confused why any other code apart from this one bit needs to worry about the fact that "on the API boundary we have a special spelling for handling for len(filters) == 1"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have added this at the manager level and also still provided the deprecated fields in the API
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Address this in the blockchain plugin
For the second case, we need to include the location if not it will not be unique and will cause problems |
@peterbroadhurst Thanks for the comments - will amend the PR with changed. I think the challenge here is the
|
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
@peterbroadhurst From the review comments and the chances I made, came up with three other worries:
|
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
@EnriqueL8 tezosconnect doesn't support event streaming yet, so don't worry about compatibility with this connector |
@denisandreenko thanks for the input - so a follow up PR might be needed to provide the user with a better error on event streaming because it seems the code does go ahead and try and create a subscription in the Tezos connector |
Looks like Docker has been update in the Github Action and the packaged docker-compose version no longer support |
The GH action is using V1 for some reason |
This should do the trick hyperledger/firefly-cli#307 - I'll have to update the action to pull in a new version |
This comment you made is highlighted, but there's no commentary on where you got to with it @EnriqueL8 . |
Before I re-review here @EnriqueL8 - can you provide a new summary (either editing the header, or providing it below) of the externals changes. I referenced one detail point here: https://github.com/hyperledger/firefly/pull/1418/files#r1643404103 ... but then I stopped, as I wasn't sure what to base my code review against as a statement of intent. There seems to be at least two behaviors we could pick between, and I'm not sure which is planned. Common requirements to option 1/2
Option 1 Migration impact, but efficient on storage
Option 2 No migration impact, less efficient in storage
|
Thanks for the steer @peterbroadhurst! I had in mind Option 2 as the migration impact of the filtering no longer working doesn't fit in by mind as a 1.3.1 but without the event and location being the I need to wrap my head around ?filters=[["location": {
"address": "0xb0cd60ade460e797e0c9d206290ac4ed45672c60"
}] What does this give you? All the contract listeners that have one filter in that location Summary of external changes
The API would look like this: {
// Old version
"event": {}.
"location": {
"address": "0xb0cd60ade460e797e0c9d206290ac4ed45672c60"
}
// or New version
"filters": [
{
"interface": {
"id": "aaa0e410-2b5b-4815-a80a-a18f2ae59f7d"
},
"eventPath": "BatchPin",
"location": {
"address": "0xb0cd60ade460e797e0c9d206290ac4ed45672c60"
}
}
],
"name": "CustomBatchPin",
"options": {
"firstEvent": "oldest"
},
"topic": "batch-pin"
} Also I have some proposed code changes to check for duplicate filters in this API, so restrict that. Easily done my checking combination of signature + location! Will commit those up |
For firefly/pkg/database/plugin.go Lines 1051 to 1063 in c2c5a95
|
Review with @peterbroadhurst This is the final proposal for the changes:Roughly Option 2 + other changes!
How the new API LooksRequest{
"filters": [
{
"eventPath": "AnotherEvent",
"interface": {
"id": "943bdd06-1233-451c-83a3-d4f5b898db3f"
}
},
{
"eventPath": "Changed",
"interface": {
"id": "943bdd06-1233-451c-83a3-d4f5b898db3f"
}
}
],
"name": "both",
"options": {
"firstEvent": "0"
},
"topic": "both"
} Response{
"id": "34d5c0d4-7b8a-49b4-814a-8ae223084bb8",
"interface": {
"id": "943bdd06-1233-451c-83a3-d4f5b898db3f"
},
"namespace": "default",
"name": "both",
"backendId": "01907403-2148-5d62-4bac-301b827f3591",
"created": "2024-07-02T15:14:40.599194Z",
"event": {
"name": "AnotherEvent",
"description": "",
"params": [
{
"name": "from",
"schema": {
"type": "string",
"details": {
"type": "address",
"internalType": "address",
"indexed": true
},
"description": "A hex encoded set of bytes, with an optional '0x' prefix"
}
},
{
"name": "value",
"schema": {
"oneOf": [
{
"type": "string"
},
{
"type": "integer"
}
],
"details": {
"type": "uint256",
"internalType": "uint256"
},
"description": "An integer. You are recommended to use a JSON string. A JSON number can be used for values up to the safe maximum."
}
}
]
},
"signature": "*:AnotherEvent(address,uint256) [i=0];*:Changed(address,uint256) [i=0]",
"topic": "both",
"options": {
"firstEvent": "0"
},
"filters": [
{
"event": {
"name": "AnotherEvent",
"description": "",
"params": [
{
"name": "from",
"schema": {
"type": "string",
"details": {
"type": "address",
"internalType": "address",
"indexed": true
},
"description": "A hex encoded set of bytes, with an optional '0x' prefix"
}
},
{
"name": "value",
"schema": {
"oneOf": [
{
"type": "string"
},
{
"type": "integer"
}
],
"details": {
"type": "uint256",
"internalType": "uint256"
},
"description": "An integer. You are recommended to use a JSON string. A JSON number can be used for values up to the safe maximum."
}
}
]
},
"interface": {
"id": "943bdd06-1233-451c-83a3-d4f5b898db3f"
},
"signature": "AnotherEvent(address,uint256) [i=0]"
},
{
"event": {
"name": "Changed",
"description": "",
"params": [
{
"name": "from",
"schema": {
"type": "string",
"details": {
"type": "address",
"internalType": "address",
"indexed": true
},
"description": "A hex encoded set of bytes, with an optional '0x' prefix"
}
},
{
"name": "value",
"schema": {
"oneOf": [
{
"type": "string"
},
{
"type": "integer"
}
],
"details": {
"type": "uint256",
"internalType": "uint256"
},
"description": "An integer. You are recommended to use a JSON string. A JSON number can be used for values up to the safe maximum."
}
}
]
},
"interface": {
"id": "943bdd06-1233-451c-83a3-d4f5b898db3f"
},
"signature": "Changed(address,uint256) [i=0]"
}
]
} |
- DB migration for old deprecated events - New signature format with location + event signature with multiple filters Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Tested this PR E2E with an EVM chain locally, feeling quite good but I'm seeing problems in the E2E tests when creating a listener. Digging into that... |
Found a problem with the way I wrote the just in time migration... fixing that |
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
Signed-off-by: Enrique Lacal <enrique.lacal@kaleido.io>
This PR adds the ability to listen to multiple types of events on the same contract listener, by adding an array of listeners, rather than a single event signature/location per listener. The old way of creating a listener is still accepted by the API, but it will always be returned in the filters array now. This is a migration concern that needs to be documented.
Open questions
One thing I'm not sure about here, is that this PR as-is removes the uniqueness constraint on listeners by topic/location/signature. It now allows multiples. I'm not sure if this is a problem or not. I can add that constraint back, but it would likely require some more sophisticated DB changes. Which brings me to the next point...
Right now all the filters for a contract listener get serialized to JSON and stored in a single column. I lated realized this means we lose the ability to query/filter (no pun intended) by signature, location, etc. which we used to do, in order to check for duplicates. I'm not sure if this is required or not, but wanted to call it out.
Example
Create contract listener request
Create contract listener response