-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optional Content addressed messages #247
Comments
TLDR: We should do this! Sorry this post is a bit long, I figured more context was better here. My proposal is at the bottom. @AgeManning 💯 for having content-addressed (or configurable) messageIDs. However, it'd be great to understand why you think it's necessary for your particular case, as well as to make sure we cover a number of the edge cases. Current StateThe places messageIDs are currently used in this repo are:
The two Pubsub usages do not require a custom messageID (although it might be nice), since it could be implemented as part of the Validator for the given topic. However, making Gossipsub efficiently deal with many parties publishing the same content would require changes. Issues to ConsiderA few cases to consider if/when implementing this in go-libp2p-pubsub:
1+2 together imply that we cannot just add an option to Pubsub or Gossipsub, but might be able to have a per-topic option. Combining this with 3 implies that for any modifications to messageID sets we make in Pubsub or Gossipsub that we should make sure to make them per-topic 4 is a reminder that there are existing pubsub users + topics that wouldn't be served if we simply started switching to content addressed messages (which seems generally like a pretty useful idea) ProposalA proposal that could certainly be improved with greater understanding of why Eth2 wants to use content addressed messages is:
Note: we could be more efficient if we could use one cache per messageID function instead of per topic, but this is likely not worth the complexity. @AgeManning @protolambda @vyzo does this seem like a reasonable solution? |
I am very open to having this functionality, and we can implement it quite easily. |
@aschmahmann per topic custom message IDs will greatly complicate things, as we can have a message being published to multiple topics. The simple thing to do is to have an optional parameter specifying the ID function, as @AgeManning suggested. |
If you do the simple thing and make the optional parameter global then we run into problems because go-libp2p-pubsub's Pubsub implementation is a singleton. There is no way for me to both publish using content addressed messageIDs and also rebroadcast a message to a topic that expects rebroadcasting to function. @vyzo If you'd like to move ahead and use an optional parameter in PubSub that's fine, but the protocol IDs should not be If the optional parameter is at least in Gossipsub instead of PubSub then we won't break rebroadcasting and so the protocols can still be compliant. |
This is application specific configuration that is good for now; when we move to per-topic routers we can make the msgID a function a topic parameter. |
Also note that we can revise the spec to allow for arbitrary message ID functions; we'll have to revise for gossipsub v1.1 anyway. |
Is there an issue or ETA on that I can track?
Sure we can allow for arbitrary message ID functions in the spec, but until the per-topic routers issue is resolved these nodes will act like malfunctioning nodes when subscribed to topics where not everyone is using the same custom message ID function. If users wanted to run a single libp2p daemon on their machine to serve many applications they're now totally sunk. |
See PR #248, I think the function |
Currently message Id's are
source_id + sequence_number
. For my particular use-case (Eth 2), it would be nice if we could optionally set this ID based on the content of message, for example:hash(message)
.This would allow us to filter messages at the pubsub layer based on content rather than source peer.
I was thinking of adding a configuration parameter to rust-libp2p, which takes a function
f: (source_id, seq_no, message) -> String
to allow the user to specify the message id, givensource_id
,seq_no
,message
.For my case, this would simply be
hash(message)
This would be a general way to allow users to specify how messages are addressed.
I'm open to all suggestions on this and thoughts of feasibility in the go implementation.
The text was updated successfully, but these errors were encountered: