New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gossipsub Protocol #898

Open
wants to merge 46 commits into
base: master
from

Conversation

4 participants
@AgeManning
Copy link

AgeManning commented Jan 29, 2019

This is an implementation of the gossipsub protocol as in the libp2p-specs.

This is feature-complete, except for the control-message piggy-backing and peer-tagging, which still need to be implemented.

Currently this implementation is not backwards compatible with floodsub nodes (see #880).

Further testing and debugging will be done over the next few weeks.


#[inline]
fn protocol_info(&self) -> Self::InfoIter {
iter::once(b"/meshsub/1.0.0")

This comment has been minimized.

@tomaka

tomaka Jan 29, 2019

Member

To expand on #880, the idea is that you can return multiple protocol names here. You can return both /meshsub/1.0.0 and /floodsub/1.0.0. Then upgrade_inbound and upgrade_outbound receive the protocol name and can generate a different output. This way you can support both protocols at the same time.

In practice, it is probably easier either:

  • To have two different implementations of ProtocolsHandler (one for gossipsub, one for floodsub) and merge then with ProtocolsHandler::select().
  • To have two different implementations of UpgradeInfo/InboundUpgrade/OutboundUpgrade and merge them with or_upgrade.
Show resolved Hide resolved protocols/gossipsub/examples/chat.rs Outdated
Show resolved Hide resolved protocols/gossipsub/regen_structs_proto.sh Outdated
Show resolved Hide resolved protocols/gossipsub/Cargo.toml Outdated
Show resolved Hide resolved protocols/gossipsub/Cargo.toml Outdated

message EncOpts {
optional EncMode mode = 1;
repeated bytes keyHashes = 2; // the hashes of the shared keys used (salted)

This comment has been minimized.

@twittner

twittner Feb 1, 2019

Member

What is used as a salt here?

This comment has been minimized.

@AgeManning

AgeManning Feb 2, 2019

Author

Good question! I don't know. I borrowed the protobuf file from the go-implementation (which is also in the specs) but it came with this comment which I didn't take notice of.
To my knowledge, message encryption is not yet implemented. See the specs here for more detail on this
and see the go-implementation, which seems to also have not implemented it yet:
https://github.com/libp2p/go-libp2p-pubsub/blob/d4589956d289b383aa66807840b5baa3f6916f9f/pubsub.go#L798

Show resolved Hide resolved protocols/gossipsub/src/layer.rs Outdated
Show resolved Hide resolved protocols/gossipsub/src/layer.rs Outdated
Show resolved Hide resolved protocols/gossipsub/src/mcache.rs Outdated
let local_peer_id = self.local_peer_id.clone();
self.forward_msg(message.clone(), local_peer_id);

let mut recipient_peers = HashMap::new();

This comment has been minimized.

@twittner

twittner Feb 1, 2019

Member

std::collections::HashSet?

This comment has been minimized.

@AgeManning

AgeManning Feb 2, 2019

Author

Yep as above.

Show resolved Hide resolved protocols/gossipsub/src/layer.rs Outdated

AgeManning added some commits Jan 15, 2019

Add logging to gossipsub
- Adds the log crate and implements logging macros
- Specifies versions for external crates
Add rust documentation to gossipsub crate.
- Adds basic documentation, overview and examples to the gossipsub
crate.
Re-introduce the initial heartbeat time config.
This commit also adds the inject_connected test.
Add subscribe tests.
- Modifies `handle_received_subscriptions` to take a reference of
subscriptions
- Adds `test_subscribe`
- Adds `test_handle_received_subscriptions`
- Adds tests for the filter in `get_random_peers`

AgeManning and others added some commits Jan 29, 2019

Add Bug fixes and further testing for gossipsub.
- Corrects the tuple use of topic_hashes
- Corrects JOIN logic around fanout and adding peers to the mesh
- Adds test_unsubscribe
- Adds test_join
Implement builder pattern for GossipsubConfig.
As suggested by @twittner - The builder pattern for building
GossipsubConfig struct is implemented.
@AgeManning

This comment has been minimized.

Copy link
Author

AgeManning commented Feb 2, 2019

@twittner - Thanks for taking the time to give the suggestions and comments!
I'll push the changes when I get back to my computer tomorrow :).

AgeManning added some commits Feb 2, 2019

Implement @twittner's suggestions.
- Uses `HashSet` where applicable
- Update `FnvHashMap` to standard `HashMap`
- Uses `min` function in code simplification.

@AgeManning AgeManning force-pushed the sigp:gossipsub branch from 578dac1 to 0a51e3f Feb 2, 2019

twittner and others added some commits Feb 2, 2019

Cleaner handling of messagelist
Co-Authored-By: AgeManning <Age@AgeManning.com>
Cleaner handling of added peers.
Co-Authored-By: AgeManning <Age@AgeManning.com>
@g-r-a-n-t

This comment has been minimized.

Copy link

g-r-a-n-t commented Feb 7, 2019

Hey guys, I'm going to be working on the piggy-backing feature.

/// A gossipsub node.
Gossipsub,
/// A Floodsub node.
Floodsub,

This comment has been minimized.

@twittner

twittner Feb 11, 2019

Member

I think this one is never used.

This comment has been minimized.

@AgeManning

AgeManning Feb 21, 2019

Author

Hey yep. It's currently not. This is reserved for when I figure out how to handle floodsub nodes in this behaviour. I'll be back on this in the next week or so. If a floodsub needs it's own behaviour, I'll remove this.

Show resolved Hide resolved protocols/gossipsub/src/layer.rs Outdated
Show resolved Hide resolved protocols/gossipsub/src/layer.rs Outdated
Show resolved Hide resolved protocols/gossipsub/src/layer.rs Outdated
// build a list of peers to forward the message to
// if we have fanout peers add them to the map
if self.fanout.contains_key(&topic_hash) {
for peer in self.fanout.get(&topic_hash).expect("Topic must exist") {

This comment has been minimized.

@twittner

twittner Feb 11, 2019

Member

If you switch to the 2018 edition this could be changed to:

if let Some(peers) = self.fanout.get(&topic_hash) {
    for p in peers {
        recipient_peers.insert(p.clone(), ());
    }
// ...

This comment has been minimized.

@AgeManning

AgeManning Feb 21, 2019

Author

Yes I will make these changes.

twittner and others added some commits Feb 21, 2019

Convert &vec to slice.
Co-Authored-By: AgeManning <Age@AgeManning.com>
Convert to lazy insert.
Co-Authored-By: AgeManning <Age@AgeManning.com>
Cleaner topic handling.
Co-Authored-By: AgeManning <Age@AgeManning.com>
@AgeManning

This comment has been minimized.

Copy link
Author

AgeManning commented Feb 21, 2019

Apologies for the delay. I'll be back finishing up small issues with this in the next week or two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment