Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: peering implementation #260

Merged
merged 13 commits into from
Jun 13, 2022
Merged

feat: peering implementation #260

merged 13 commits into from
Jun 13, 2022

Conversation

alvin-reyes
Copy link
Contributor

@alvin-reyes alvin-reyes commented Jun 9, 2022

Changes

Implemented Peering based on IPFS peering

  • Introduce Peering Service
  • Introduce Peering Endpoints (add, remove, list, start, stop, get state, check connection)
  • Introduce Pre-defined PeeringPeers based on content providers here

Related Issue

#259

How to use

Add peering.peers via estuary cli

./estuary --peering-peers '[{"ID": "12D3KooWEGeZ19Q79NdzS6CJBoCwFZwujqi5hoK8BtRcLa48fJdu", "Addrs": ["/ip4/145.40.96.233/tcp/4001"] },{ "ID": "12D3KooWBnmsaeNRP6SCdNbhzaNHihQQBPDhmDvjVGsR1EbswncV", "Addrs":["/ip4/18.1.1.2/tcp/4001", "/ip4/147.75.87.85/tcp/4001"]}]'

Add peering.peers via estuary-shuttle cli

./estuary-shuttle --peering-peers '[{"ID": "12D3KooWEGeZ19Q79NdzS6CJBoCwFZwujqi5hoK8BtRcLa48fJdu", "Addrs": ["/ip4/145.40.96.233/tcp/4001"] },{ "ID": "12D3KooWBnmsaeNRP6SCdNbhzaNHihQQBPDhmDvjVGsR1EbswncV", "Addrs":["/ip4/18.1.1.2/tcp/4001", "/ip4/147.75.87.85/tcp/4001"]}]'

Peering endpoints

Add peers

curl --location --request POST 'localhost:3004/admin/peering/peers/add' \
--header 'Authorization: Bearer [REDACTED]' \
--header 'Content-Type: application/json' \
--data-raw '[{"ID": "12D3KooWEGeZ19Q79NdzS6CJBoCwFZwujqi5hoK8BtRcLa48fJdu", "Addrs": ["/ip4/145.40.96.233/tcp/4001"] },{ "ID": "12D3KooWBnmsaeNRP6SCdNbhzaNHihQQBPDhmDvjVGsR1EbswncV", "Addrs":["/ip4/18.1.1.2/tcp/4001", "/ip4/147.75.87.85/tcp/4001"]}]'

Remove

curl --location --request DELETE 'localhost:3004/admin/peering/peers/remove' \
--header 'Authorization: Bearer [REDACTED]' \
--header 'Content-Type: application/json' \
--data-raw '["12D3KooWEGeZ19Q79NdzS6CJBoCwFZwujqi5hoK8BtRcLa48fJdu"]'

List

curl --location --request GET 'localhost:3004/admin/peering/peers/list' \
--header 'Authorization: Bearer [REDACTED]'

Start

curl --location --request POST 'localhost:3004/admin/peering/start' \
--header 'Authorization: Bearer [REDACTED]'

Stop

curl --location --request POST 'localhost:3004/admin/peering/stop' \
--header 'Authorization: Bearer [REDACTED]'

Verification

Add peers

image

Remove peers

image

List all peers

image

Start peering

image

Stop peering

image

@en0ma
Copy link
Contributor

en0ma commented Jun 9, 2022

@alvin-reyes why not use the go-ipfs peering package at https://github.com/ipfs/go-ipfs/tree/master/peering

@alvin-reyes alvin-reyes marked this pull request as ready for review June 9, 2022 19:03
@alvin-reyes alvin-reyes requested review from gmelodie, whyrusleeping and en0ma and removed request for gmelodie and whyrusleeping June 9, 2022 19:12
node/node.go Outdated Show resolved Hide resolved
node/node.go Outdated Show resolved Hide resolved
node/node.go Outdated Show resolved Hide resolved
main.go Outdated Show resolved Hide resolved
main.go Outdated Show resolved Hide resolved
handlers.go Outdated

peerParamId, _ := peer.Decode(peerParam.ID)
// add the peer(s)
s.Node.Peering.AddPeer(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not build/verify the multiaddr before adding it for peering? if it fails halfway, it means partial adding. Except this it your intention

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@en0ma Not sure what you mean. multiaddr.NewMultiaddr parses and verifies the multiaddr. So we only add a peer if all their multiaddrs are correctly parsed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant the complete peer sent for the request - if there are multiple peers, 1 fails and 1 pass - it adds 1, yet returns an error for the request. So my question is, is that intended?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah maybe we should add all of them together. If one fails we don't add anything

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if one of the address is invalid, then the entire request should be invalid. It would be best for us to impose an absolute correctness of the addresses and inform the consumer if one of them is invalid.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alvin-reyes on line 479, it is not all or nothing - which is what I am pointing at

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That's why we return a json response immediately if one of the addrs is invalid.

Line 466 - 475

for _, addr := range peerParam.Addrs {
			a, err := multiaddr.NewMultiaddr(addr)
			if err != nil {
				log.Errorf("handlePeeringPeersAdd error: %s", err)
				return c.JSON(http.StatusBadRequest,
					util.PeeringPeerAddMessage{
						"Adding Peer(s) on Peering failed, the addr is invalid" + addr, params})
			}
			multiAddrs = append(multiAddrs, a)
		}

If one of them is invalid, the entire request to add will be cancelled with StatusBadRequest response. It'll also show the message and list of invalid addrs.

image

Copy link
Contributor

@en0ma en0ma Jun 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are looking at per addr level. I am talking about per peer level. If 2 peers are sent - if 1 is added and the other fails

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If one of the peers are invalid, then we reject the entire request and raise the error similar to how we handle peer addrs level.

image

It make sense to reject the entire request if one of the peers or peer.addrs are invalid.

handlers.go Outdated Show resolved Hide resolved
handlers.go Show resolved Hide resolved
cmd/estuary-shuttle/main.go Outdated Show resolved Hide resolved
main.go Show resolved Hide resolved
handlers.go Outdated Show resolved Hide resolved
handlers.go Outdated Show resolved Hide resolved
handlers.go Show resolved Hide resolved
handlers.go Outdated

peerParamId, _ := peer.Decode(peerParam.ID)
// add the peer(s)
s.Node.Peering.AddPeer(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@en0ma Not sure what you mean. multiaddr.NewMultiaddr parses and verifies the multiaddr. So we only add a peer if all their multiaddrs are correctly parsed

handlers.go Outdated
Comment on lines 486 to 489
type PeeringPeerRemoveMessage struct {
Message string `json: Message`
PeersRemove []peer.ID `json: Peers`
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: put this on util/peering.go

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. I think this should be in a model package rather than util. We can revisit once we start refactoring the codebase.

handlers.go Outdated Show resolved Hide resolved
handlers.go Show resolved Hide resolved
Comment on lines 31 to 45
func (ps *EstuaryPeeringService) Stop() error {
return ps.PeeringService.Stop()
}

func (ps *EstuaryPeeringService) ListPeers() []peer.AddrInfo {
return ps.PeeringService.ListPeers()
}

func (ps *EstuaryPeeringService) AddPeer(info peer.AddrInfo) {
ps.PeeringService.AddPeer(info)
}

func (ps *EstuaryPeeringService) RemovePeer(peerId peer.ID) {
ps.PeeringService.RemovePeer(peerId)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why we need to wrap everything up under the name of EstuaryPeeringService. Can't we simply use ps.PeeringService altogether?

Copy link
Contributor Author

@alvin-reyes alvin-reyes Jun 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to make sure that we are loosely couple from the implementation so that we can extend the functionality. I wrapped it so that can add more features and decorate this later if needed.

node/node.go Outdated Show resolved Hide resolved
node/node.go Outdated Show resolved Hide resolved
handlers.go Outdated Show resolved Hide resolved
@alvin-reyes alvin-reyes requested a review from en0ma June 12, 2022 13:39
@alvin-reyes alvin-reyes dismissed gmelodie’s stale review June 12, 2022 14:58

Changed based on feedback

handlers.go Outdated Show resolved Hide resolved
@alvin-reyes alvin-reyes requested a review from en0ma June 13, 2022 15:45
Copy link
Contributor

@en0ma en0ma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alvin-reyes alvin-reyes merged commit 6042a9a into master Jun 13, 2022
@en0ma en0ma deleted the peering branch June 14, 2022 14:35
@alvin-reyes alvin-reyes self-assigned this Jun 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants