prefix lookup implementation #2

noot · 2022-09-13T17:23:52Z

add prefixLength as a config option to the DHT
lookups: if prefixLength is set, only lookup prefix of hash(CID)
queried nodes: if the query key is a prefix (ie. <32 bytes), look up and return providers via prefix
also return the full key the provider providers, so the lookup node knows what peers to discard
had to modify the Message_Peer message type to include the key the peer provides (maybe there's a nicer way to do this?)

guillaumemichel

Looks good to me!

guillaumemichel · 2022-09-28T13:41:31Z

providers/providers_manager.go

@@ -317,6 +361,7 @@ func (pm *ProviderManager) getProviderSetForKey(ctx context.Context, k []byte) (
 	}

 	if len(pset.providers) > 0 {
+		// TODO: if this is a prefix, do we want to cache it?


Is it possible to add only the target CID to the cache (if found), and not all content matching the prefix?

yes, I can change this to only cache full CIDs!

…-dht into noot/prefix-lookup

guillaumemichel · 2022-10-20T10:20:54Z

providers/providers_manager.go

@@ -296,6 +319,27 @@ func (pm *ProviderManager) GetProviders(ctx context.Context, k []byte) ([]peer.A
 	}
 }

+// GetProvidersForPrefix returns the set of providers with the given prefix, as well
+// as the full key they provide.
+func (pm *ProviderManager) GetProvidersForPrefix(ctx context.Context, k []byte) (map[peer.ID][][]byte, error) {


Why do we return map[peer.ID][][]byte here? Wouldn't it be better to return map[[]byte]peer.ID? In the end we only care about one provider record associated with the CID. If we have a map[peer.ID][][]byte we need to iterate on the full map.
How does it integrate with provider record decryption?

the map value is what keys with the prefix the peer has. it's possible the peer has multiple keys with the same prefix, so that's why it's [][]byte.

for provider record decryption, the peer.ID is actually encrypted.

But when requesting a specific key, we expect Pr[many peerID for one CID matching prefix] >> Pr[many CIDs matching prefix provided by the same peerID] as if a provider provides multiples CIDs, they will be stored at random locations in the DHT, hence we don't expect to have multiple hits for the same content provider for a single request (targeting a specific location).

I think that it would make more sense to have a map: second hash (dht identifier) -> list of encrypted peer.ID

What do you think?

okay, I guess even though that case is unlikely, it still could potentially happen (especially if the prefix length chosen isn't that short). =

I think a map of double hash -> encrypted peer IDs would also work, I think the reason I didn't do that is due to the format of the Message_Peer type and how the pb messages are sent generally - they are sent by peer, not by provided data (since before there were no prefixes). let me revisit this and try to find a better solution!

update: I've updated the code to now use map[string][]*peer.AddrInfo for prefix lookups, where the map key is the full lookup key. I think it's cleaner now, let me know what you think!

dht.go

handlers.go

…uery is for a prefix key

guillaumemichel · 2022-11-07T13:15:25Z

Would it make things easier to add a Prefix type? It may be easier to manage prefix length and could be more efficient than bit strings in the map[string][]*peer.AddrInfo

…okup

guillaumemichel · 2022-12-06T15:38:54Z

dht.go

@@ -419,6 +431,21 @@ func makeRoutingTable(dht *IpfsDHT, cfg dhtcfg.Config, maxLastSuccessfulOutbound
 	return rt, err
 }

+// SetPrefixLength sets the prefix length for DHT provider lookups.
+func (dht *IpfsDHT) SetPrefixLength(prefixLength int) error {


Is SetPrefixLength called somewhere in go-libp2p-kad-dht? Or is this function meant to be called by kubo or go-libp2p to configure the DHT settings?

no, it's meant to be called externally. I needed this for testing, but thought it could be useful for users of the library also, since the only other way to set the prefix length is at DHT initialization.

guillaumemichel · 2022-12-06T15:41:33Z

internal/net/message_manager.go

@@ -97,7 +97,7 @@ func (m *messageSenderImpl) SendRequest(ctx context.Context, p peer.ID, pmes *pb

 	stats.Record(ctx,
 		metrics.SentRequests.M(1),
-		metrics.SentBytes.M(int64(pmes.Size())),
+		metrics.SentBytes.M(int64(pmes.XXX_Size())),


What does the XXX stand for?

I don't know, it's auto-generated from protobuf, haha

guillaumemichel · 2022-12-06T15:58:16Z

providers/provider_set.go

-	set       map[peer.ID]time.Time
+	providers      []peer.ID
+	set            map[peer.ID]time.Time
+	keyToProviders map[string][]peer.ID // TODO change to map of maps, otherwise there can be duplicate peer IDs


I think that it would indeed make sense to have providers as a map, and keyToProviders as a map of maps 👍🏻

actually now that I thought about it, I think these are okay as slices because there can't be duplicate provider records in the datastore, since the DB key is the provider ID + provided CID essentially. so when these slices are written to while the database is being iterated over, it shouldn't be possible to have duplicate peer IDs.

guillaumemichel · 2022-12-07T09:18:53Z

pb/message.go

+		Type: typ,
+		Key: &Message_Key{
+			Key:             key,
+			PrefixBitLength: uint32(prefixBitLength),


uint16 should be enough, but I am totally unfamiliar with how pb works and whether the type can actually be changed

there's no uint16 type in protobuf sadly :( uint32 is the smallest

guillaumemichel · 2022-12-07T10:57:20Z

dht.go

@@ -197,6 +203,12 @@ func New(ctx context.Context, h host.Host, options ...Option) (*IpfsDHT, error)
 		return nil, err
 	}

+	dht.prefixLength = cfg.PrefixLookupLength
+	if dht.prefixLength >= 256 {


General question: Is the prefix length set globally for the kubo node, or can it be changed for each request (e.g with Options, or argument)?

it can be changed for each request by calling SetPrefixLength right before the lookup. Another option I just thought of (that might be nicer API-wise?) would be to create an option specifically for the FindProviders function that the caller can optionally pass in. what do you think?

Yeah, that would be awesome if the prefix can be set on a per request basis, with a global default value (either static e.g taken from the conf, or adaptive) for if the option isn't specifically set.

noot added 10 commits August 31, 2022 16:30

wip on prefix lookup

cc4e42b

merge

a8900f4

wip

648fd4e

merge, fix kbucket dependency, start impl

f528622

modify providers manager to return full keys for prefix lookup

1fc0b02

update pb.Message_Peer to have provides keys for prefix lookup

681dd33

discard peers that don't provide desired content

adda234

cleanup comments, update fullrt for prefix lookup

8bba3ce

cleanup

a0577de

cleanup

6c648f1

noot changed the title ~~[wip] prefix lookup implementation~~ prefix lookup implementation Sep 20, 2022

noot marked this pull request as ready for review September 20, 2022 10:07

guillaumemichel approved these changes Sep 28, 2022

View reviewed changes

noot added 4 commits September 29, 2022 13:12

Merge branch 'noot/double-hash' of github.com:ChainSafe/go-libp2p-kad…

063e99e

…-dht into noot/prefix-lookup

don't cache for prefix lookups

88d68c7

update go-datastore dep

a9ce60b

add prefixLength check

5c1eda5

guillaumemichel reviewed Oct 20, 2022

View reviewed changes

noot added 7 commits October 27, 2022 16:02

slice lookup key by bits instead of bytes

675dfaa

add SetPrefixLength

32b9a49

update prefix lookup to be by bits

e2434a9

update protobuf message

2dcb661

use map of keys to list of peers for prefix lookups

bf1f5b4

fixes and cleanups, unit tests ok

780e8b7

merge, fix lint errors

af92e26

noot commented Nov 2, 2022

View reviewed changes

dht.go Outdated Show resolved Hide resolved

noot commented Nov 2, 2022

View reviewed changes

handlers.go Outdated Show resolved Hide resolved

noot added 2 commits November 2, 2022 18:01

cleanup

8abd351

update pb.Message.Key to have prefixBitLength; use this to check if q…

3129f21

…uery is for a prefix key

noot added 5 commits November 17, 2022 11:27

merge

d8b4570

small cleanups

ca5e8ca

cleanup

6a48f02

update pb not to use map for provider peers, appears to fix prefix lo…

3708392

…okup

fix various prefix lookup bugs; change key encoding in db to hex

6884c30

umgefahren mentioned this pull request Nov 25, 2022

protocols/kad: Add double hasing for privacy to libp2p-kad libp2p/rust-libp2p#3169

Open

noot added 9 commits November 29, 2022 11:14

prefix testing

236de05

finally fix routing issue

3c81d7f

cleanup, add unit tests

24e8f3b

fix db check

9a733d2

add concurrency safety for dht.prefixLength

af9c81c

cleanup

2415ab3

update Sha256Multihas to return extra length

96cedef

cleanup

bd3535e

prefixLength 0 set to 256

19a3d31

guillaumemichel reviewed Dec 6, 2022

View reviewed changes

guillaumemichel reviewed Dec 7, 2022

View reviewed changes

noot added 4 commits December 8, 2022 15:07

cleanup

3b82cf8

update comment

30979a1

remove chainsafe/go-datastore usage

0feb15d

log cleanup

74ff325

umgefahren mentioned this pull request Jul 27, 2023

AutoNATv2 implementation for rust-libp2p filecoin-project/devgrants#1421

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prefix lookup implementation #2

prefix lookup implementation #2

noot commented Sep 13, 2022 •

edited

guillaumemichel left a comment

guillaumemichel Sep 28, 2022

noot Sep 29, 2022

guillaumemichel Oct 20, 2022

noot Oct 21, 2022

guillaumemichel Oct 23, 2022

noot Nov 2, 2022 •

edited

guillaumemichel commented Nov 7, 2022

guillaumemichel Dec 6, 2022

noot Dec 7, 2022

guillaumemichel Dec 6, 2022

noot Dec 7, 2022

guillaumemichel Dec 6, 2022

noot Dec 8, 2022

guillaumemichel Dec 7, 2022

noot Dec 7, 2022

guillaumemichel Dec 7, 2022

noot Dec 7, 2022

guillaumemichel Dec 7, 2022

prefix lookup implementation #2

Are you sure you want to change the base?

prefix lookup implementation #2

Conversation

noot commented Sep 13, 2022 • edited

guillaumemichel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noot Nov 2, 2022 • edited

Choose a reason for hiding this comment

guillaumemichel commented Nov 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noot commented Sep 13, 2022 •

edited

noot Nov 2, 2022 •

edited