Add extended providers support #292

ischasny · 2022-11-11T10:49:45Z

This PR adds support for extended providers. It's a programmatic API only. CLI and configuration support will follow up if there going to be a need for those.

In particular, the PR adds xproviders.AdBuilder class - a builder for constructing, signing and verifying an ad that can be then published using Engine.

codecov-commenter · 2022-11-11T10:59:35Z

Codecov Report

Merging #292 (cdba1f6) into main (8ddbc29) will decrease coverage by 0.09%.
The diff coverage is 45.65%.

❗ Current head cdba1f6 differs from pull request most recent head 87cebe6. Consider uploading reports for the commit 87cebe6 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #292      +/-   ##
==========================================
- Coverage   44.64%   44.55%   -0.10%     
==========================================
  Files          78       80       +2     
  Lines        6018     6202     +184     
==========================================
+ Hits         2687     2763      +76     
- Misses       2907     3011     +104     
- Partials      424      428       +4

Impacted Files	Coverage Δ
engine/extended_providers/extended_providers.go	`0.00% <0.00%> (ø)`
engine/xproviders/xproviders.go	`80.95% <80.95%> (ø)`
testutil/testutil.go	`94.26% <100.00%> (+0.86%)`	⬆️
mirror/selectors.go	`82.35% <0.00%> (-17.65%)`	⬇️
mirror/store.go	`45.55% <0.00%> (-2.23%)`	⬇️
mirror/mirror.go	`52.86% <0.00%> (-1.02%)`	⬇️

ischasny · 2022-11-11T11:48:19Z

@gmelodie @hannahhoward this PR adds support for provider families. Please have a look whether it's sufficient for your usecase. Or would you need to have CLI support too?

ischasny · 2022-11-11T11:48:57Z

go.mod

 	github.com/filecoin-project/go-state-types v0.1.0
-	github.com/filecoin-project/storetheindex v0.4.28
+	github.com/filecoin-project/storetheindex v0.4.30-0.20221109112339-8dad1e315c7e


I will update the reference before merging the PR.

README.md

masih · 2022-11-11T12:00:50Z

engine/extended_provider/extended_providers_publisher.go

@@ -0,0 +1,155 @@
+package extended_provider


What is the rationale for separating the package? Would be nice if we don't have to do this and the functionality simply extends the existing API?

My original idea was to add this API into the main interface, however if you think about it - there is nothing fundamentally new that has been added. I.e. the core engine functionality and the APIs that are exposed from it already are sufficient for the extended providers usecase too.

Hence instead of modifying the existing APIs that would require at least 2 new parameters (override and ExtendedProviders) I decided to go with a layer on top of the existing engine. I also envision that the extended providers APIs are going to be used way less frequent comparing to the main Publish and NotifyPut. So modifying those to satisfy extended providers usecase was not an option either.

masih · 2022-11-11T12:05:27Z

engine/extended_provider/extended_providers_publisher.go

+//
+// - contextID and metadata fields are optional and should be used according to the specification
+// - providers list should contain only those providers whos infos have been registered in the publisher
+func (pub *ExtendedProvidersPublisher) Publish(ctx context.Context, contextID []byte, metadata []byte, override bool, providers ...peer.ID) (cid.Cid, error) {


Wy not take Advertisement straight up? The number of arguments to this function seems to be a tad high.

If the idea is to assist with construction of Advertisement then maybe it would be more intuitive to abstract away AdvertisementBuilder instead of Publish.

I thought about the builder. The issue is that assistance is required not just in the building, but in the signing too. Would your concern be alleviated if instead of passing in providers as a multiarg, I use array instead?

It's just +1 argument to what NotifyPut takes currently:

NotifyPut(ctx context.Context, provider *peer.AddrInfo, contextID []byte, md metadata.Metadata) (cid.Cid, error)

I thought about the builder while jogging and liked the idea :) Pushed the change in.

ischasny · 2022-11-11T16:12:06Z

@masih ready for re-review

gammazero

I must insist on package and type name changes. See comments.

README.md

engine/extended_provider/extended_providers.go

engine/extended_provider/extended_providers_test.go

ischasny · 2022-11-11T16:32:37Z

@gammazero fixes pushed please re-review

gammazero

A couple more suggested changes, but not blocking. Do please change the directory name.

engine/extended_providers/extended_providers.go

gmelodie · 2022-11-11T18:12:31Z

Tagging @elijaharita

willscott · 2022-11-14T12:06:30Z

engine/extended_providers/extended_providers.go

+	// Priv contains a provtae key of the extended provider
+	Priv crypto.PrivKey


Suggested change

// Priv contains a provtae key of the extended provider

Priv crypto.PrivKey

// Priv contains a private key of the extended provider

Priv crypto.PrivKey

are there cases where the publishing process won't have the private keys of the whole family?

I thought a lot about it. In our current usecase - allow a provider to scale up by adding new bitswap aliases on other libp2p hosts - the OG provider should have access too all XP keys.

If going forwards we have such usecase when the main provider and it's XPs are in different security domains, then we can extend the protocol between two different index-providers to be able to send partially signed ads to each other and assemble all required signatures that way.

I'd encourage checking with the providers to understand how this works with their current key management strategy - i don't think they have the keys being shared back currently, so if they're going to add a message protocol for comms to support that, we may need to jump to also supporting the partial signing

Good point, noted.

engine/xproviders/xproviders.go

xproviders.AdBuilder allows to easily create, verify and sign extended provider ads that then can be published using Engine

hannahhoward

I may be missing it, but the missing feature for me is removing extended providers. While I suspect it will be a while before we do this on an cid by cid basis, from the beginning providers should be able to turn off bitswap after they enable it. If I'm missing something here, let me know.

ischasny · 2022-11-14T18:56:23Z

I may be missing it, but the missing feature for me is removing extended providers. While I suspect it will be a while before we do this on an cid by cid basis, from the beginning providers should be able to turn off bitswap after they enable it. If I'm missing something here, let me know.

Hey @hannahhoward , the idea is that you can publish an ad less the provider that needs to be removed. I.e. there is no a dedicated "remove" ad, just a new ad with an updated list. Would that work for you?

hannahhoward · 2022-11-14T19:35:02Z

I think as long as you can do republish without the extended provider to remove, that works.

ischasny force-pushed the ivan/provider-families branch from 35cc24e to d718b4c Compare November 11, 2022 11:36

ischasny requested review from masih, gammazero and willscott November 11, 2022 11:39

ischasny marked this pull request as ready for review November 11, 2022 11:39

ischasny commented Nov 11, 2022

View reviewed changes

masih reviewed Nov 11, 2022

View reviewed changes

gammazero requested changes Nov 11, 2022

View reviewed changes

gammazero approved these changes Nov 11, 2022

View reviewed changes

engine/extended_providers/extended_providers.go Outdated Show resolved Hide resolved

engine/extended_providers/extended_providers.go Outdated Show resolved Hide resolved

ischasny force-pushed the ivan/provider-families branch 3 times, most recently from be80da9 to 2991f96 Compare November 14, 2022 11:55

willscott reviewed Nov 14, 2022

View reviewed changes

ischasny force-pushed the ivan/provider-families branch from cdba1f6 to 87cebe6 Compare November 14, 2022 12:18

Add extended providers builder

5afd290

xproviders.AdBuilder allows to easily create, verify and sign extended provider ads that then can be published using Engine

ischasny force-pushed the ivan/provider-families branch from 87cebe6 to 5afd290 Compare November 14, 2022 12:27

hannahhoward requested changes Nov 14, 2022

View reviewed changes

hannahhoward approved these changes Nov 15, 2022

View reviewed changes

hannahhoward mentioned this pull request Nov 15, 2022

Support exposing booster bitswap peer directly to indexer via extended providers filecoin-project/boost#951

Closed

ischasny merged commit 60add5f into main Nov 15, 2022

ischasny deleted the ivan/provider-families branch November 15, 2022 10:38

willscott mentioned this pull request Nov 21, 2022

Implement signing mechanism for families of providers ipni/storetheindex#678

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add extended providers support #292

Add extended providers support #292

ischasny commented Nov 11, 2022 •

edited

codecov-commenter commented Nov 11, 2022 •

edited

ischasny commented Nov 11, 2022

ischasny Nov 11, 2022

masih Nov 11, 2022

ischasny Nov 11, 2022

masih Nov 11, 2022

ischasny Nov 11, 2022

ischasny Nov 11, 2022

ischasny commented Nov 11, 2022

gammazero left a comment

ischasny commented Nov 11, 2022

gammazero left a comment

gmelodie commented Nov 11, 2022

willscott Nov 14, 2022

ischasny Nov 14, 2022

willscott Nov 14, 2022

ischasny Nov 14, 2022

hannahhoward left a comment

ischasny commented Nov 14, 2022

hannahhoward commented Nov 14, 2022

		// Priv contains a provtae key of the extended provider
		Priv crypto.PrivKey

Add extended providers support #292

Add extended providers support #292

Conversation

ischasny commented Nov 11, 2022 • edited

codecov-commenter commented Nov 11, 2022 • edited

Codecov Report

ischasny commented Nov 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ischasny commented Nov 11, 2022

gammazero left a comment

Choose a reason for hiding this comment

ischasny commented Nov 11, 2022

gammazero left a comment

Choose a reason for hiding this comment

gmelodie commented Nov 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hannahhoward left a comment

Choose a reason for hiding this comment

ischasny commented Nov 14, 2022

hannahhoward commented Nov 14, 2022

ischasny commented Nov 11, 2022 •

edited

codecov-commenter commented Nov 11, 2022 •

edited