-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
booster bitswap MVP executable #707
Conversation
Note: the build is probably broken until the libp2p 0.21 updates filter through the Lotus stack |
Thanks for putting this together 🙏
|
What's the plan for index update to indicate bitswap is available? is this a separate libp2p identity (or at least multiaddr host / port endpoint) the provider will expose? |
I'd say it should be configurable on the SP side: "announce using X and Y" |
I added an issue for tracking the index announcement work, #711, we should clarify what needs to happen in there. I added the address registration as a consideration we need to account for. |
add commands to build booster-bitswap, and very a round tripped successful fetch from booster-bitswap
a1b2c17
to
6c15199
Compare
I did a bit of refactoring to clean up some unused variables etc, and rebased on top of |
I was able to retrieve a block using IPFS:
|
Note: As part of productionizing this PR we need to look at optimizations in the DAG store PR: at the moment it's not very efficient in the way that it checks if a block is in a piece that is unsealed and free (cost is zero) to download. |
👍 on refactoring changes @dirkmc |
I'm not sure we need the
I'd suggest that we pass the piece CID but not the payload CID to We should also add caching per piece cid for |
@dirkmc I've made the requested changes in go-fil-markets and move the rest of the is-free-and-unsealed code within this repo. |
I've updated the base branch on this to release/lotus1.17.2, we can use that for landing all upcoming changes that require the libp2p update that will land with 1.17.2 |
For the rest of the outstanding TODO's we want to complete after merging this, can we get issues for those created and tracked in #709 so we can move forward with the merge here? Is there anything else outstanding to land this PR other than creating those issues? I'd like to land this by eod Tuesday to unblock the tracing integration. |
2077903
to
2d43d71
Compare
* feat(booster-bitswap): booster bitswap MVP untested * refactor(booster-bitswap): use API for fetching blocks * fix(deps): update deps to compile * feat(booster-bitswap): makefile & fixes add commands to build booster-bitswap, and very a round tripped successful fetch from booster-bitswap * refactor: clean up unused vars etc * fix: booster-bitsawp - check error when creating libp2p key * refactor(node): avoid FreeAndUnsealed method Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com>
* booster bitswap MVP executable (#707) * feat(booster-bitswap): booster bitswap MVP untested * refactor(booster-bitswap): use API for fetching blocks * fix(deps): update deps to compile * feat(booster-bitswap): makefile & fixes add commands to build booster-bitswap, and very a round tripped successful fetch from booster-bitswap * refactor: clean up unused vars etc * fix: booster-bitsawp - check error when creating libp2p key * refactor(node): avoid FreeAndUnsealed method Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * booster-bitswap devnet and tracing (#796) * return ipld ErrNotFound from remote blockstore interface (#798) * fix: return ipld ErrNotFound from remote blockstore interface * test: add more tests for ipld ErrNotFound * test: comment out part of TestDummydealOnline that is flaky due to a bug in latest lotus (#802) * fix normaliseError nil ptr dereference (#803) * feat: shard selector (#807) * LoadBalancer for bitswap (and later, more of libp2p) (#786) * feat(loadbalancer): add message types * feat(messages): add utility functions * feat(loadbalancer): initial load balancer impl implementation of the load balancer node itself * feat(loadbalancer): add service node implements code for running a service node * feat(loadbalancer): integrate into boost and booster-bitswap * Update loadbalancer/loadbalancer.go Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/servicenode.go Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/servicenode.go Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/messages/messages.ipldsch Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/messages/messages.ipldsch Co-authored-by: Rod Vagg <rod@vagg.org> * refactor(loadbalancer): remove routing protocol remove the routing protocol, instead relying on a set config. also remove forwarding response for inbound requests * fix(loadbalancer): update tests * refactor(loadbalancer): integrate simplified load balancer removed pub keys to minimize network traffic, added api's to configure and update bitswap peer id, added auto config of bitswap peer id in booster-bitswap * docs(gen): regenerate api docs * chore(lint): fix lint errors * fix(loadbalancer): minor bridgestream fix * Update loadbalancer/servicenode.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * refactor(protocolproxy): address PR comments renames, reconfigured architecture, etc * refactor(make init print out peer id): remove apis and transparent peer id setting. have init print Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: dirkmc <dirkmdev@gmail.com> * Add block filter via BadBits (#825) * feat(booster-bitswap): add block filter via BadBits * refactor(booster-bitswap): use bitswap blockfilter for filtering * feat(blockfilter): only update when list is modified * feat(blockFilter): add on disk caching * Update cmd/booster-bitswap/blockfilter/blockfilter.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * fix(blockfilter): minor PR fixups Co-authored-by: dirkmc <dirkmdev@gmail.com> * Libp2p 0.22 upgrade (#837) * chore(deps): upgrade to Lotus RC & libp2p v0.22 * chore(deps): update go to 1.18 * ci(circle): update circle to go 1.18 * style(imports): fix imports * fix(build): update ffi * fix(lint): fix deprecated strings.Title method * fix(mod): mod tidy * Protocol Proxy cleanup (#836) * refactor(booster-bitswap): minor UI fixes for booster-bitswap UI * Update cmd/booster-bitswap/init.go Co-authored-by: dirkmc <dirkmdev@gmail.com> Co-authored-by: dirkmc <dirkmdev@gmail.com> * feat: update to dagstore v0.5.5 (#849) * add booster-bitswap to devnet (#866) * bump lotus-test version * add docker/booster-bitswap target in Makefile Co-authored-by: Hannah Howard <hannah@hannahhoward.net> Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org>
* booster bitswap MVP executable (#707) * feat(booster-bitswap): booster bitswap MVP untested * refactor(booster-bitswap): use API for fetching blocks * fix(deps): update deps to compile * feat(booster-bitswap): makefile & fixes add commands to build booster-bitswap, and very a round tripped successful fetch from booster-bitswap * refactor: clean up unused vars etc * fix: booster-bitsawp - check error when creating libp2p key * refactor(node): avoid FreeAndUnsealed method Co-authored-by: Dirk McCormick <dirkmdev@gmail.com> Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> * booster-bitswap devnet and tracing (#796) * return ipld ErrNotFound from remote blockstore interface (#798) * fix: return ipld ErrNotFound from remote blockstore interface * test: add more tests for ipld ErrNotFound * test: comment out part of TestDummydealOnline that is flaky due to a bug in latest lotus (#802) * fix normaliseError nil ptr dereference (#803) * feat: shard selector (#807) * LoadBalancer for bitswap (and later, more of libp2p) (#786) * feat(loadbalancer): add message types * feat(messages): add utility functions * feat(loadbalancer): initial load balancer impl implementation of the load balancer node itself * feat(loadbalancer): add service node implements code for running a service node * feat(loadbalancer): integrate into boost and booster-bitswap * Update loadbalancer/loadbalancer.go Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/servicenode.go Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/servicenode.go Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/messages/messages.ipldsch Co-authored-by: Rod Vagg <rod@vagg.org> * Update loadbalancer/messages/messages.ipldsch Co-authored-by: Rod Vagg <rod@vagg.org> * refactor(loadbalancer): remove routing protocol remove the routing protocol, instead relying on a set config. also remove forwarding response for inbound requests * fix(loadbalancer): update tests * refactor(loadbalancer): integrate simplified load balancer removed pub keys to minimize network traffic, added api's to configure and update bitswap peer id, added auto config of bitswap peer id in booster-bitswap * docs(gen): regenerate api docs * chore(lint): fix lint errors * fix(loadbalancer): minor bridgestream fix * Update loadbalancer/servicenode.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * refactor(protocolproxy): address PR comments renames, reconfigured architecture, etc * refactor(make init print out peer id): remove apis and transparent peer id setting. have init print Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: dirkmc <dirkmdev@gmail.com> * Add block filter via BadBits (#825) * feat(booster-bitswap): add block filter via BadBits * refactor(booster-bitswap): use bitswap blockfilter for filtering * feat(blockfilter): only update when list is modified * feat(blockFilter): add on disk caching * Update cmd/booster-bitswap/blockfilter/blockfilter.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * fix(blockfilter): minor PR fixups Co-authored-by: dirkmc <dirkmdev@gmail.com> * Libp2p 0.22 upgrade (#837) * chore(deps): upgrade to Lotus RC & libp2p v0.22 * chore(deps): update go to 1.18 * ci(circle): update circle to go 1.18 * style(imports): fix imports * fix(build): update ffi * fix(lint): fix deprecated strings.Title method * fix(mod): mod tidy * Protocol Proxy cleanup (#836) * refactor(booster-bitswap): minor UI fixes for booster-bitswap UI * Update cmd/booster-bitswap/init.go Co-authored-by: dirkmc <dirkmdev@gmail.com> Co-authored-by: dirkmc <dirkmdev@gmail.com> * feat: update to dagstore v0.5.5 (#849) * feat: bitswap client * feat: bitswap client - output car file * refactor: bitswap client - remove tracing * feat: debug logs * fix: write blocks to blockstore * fix: duration output * fix: duration output for block received * feat: add pprof to bitswap client * feat: protocol proxy logging * feat: bitswap client - check host supports bitswap protocol * feat: listen for bitswap requests locally as well as through forwarding protocol Co-authored-by: Hannah Howard <hannah@hannahhoward.net> Co-authored-by: Anton Evangelatov <anton.evangelatov@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org>
Goals
This is basically a "mashup" of booster-http, #377, and filecoin-project/dagstore#116 to kick the tires and see how to wire everything up for an independent boost executable.
Implementation
TODO
ErrNotFound
fromGetSize()
if the block is not found, because Bitswap checks for this error to decide if a block is presentNote that the RemoteBlockstore calls across an RPC boundary, so the code will need to do string matching to check if the error message contains "not found"
blockstore.Has
andblockstore.GetSize
are efficientMerge Note
This PR is using the base branch of release/lotus1.17.2, so that we can use that for landing all upcoming changes that require the a libp2p update that is landing in Lotus 1.17.2 (early October). This is a workaround until we get better separation of the dependency chain with Lotus.
For discussion
Key issues identified: