Update to use storethehash with multi-file primary with GC #816
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #816 +/- ##
==========================================
+ Coverage 46.05% 46.94% +0.89%
==========================================
Files 92 92
Lines 7780 7854 +74
==========================================
+ Hits 3583 3687 +104
+ Misses 3734 3711 -23
+ Partials 463 456 -7
|
|
Do not Merge: being tested in dev first |
masih
added a commit
that referenced
this pull request
Sep 15, 2022
masih
added a commit
that referenced
this pull request
Sep 15, 2022
Since the change in GC will rescan the entire index, deploy it onto `dev` first to help de-risk the rollout in case there is some edge case we have missed when it comes to potentially corrupt records.
gammazero
force-pushed
the
primary-gc
branch
from
76b8820
to
fc4984e
Compare
willscott
reviewed
Sep 24, 2022
|
Suggested version: Changes in diff --git a/go.mod b/go.mod
index c6144c8..70af59a 100644
--- a/go.mod
+++ b/go.mod
@@ -7,8 +7,8 @@ require (
github.com/cockroachdb/pebble v0.0.0-20220726144858-a78491c0086f
github.com/filecoin-project/go-address v0.0.5
github.com/filecoin-project/go-dagaggregator-unixfs v0.2.0
- github.com/filecoin-project/go-indexer-core v0.6.2
- github.com/filecoin-project/go-legs v0.4.12
+ github.com/filecoin-project/go-indexer-core v0.6.6
+ github.com/filecoin-project/go-legs v0.4.14
github.com/frankban/quicktest v1.14.3
github.com/gammazero/deque v0.2.0
github.com/gogo/protobuf v1.3.2
@@ -17,13 +17,13 @@ require (
github.com/hashicorp/go-retryablehttp v0.7.1
github.com/ipfs/go-cid v0.2.0
github.com/ipfs/go-datastore v0.5.1
- github.com/ipfs/go-delegated-routing v0.3.0
+ github.com/ipfs/go-delegated-routing v0.5.1
github.com/ipfs/go-ds-leveldb v0.5.0
github.com/ipfs/go-log/v2 v2.5.1
github.com/ipfs/kubo v0.14.0
github.com/ipld/go-ipld-adl-hamt v0.0.0-20220616142416-9004dbd839e0
- github.com/ipld/go-ipld-prime v0.17.0
- github.com/ipld/go-storethehash v0.2.8
+ github.com/ipld/go-ipld-prime v0.18.0
+ github.com/ipld/go-storethehash v0.3.4
github.com/libp2p/go-libp2p v0.21.0
github.com/libp2p/go-libp2p-core v0.19.1
github.com/libp2p/go-msgio v0.2.0
@@ -118,7 +118,7 @@ require (
github.com/ipfs/go-peertaskqueue v0.7.1 // indirect
github.com/ipfs/go-unixfs v0.4.0 // indirect
github.com/ipfs/go-verifcid v0.0.1 // indirect
- github.com/ipld/edelweiss v0.1.4 // indirect
+ github.com/ipld/edelweiss v0.2.0 // indirect
github.com/ipld/go-codec-dagpb v1.4.0 // indirect
github.com/jackpal/go-nat-pmp v1.0.2 // indirect
github.com/jbenet/go-temp-err-catcher v0.1.0 // indirect
|
|
I am going to merge this PR as it has become clear that it is necessary if we are going to use STH. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Newer storethehash splits primary storage across multiple files, and has garbage collections for the primary storage.
Conversion from the old to the new primary is done automatically when the new indexer starts up. This process will take significant time as the entire index needs to be updated with the new locations of primary data. Progress is logged.