Add Collaborative IPFS Cluster Tooling For Data Replication & Downloader Bug Fix #286
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bonedaddy
changed the title
bonedaddy
changed the title
bonedaddy
changed the title
bonedaddy
changed the title
|
I'm putting this back into draft mode as I need to update the downloader to use |
added 9 commits
June 6, 2020 23:21
|
Closing this PR. I would recommend moving this tool to its own repo. We could even have that repo under the 2020pb org if you like @bonedaddy |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Follow up to #222, and enables closing #265 and #265 by creating a public "IPFS Follow Cluster" that anyone can participate in and replicate the set of media linked to by the incident reports in
data_build. Additionally this includes a bug fix which prevented the video downloader from working at all. Closes #306. Addtionally a new feature was added that avoids re-downloading any media previously downloadedIt provides two methods of operation:
Preliminary public cluster is online that can be joined with:
$> ipfs-cluster-follow 2020pb-dataset run --init 2020pb.temporal.cloudA corresponding wiki page has been created
TODO
Spin up a public cluster clusterUpdate configurations (copy & pasted them from a different follow cluster i run)Finish readmeFinish webhook server to auto-updated the cluster when new data is added (i'll do this in a different PR, if you want to follow development of that see hereWarning
As part of this PR I'll spin up a public cluster that enables anyone to start mirroring the data. If you value your anonymity and privacy, do not participate in any public cluster belonging to this dataset. IPFS is supremely self-doxxing, and if you participate in the cluster anyone interested in finding out who you are will be able to do so. Even if you come from behind something like Tor and a VPN, IPFS can still dox you.