Feature Request: Add support for sharing Snapshots with other ArchiveBox instances (to enable distributed / federated archiving)

[pirate]

Next big thing I'm thinking about for BA is turning it into a distributed way-back machine! Everyone's personal archives can still be kept separate, but as part of the archive process you're prompted if you want to share the pages you've archived with a federated public archive. Each archived url then gets a deterministic "federated id" which other people will be able to use to find all archived versions of a specific url.

So when I visit my personal archive and see "example.com/blog/123.html" I can click a "show all versions" button which shows an archive in 2010 by alice, one in 2013 by bob, and one in 2017 by frank. I can click on links to view each of their versions in case mine is bad or corrupted somehow.

On the search page you'll be able to search for any url (like the wayback machine), and if it's not in your personal archive it'll show results from other people's archives.

[pirate]

In theory, with enough people running BA we could archive a significant portion of Soundcloud before they go out of business. 😁

[pirate]

Step 1: merkel tree for identifying and querying archive blobs across a distributed system: https://gist.github.com/pirate/0a3545254615b985727b49bc5c3d99cf

[pirate]

I like what ZeroNet is doing in this space: https://github.com/HelloZeroNet/ZeroNet

[pirate]

Blocked by: #74

Once we have a good unique UUID/ULID ID scheme for Snapshots we can begin thinking about how to broadcast that with some metadata to other ArchiveBox instances / endpoints.

Planned baby steps towards this goal in the far-far-future:

1. Finalize ArchiveBox Add REST API endpoint to allow other services to POST new URLs and snapshots to ArchiveBox
2. Add functionality for ArchiveBox to announce new snapshots to the world via RSS/webhooks/realtime endpoint of some kind:
   • rest webhook support: i.e. add the ability to configure ArchiveBox to ping outside endpoints whenever a new Snapshot/ArchiveResult is created
   • RSS feed support, i.e. publish an RSS feed on the ArchiveBox server of all recent snapshots (like Pocket does for your pocket bookmarks)
3. Add native ArchiveBox UI support for searching some of these global federation mechanisms on your own instance so that you can browse snapshots from other instances and providers without leaving your one unified UI

External tools could then be developed that injest this feed to publish archivebox content on other platforms, e.g.:

• archivebox RSS -> proof-of-history blockchain e.g. Solana
• archivebox RSS -> bittorrent's magnet DHT and tracker sites
• archivebox RSS -> IFTTT/zapier/slack/zulip/etc. webhooks

Then later we can add functionality for ArchiveBox to publish snapshots/metadata to global lookup systems like proof-of-history blockchains (e.g. Solana), DHT's like bittorrent's magent system uses, distributed filesystems like IPFS, etc.