Skip to content
This repository has been archived by the owner. It is now read-only.

Alpine-linux packages #83

Open
victorb opened this issue Jan 11, 2017 · 3 comments
Open

Alpine-linux packages #83

victorb opened this issue Jan 11, 2017 · 3 comments

Comments

@victorb
Copy link
Contributor

@victorb victorb commented Jan 11, 2017

Made an effort around half year ago to mirror all the alpine-linux packages, which I completely forgot about but would be worth continuing if someone would like to pick it up.

It mirrors and adds all the packages for alpine-linux 3.4 and 3.5. Each one is about 30GB big.

Hash for 3.4: QmRsvEpJggeu4HhoafzRFobV4sbwVVTXMrdb2p8XWv7bCS
Hash for 3.5: QmQeE2YmmmWakwXs42NpcEzTYtEWYXDHjrZ1U5CfhZzjYq
Hash for edge: QmQ3HQWa3yySXstXLbRZmaRPj9G3KG19voBXvqpNCCgCvW
Full mirror: QmWMpm2dmzSfVphzZGJBAYfSPZha8S3skkJeLEzbJ1BGRj

Code and steps to reproduce: https://github.com/VictorBjelkholm/alpine-mirror
Pinned on: Pollux

@MarkusTeufelberger
Copy link

@MarkusTeufelberger MarkusTeufelberger commented Jan 25, 2017

Sounds really cool, have you considered using IPNS and a mirroring machine to get periodical updates?

@victorb
Copy link
Contributor Author

@victorb victorb commented Feb 2, 2017

@MarkusTeufelberger right now it's fully pinned on one of our hosts, pollux and right now there is no IPNS publishing the latest hash. That would make sense for the edge mirror to update continuously. Added a issue in the alpine-mirror repository: victorb/alpine-mirror#1

@RubenKelevra
Copy link

@RubenKelevra RubenKelevra commented Feb 20, 2020

@victorb If you're still interested in this and running updates: I've set up a collab cluster for ArchLinux-packages a month ago.

It's basically an rsync to ipfs and ipfs cluster script, which also maintains snapshots after every sync with changes. The cluster will automatically unpin snapshots older than 2 month, but users can obviously pin and provide older snapshots if they like.

The listing of the snapshots (as a simple html page) will never be purged, so old CIDs can be found, even after the cluster unpinned and garbage collected them.

The whole script is somewhat lengthy, since I want to pin every single package file instead of pinning full directories recursively. This had the advantage of sharding (if you just want to hold 10 copies of each file in a cluster) and you can set different pin expire dates for files and folders.

The script is also very efficient, by just modifying the least amount of data with ipfs commands corresponding to every file add/change/delete rsync has fetched.

So it's not tediously adding recursively all files to IPFS and let it figure out the changes and redundancies.

Note that it's currently not supported to run multiple Linux distributions with this script, since it would wipe the folders on the second --force-full-add while the directory structure is meant to be able to handle multiple architectures, distributions, repositories.

This limitations is probably going away soon, when I'm looking into supporting different pacman based distributions.

But your usage scenario look probably very different anyway :)

https://github.com/RubenKelevra/pacman.store/blob/master/toolset/repo2cluster.sh

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.