New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ipfs and pacman #84

Open
jbenet opened this Issue Dec 7, 2015 · 79 comments

Comments

Projects
None yet
@jbenet
Member

jbenet commented Dec 7, 2015

notes integrating with pacman here

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat commented Dec 7, 2015

Relevant thread on Arch Linux forums: https://bbs.archlinux.org/viewtopic.php?id=203853

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 7, 2015

Member

@robcat thanks! let us know how we can help. we'd love to contribute to making this easy + nice. cc @whyrusleeping

Member

jbenet commented Dec 7, 2015

@robcat thanks! let us know how we can help. we'd love to contribute to making this easy + nice. cc @whyrusleeping

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 7, 2015

(adding @anatol, Arch Linux developer that has shown interest on the forums)

So, in general ipfs can help in many ways the package distribution (distributed storage, versioning, package signing).

The low hanging fruit is the distributed storage: given a "package entry" in the pacman database, get directly the package from ipfs by hash.
A custom downloader can be already plugged in pacman using the XferCommand configuration variable (some examples)

The big problem is the following: the pacman database includes only SHA256 and MD5 hashes, and no ipfs-style multihashes (that unfortunately cannot be constructed using the already included hashes).

My plan is to build a custom XferCommand script that, querying some kind of service, translates the SHA256 hash into a "standard" ipfs hash, and then does ipfs add -o <cache-path> <ipfs-hash>

This hash translation service can be centralized in this initial stage, but it adds a constant lag to each package download (the next step should be to build and distribute a package database with ipfs-multihashes).

I'm still in search of a more elegant solution that doesn't require a central translation service, ideas are welcome

robcat commented Dec 7, 2015

(adding @anatol, Arch Linux developer that has shown interest on the forums)

So, in general ipfs can help in many ways the package distribution (distributed storage, versioning, package signing).

The low hanging fruit is the distributed storage: given a "package entry" in the pacman database, get directly the package from ipfs by hash.
A custom downloader can be already plugged in pacman using the XferCommand configuration variable (some examples)

The big problem is the following: the pacman database includes only SHA256 and MD5 hashes, and no ipfs-style multihashes (that unfortunately cannot be constructed using the already included hashes).

My plan is to build a custom XferCommand script that, querying some kind of service, translates the SHA256 hash into a "standard" ipfs hash, and then does ipfs add -o <cache-path> <ipfs-hash>

This hash translation service can be centralized in this initial stage, but it adds a constant lag to each package download (the next step should be to build and distribute a package database with ipfs-multihashes).

I'm still in search of a more elegant solution that doesn't require a central translation service, ideas are welcome

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 7, 2015

A possible solution could be creating a dag with all packages: For example having the repo mirror servers add everything to ipfs and provide the hash of the folder, then deduplication takes care of efficiency and the pacman client only needs that one hash of the latest version of all the packages and the .db repository files 👍

For this to work smoothly we would probably need to patch pacman. Or maybe a tool that wraps pacman but uses IPFS to get the data could be built.

fazo96 commented Dec 7, 2015

A possible solution could be creating a dag with all packages: For example having the repo mirror servers add everything to ipfs and provide the hash of the folder, then deduplication takes care of efficiency and the pacman client only needs that one hash of the latest version of all the packages and the .db repository files 👍

For this to work smoothly we would probably need to patch pacman. Or maybe a tool that wraps pacman but uses IPFS to get the data could be built.

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 7, 2015

@fazo96 About deduplication: I downloaded the pool of archlinux packages and tried both the chunking algorithms provided by ipfs (fixed blocks and rabin). Apparently there is no detectable deduplication in my case in either case. Can you suggest an effective chunking strategy for xz compressed packages?

About the dag of packages: cool idea, but it requires to manage a fat central mirror that regularly rsyncs and compiles a new dag at every update. Unfortunately I don't have such a server available (but maybe an existing Arch mirror could be interested in running the ipfs daemon?).

robcat commented Dec 7, 2015

@fazo96 About deduplication: I downloaded the pool of archlinux packages and tried both the chunking algorithms provided by ipfs (fixed blocks and rabin). Apparently there is no detectable deduplication in my case in either case. Can you suggest an effective chunking strategy for xz compressed packages?

About the dag of packages: cool idea, but it requires to manage a fat central mirror that regularly rsyncs and compiles a new dag at every update. Unfortunately I don't have such a server available (but maybe an existing Arch mirror could be interested in running the ipfs daemon?).

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 7, 2015

@robcat having an existing mirror run an ipfs daemon would be ideal: by mounting its IPNS publication using fuse, it could store the data directly in IPFS while also being able to read it from the file system and when it updates the files it would automagically update what's on IPFS.

About deduplication, there's probably no trivial way to apply it inside packages but at least two copies of the same package will produce the same hash and thus will become the same file in IPFS, so that the pacman client can get a package from any IPFS node that has a copy without having any configuration telling it the nodes' ip addresses.

This way choosing the best mirror, ranking them etc will not be needed: ipfs will be in charge of downloading the package from the best location. This will mean that updating even when an Internet connection is not there could be implemented if there's a reachable computer that is serving the packages and .db files.

fazo96 commented Dec 7, 2015

@robcat having an existing mirror run an ipfs daemon would be ideal: by mounting its IPNS publication using fuse, it could store the data directly in IPFS while also being able to read it from the file system and when it updates the files it would automagically update what's on IPFS.

About deduplication, there's probably no trivial way to apply it inside packages but at least two copies of the same package will produce the same hash and thus will become the same file in IPFS, so that the pacman client can get a package from any IPFS node that has a copy without having any configuration telling it the nodes' ip addresses.

This way choosing the best mirror, ranking them etc will not be needed: ipfs will be in charge of downloading the package from the best location. This will mean that updating even when an Internet connection is not there could be implemented if there's a reachable computer that is serving the packages and .db files.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 10, 2015

Member

@fazo96 About deduplication: I downloaded the pool of archlinux packages and tried both the chunking algorithms provided by ipfs (fixed blocks and rabin). Apparently there is no detectable deduplication in my case in either case. Can you suggest an effective chunking strategy for xz compressed packages?

Are they tarballs? Did you try ipfs tar? it imports tars smartly.

@whyrusleeping what is the tar branch that detects tarballs on add?

Member

jbenet commented Dec 10, 2015

@fazo96 About deduplication: I downloaded the pool of archlinux packages and tried both the chunking algorithms provided by ipfs (fixed blocks and rabin). Apparently there is no detectable deduplication in my case in either case. Can you suggest an effective chunking strategy for xz compressed packages?

Are they tarballs? Did you try ipfs tar? it imports tars smartly.

@whyrusleeping what is the tar branch that detects tarballs on add?

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 10, 2015

@jbenet

Did you try ipfs tar? it imports tars smartly.

Cool feature! I didn't know about that. (at the moment it doesn't seem to work for some tars, I'm opening a separate issue)

In the Arch Linux case it's unfortunately not very useful, since the packages are signed only after compression. Distributing the packages in uncompressed form would mean:

  • having to recompress the package at installation time to verify the signature (at a great CPU cost)
  • guessing which compression settings were used by the developers on their machines, in order to reproduce the same exact tar.xz file

robcat commented Dec 10, 2015

@jbenet

Did you try ipfs tar? it imports tars smartly.

Cool feature! I didn't know about that. (at the moment it doesn't seem to work for some tars, I'm opening a separate issue)

In the Arch Linux case it's unfortunately not very useful, since the packages are signed only after compression. Distributing the packages in uncompressed form would mean:

  • having to recompress the package at installation time to verify the signature (at a great CPU cost)
  • guessing which compression settings were used by the developers on their machines, in order to reproduce the same exact tar.xz file
@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 10, 2015

Well, packages aren't that heavy. I mean, the biggest package I ever downloaded is probably netbeans which is less than 300 MB, and it's wayyyyyy bigger than the average package which is like a few megabytes (quanitity pulled out of thin air).

I don't think there are many packages that share common data except multiple versions of the same package. It would be very nice to figure out how to take advantage of deduplication with the current way pacman packages stuff, but even without that, an IPFS transport for pacman would be a huge step forward in terms of efficiency and repository management.

Just having the official repository publish the hash of the latest version of the packages folder and all the mirrors just automatically pinning that when it updates would propagate changes pretty fast and no one would have to worry about choosing the fastest mirror except bitswap developers 👍

To sum it up my humble suggestion is not to try too hard to fix this problem (applying deduplication to arch packages) now. IPFS is still very useful in this use case, even without that, and in the future pacman could consider packaging stuff differently if IPFS becomes the standard for distribution.

fazo96 commented Dec 10, 2015

Well, packages aren't that heavy. I mean, the biggest package I ever downloaded is probably netbeans which is less than 300 MB, and it's wayyyyyy bigger than the average package which is like a few megabytes (quanitity pulled out of thin air).

I don't think there are many packages that share common data except multiple versions of the same package. It would be very nice to figure out how to take advantage of deduplication with the current way pacman packages stuff, but even without that, an IPFS transport for pacman would be a huge step forward in terms of efficiency and repository management.

Just having the official repository publish the hash of the latest version of the packages folder and all the mirrors just automatically pinning that when it updates would propagate changes pretty fast and no one would have to worry about choosing the fastest mirror except bitswap developers 👍

To sum it up my humble suggestion is not to try too hard to fix this problem (applying deduplication to arch packages) now. IPFS is still very useful in this use case, even without that, and in the future pacman could consider packaging stuff differently if IPFS becomes the standard for distribution.

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 10, 2015

Just having the official repository publish the hash of the latest version of the packages folder and all the mirrors just automatically pinning that when it updates would propagate changes pretty fast and no one would have to worry about choosing the fastest mirror except bitswap developers

@fazo96
I hear you, this would be a pretty good solution.

But the problem is, to follow your plan would mean to convince the central official mirror to:

  • hash the whole repository using ipfs
  • manage and run an ipfs node (at least to initially seed the new packages)
  • use twice the storage space (to store the ipfs blocks of the whole repository)
  • sign and publish the ipns periodically (right now there's no central authority, only individual packages are signed)

There is a problem of incentives here: why should the central authority to do all this, if nobody is already using ipfs?

robcat commented Dec 10, 2015

Just having the official repository publish the hash of the latest version of the packages folder and all the mirrors just automatically pinning that when it updates would propagate changes pretty fast and no one would have to worry about choosing the fastest mirror except bitswap developers

@fazo96
I hear you, this would be a pretty good solution.

But the problem is, to follow your plan would mean to convince the central official mirror to:

  • hash the whole repository using ipfs
  • manage and run an ipfs node (at least to initially seed the new packages)
  • use twice the storage space (to store the ipfs blocks of the whole repository)
  • sign and publish the ipns periodically (right now there's no central authority, only individual packages are signed)

There is a problem of incentives here: why should the central authority to do all this, if nobody is already using ipfs?

@pbronez

This comment has been minimized.

Show comment
Hide comment
@pbronez

pbronez Dec 10, 2015

convince the central official mirror

@robcat shouldn't any mirror work, as long as IPFS users trust it? Anyone could set up a mirror to serve as the bridge to IPFS. Then once people start using it, down the line the official repo could take over that function.

Setting up a new mirror to copy packages into IPFS is closely related to the archival efforts discussed at ipfs/archives#5

pbronez commented Dec 10, 2015

convince the central official mirror

@robcat shouldn't any mirror work, as long as IPFS users trust it? Anyone could set up a mirror to serve as the bridge to IPFS. Then once people start using it, down the line the official repo could take over that function.

Setting up a new mirror to copy packages into IPFS is closely related to the archival efforts discussed at ipfs/archives#5

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 10, 2015

manage and run an ipfs node (at least to initially seed the new packages)

It's not hard at all to do, there are also docker images.

use twice the storage space (to store the ipfs blocks of the whole repository)

You can mount IPNS with FUSE so that you don't need twice the space

sign and publish the ipns periodically (right now there's no central authority, only individual packages are signed)

Well, if the packages are all in one folder, and that folder is inside the IPNS mountpoint (using FUSE), when a file changes everything gets republished. It's not the best of solution but it's worth a shot (not by the official mirror maintainers of course, but by some interest third party)

@robcat shouldn't any mirror work, as long as IPFS users trust it? Anyone could set up a mirror to serve as the bridge to IPFS. Then once people start using it, down the line the official repo could take over that function.

Yes, that's the point! 👍 I don't see why a random guy can't set up an IPFS mirror (except the actual hassle of downloading every package). Also no need to trust IPFS or the mirror since the packages are signed, if my understanding is correct.

If somebody has the resources needed to set up an IPFS mirror of the arch repositories and some people manage to start using it, it will be a lot easier to get more adoption. I could do it, but I only have 1 Mbit upload bandwidth, for four people and at least 6 devices...

fazo96 commented Dec 10, 2015

manage and run an ipfs node (at least to initially seed the new packages)

It's not hard at all to do, there are also docker images.

use twice the storage space (to store the ipfs blocks of the whole repository)

You can mount IPNS with FUSE so that you don't need twice the space

sign and publish the ipns periodically (right now there's no central authority, only individual packages are signed)

Well, if the packages are all in one folder, and that folder is inside the IPNS mountpoint (using FUSE), when a file changes everything gets republished. It's not the best of solution but it's worth a shot (not by the official mirror maintainers of course, but by some interest third party)

@robcat shouldn't any mirror work, as long as IPFS users trust it? Anyone could set up a mirror to serve as the bridge to IPFS. Then once people start using it, down the line the official repo could take over that function.

Yes, that's the point! 👍 I don't see why a random guy can't set up an IPFS mirror (except the actual hassle of downloading every package). Also no need to trust IPFS or the mirror since the packages are signed, if my understanding is correct.

If somebody has the resources needed to set up an IPFS mirror of the arch repositories and some people manage to start using it, it will be a lot easier to get more adoption. I could do it, but I only have 1 Mbit upload bandwidth, for four people and at least 6 devices...

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 10, 2015

I could do it, but I only have 1 Mbit upload bandwidth, for four people and at least 6 devices...

Ok, so what about this three pronged approach:

  • @fazo96 publishes the ipns entry (minimal bandwidth required)
  • I keep a non-authoritative mirror with a lot of bandwidth that will sync the most requested packages
  • users will put https://ipfs.io/ipns/<faso96-hash>/ at the top of their mirrorlist

My node is already up at 178.62.202.191 and it independently adds all the x86_64 packages from core, extra, multilib, testing and multilib-testing (it syncs to an official arch mirror and adds the packages to ipfs).

@fazo96 you can go ahead and publish the ipns entry, if IPFS works the bulk of the requests will not even hit your node :)

robcat commented Dec 10, 2015

I could do it, but I only have 1 Mbit upload bandwidth, for four people and at least 6 devices...

Ok, so what about this three pronged approach:

  • @fazo96 publishes the ipns entry (minimal bandwidth required)
  • I keep a non-authoritative mirror with a lot of bandwidth that will sync the most requested packages
  • users will put https://ipfs.io/ipns/<faso96-hash>/ at the top of their mirrorlist

My node is already up at 178.62.202.191 and it independently adds all the x86_64 packages from core, extra, multilib, testing and multilib-testing (it syncs to an official arch mirror and adds the packages to ipfs).

@fazo96 you can go ahead and publish the ipns entry, if IPFS works the bulk of the requests will not even hit your node :)

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 11, 2015

@robcat I'm very interested in trying this out! :)

However:

  • I need the hash to be able to publish something
  • You can publish it yourself if you want, or you can mount ipns with FUSE and drop the root of the mirror in <mountpoint>/local so that they are automatically published on your node's IPNS (this is the suggested approach) while also being accessible on the filesystem thus not taking up twice the space! Try publishing the hash of the repo before doing this so that the files will already be there
  • If you want to quickly publish to ipns, cd to the root of the repo then run ipfs name publish $(ipfs add -r -q -w . | tail -n1). It will publish the folder you are in and tell you the IPNS name and the hash. You can then tell us the IPNS name or the IPFS hash so we can try 👍
  • users shouldn't use ipfs.io as the gateway or all the traffic will have to come from ipfs.io to the users and it doesn't make much sense: they would have to use their local gateway (localhost)
  • I have quite some credit on digital ocean thanks to github's student developer pack, and I'm willing to set up a mirror for pacman using IPFS 👍
  • I don't know if there's a way to disable the pacman cache, but I hope there is because using this approach would make it a waste of disk space except for downgrading.

Also, another use case for this is local package sharing: I have two computers in LAN and a shitty bandwidth to the internet, and all available solutions to share packages in LAN are pretty ugly compared to this.

If you want we can meet on freenode (I'm Fazo) later today or some other time to try this out 👍 If we succeed (we probably will) we can set up a page on the arch wiki or a github repo that explains how to use IPFS with pacman this way (without any external tools except ipfs)

fazo96 commented Dec 11, 2015

@robcat I'm very interested in trying this out! :)

However:

  • I need the hash to be able to publish something
  • You can publish it yourself if you want, or you can mount ipns with FUSE and drop the root of the mirror in <mountpoint>/local so that they are automatically published on your node's IPNS (this is the suggested approach) while also being accessible on the filesystem thus not taking up twice the space! Try publishing the hash of the repo before doing this so that the files will already be there
  • If you want to quickly publish to ipns, cd to the root of the repo then run ipfs name publish $(ipfs add -r -q -w . | tail -n1). It will publish the folder you are in and tell you the IPNS name and the hash. You can then tell us the IPNS name or the IPFS hash so we can try 👍
  • users shouldn't use ipfs.io as the gateway or all the traffic will have to come from ipfs.io to the users and it doesn't make much sense: they would have to use their local gateway (localhost)
  • I have quite some credit on digital ocean thanks to github's student developer pack, and I'm willing to set up a mirror for pacman using IPFS 👍
  • I don't know if there's a way to disable the pacman cache, but I hope there is because using this approach would make it a waste of disk space except for downgrading.

Also, another use case for this is local package sharing: I have two computers in LAN and a shitty bandwidth to the internet, and all available solutions to share packages in LAN are pretty ugly compared to this.

If you want we can meet on freenode (I'm Fazo) later today or some other time to try this out 👍 If we succeed (we probably will) we can set up a page on the arch wiki or a github repo that explains how to use IPFS with pacman this way (without any external tools except ipfs)

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 11, 2015

Member

If you have troubles with IPNS bug me directly. we'll improve the perf to fix it for you.

Member

jbenet commented Dec 11, 2015

If you have troubles with IPNS bug me directly. we'll improve the perf to fix it for you.

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 11, 2015

@jbenet thanks a lot, but I think with the 0.3.10 caching we'll be fine 👍

It would be cool if records would be kept for a long time if nothing more recent is available, not sure if that's already in the codebase.

fazo96 commented Dec 11, 2015

@jbenet thanks a lot, but I think with the 0.3.10 caching we'll be fine 👍

It would be cool if records would be kept for a long time if nothing more recent is available, not sure if that's already in the codebase.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 11, 2015

Member

@fazo96 right now records expire. i do hear you though, most IPNS users today would prefer non-expiring records. my crypto paranoia could be deferred.

Member

jbenet commented Dec 11, 2015

@fazo96 right now records expire. i do hear you though, most IPNS users today would prefer non-expiring records. my crypto paranoia could be deferred.

@anatol

This comment has been minimized.

Show comment
Hide comment
@anatol

anatol Dec 11, 2015

Also, another use case for this is local package sharing: I have two computers in LAN and a shitty bandwidth to the internet

@fazo96 A bit of out-of-topic. I was looking for a simple Arch package sharing tool for LAN and did not find what matches my expectations. So I wrote my own tool - https://github.com/anatol/pacoloco an Arch package proxy repo. It is not published anywhere and just a bunch of code without much documentation. But it works well for several months for me. Advantages of this tool - pure C, no external dependencies except glibc, run perfectly at OpenWRT MIPS router, single-threaded, event-loop based architecture, HTTP pipeline support to reduce request latency.

anatol commented Dec 11, 2015

Also, another use case for this is local package sharing: I have two computers in LAN and a shitty bandwidth to the internet

@fazo96 A bit of out-of-topic. I was looking for a simple Arch package sharing tool for LAN and did not find what matches my expectations. So I wrote my own tool - https://github.com/anatol/pacoloco an Arch package proxy repo. It is not published anywhere and just a bunch of code without much documentation. But it works well for several months for me. Advantages of this tool - pure C, no external dependencies except glibc, run perfectly at OpenWRT MIPS router, single-threaded, event-loop based architecture, HTTP pipeline support to reduce request latency.

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 11, 2015

@anatol Cool, if you could set up some documentation on how to use it and maybe even a PKGBUILD to the AUR, I'd use it.

fazo96 commented Dec 11, 2015

@anatol Cool, if you could set up some documentation on how to use it and maybe even a PKGBUILD to the AUR, I'd use it.

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 11, 2015

Ok, time for some results!

me and @robcat did it. He set up an arch linux mirror on a VPS then we got it to store files inside the FUSE mountpoint (/ipns/local) so that they are always available via IPNS.

Using IPFS to download arch packages

The node that is serving them is QmPY3tsmwoCjePd9SEudAGX2bZU65SwiL8KBKaSRKyUdzt. If you want to try it, you can put Server = http://localhost:8080/ipns/QmPY3tsmwoCjePd9SEudAGX2bZU65SwiL8KBKaSRKyUdzt/$repo/os/$arch on top of your pacman mirrorlist. Make sure your daemon is running, then it will download packages from IPFS when possible! It's that simple.

It has a few hiccups, for example the timeout is too low and if it's your first time downloading package it's going to fall back to boring centralized mirrors. Also the mirror we have is incomplete due to the resources needed to hash tens of gigabytes of files.

But it works! Keep in mind that the IPFS mirror mentioned was just used for testing, we are not guaranteeing anything.

I achieved the best result by using this XferCommand: XferCommand = /usr/bin/wget --quiet --show-progress --timeout=180 -O %o %u

This gives IPFS the time to answer (long timeout) and tells wget not to clutter the output and display a nice progress bar.

Setting up a mirror

Just follow the steps to set up a regular arch mirror, but mount IPNS to /ipns before and then drop your mirror files in /ipns/local/. That's it, it's that simple. go-ipfs + FUSE works fine on ubuntu 15.10 and Arch Linux.

EDIT: another issue is that signatures for the .db files are missing from the mirror:

09:39:50.913 ERROR core/serve: Path Resolve error: no link named "core.db.sig" under QmbEx32ruv1FLp6dHdsgouKSsg1hdaidnZnwSLA52mvAiu gateway_handler.go:458

fazo96 commented Dec 11, 2015

Ok, time for some results!

me and @robcat did it. He set up an arch linux mirror on a VPS then we got it to store files inside the FUSE mountpoint (/ipns/local) so that they are always available via IPNS.

Using IPFS to download arch packages

The node that is serving them is QmPY3tsmwoCjePd9SEudAGX2bZU65SwiL8KBKaSRKyUdzt. If you want to try it, you can put Server = http://localhost:8080/ipns/QmPY3tsmwoCjePd9SEudAGX2bZU65SwiL8KBKaSRKyUdzt/$repo/os/$arch on top of your pacman mirrorlist. Make sure your daemon is running, then it will download packages from IPFS when possible! It's that simple.

It has a few hiccups, for example the timeout is too low and if it's your first time downloading package it's going to fall back to boring centralized mirrors. Also the mirror we have is incomplete due to the resources needed to hash tens of gigabytes of files.

But it works! Keep in mind that the IPFS mirror mentioned was just used for testing, we are not guaranteeing anything.

I achieved the best result by using this XferCommand: XferCommand = /usr/bin/wget --quiet --show-progress --timeout=180 -O %o %u

This gives IPFS the time to answer (long timeout) and tells wget not to clutter the output and display a nice progress bar.

Setting up a mirror

Just follow the steps to set up a regular arch mirror, but mount IPNS to /ipns before and then drop your mirror files in /ipns/local/. That's it, it's that simple. go-ipfs + FUSE works fine on ubuntu 15.10 and Arch Linux.

EDIT: another issue is that signatures for the .db files are missing from the mirror:

09:39:50.913 ERROR core/serve: Path Resolve error: no link named "core.db.sig" under QmbEx32ruv1FLp6dHdsgouKSsg1hdaidnZnwSLA52mvAiu gateway_handler.go:458
@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 12, 2015

Member

This is awesome! congrats @robcat and @fazo96 ! \o/


Please take a look at @diasdavid's recent registry-mirror work. It has very similar goal (replicate npm), and has been addressing many of the issues.

@diasdavid uses the new mfs (files) API in 0.4.0-dev, to make life much easier. I suspect you can likely use that too programmatically, instead of relying on the /ipns mount. then can publish the hash when you're done (ipfs name publish $hash). ipfs's fuse interface is not the best of things yet (apologies there, if you found UX issues), so definitely give the files api a try.

Also 0.4.0 is way faster than 0.3.x.

Member

jbenet commented Dec 12, 2015

This is awesome! congrats @robcat and @fazo96 ! \o/


Please take a look at @diasdavid's recent registry-mirror work. It has very similar goal (replicate npm), and has been addressing many of the issues.

@diasdavid uses the new mfs (files) API in 0.4.0-dev, to make life much easier. I suspect you can likely use that too programmatically, instead of relying on the /ipns mount. then can publish the hash when you're done (ipfs name publish $hash). ipfs's fuse interface is not the best of things yet (apologies there, if you found UX issues), so definitely give the files api a try.

Also 0.4.0 is way faster than 0.3.x.

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 12, 2015

@jbenet Thanks!!

We know about 0.4.0 being faster and the issues with the mount, but we wanted to make it as easy as possible for arch mirror maintainers to integrate IPFS in the mix, and the IPNS mounts really helps because it makes it very simple to use the regular arch mirror administration tools and practices with IPFS. However, we'll surely consider a better approach and test some more.

fazo96 commented Dec 12, 2015

@jbenet Thanks!!

We know about 0.4.0 being faster and the issues with the mount, but we wanted to make it as easy as possible for arch mirror maintainers to integrate IPFS in the mix, and the IPNS mounts really helps because it makes it very simple to use the regular arch mirror administration tools and practices with IPFS. However, we'll surely consider a better approach and test some more.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 12, 2015

Member

Just follow the steps to set up a regular arch mirror, but mount IPNS to /ipns before and then drop your mirror files in /ipns/local/. That's it, it's that simple. go-ipfs + FUSE works fine on ubuntu 15.10 and Arch Linux.

I have to say, it is absolutely awesome that it's this simple. ❤️ to all the hard work making the abstractions nice. (cc @whyrusleeping)

@robcat @fazo96 -- @whyrusleeping and i have discussed making the files api mountable as well. that interface still needs to be defined. potentially something like this ipfs/go-ipfs#2060 (comment)

Member

jbenet commented Dec 12, 2015

Just follow the steps to set up a regular arch mirror, but mount IPNS to /ipns before and then drop your mirror files in /ipns/local/. That's it, it's that simple. go-ipfs + FUSE works fine on ubuntu 15.10 and Arch Linux.

I have to say, it is absolutely awesome that it's this simple. ❤️ to all the hard work making the abstractions nice. (cc @whyrusleeping)

@robcat @fazo96 -- @whyrusleeping and i have discussed making the files api mountable as well. that interface still needs to be defined. potentially something like this ipfs/go-ipfs#2060 (comment)

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Dec 12, 2015

Member

We know about 0.4.0 being faster and the issues with the mount, but we wanted to make it as easy as possible for arch mirror maintainers to integrate IPFS in the mix, and the IPNS mounts really helps because it makes it very simple to use the regular arch mirror administration tools and practices with IPFS. However, we'll surely consider a better approach and test some more.

That's fantastic. it really is ❤️

@fazo96 we may want to make a simple sh script or Makefile that does all of this

Member

jbenet commented Dec 12, 2015

We know about 0.4.0 being faster and the issues with the mount, but we wanted to make it as easy as possible for arch mirror maintainers to integrate IPFS in the mix, and the IPNS mounts really helps because it makes it very simple to use the regular arch mirror administration tools and practices with IPFS. However, we'll surely consider a better approach and test some more.

That's fantastic. it really is ❤️

@fazo96 we may want to make a simple sh script or Makefile that does all of this

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 12, 2015

We're still figuring out issues with rsync crashing go-ipfs when used with a target inside /ipns/local. If we figure out exactly what is causing the problem, we'll open an issue about it 👍

EDIT: looks like we were using a build from the master branch... made a few modifications to our script and used 0.3.10, now it looks like it doesn't crash anymore. Here's the script, but it's not done at all and still requires some manual intervention.

The biggest issue now is that for some reason rsync can't detect that two files are the same if one of the two is in /ipns/local so it redownloads everything every time

fazo96 commented Dec 12, 2015

We're still figuring out issues with rsync crashing go-ipfs when used with a target inside /ipns/local. If we figure out exactly what is causing the problem, we'll open an issue about it 👍

EDIT: looks like we were using a build from the master branch... made a few modifications to our script and used 0.3.10, now it looks like it doesn't crash anymore. Here's the script, but it's not done at all and still requires some manual intervention.

The biggest issue now is that for some reason rsync can't detect that two files are the same if one of the two is in /ipns/local so it redownloads everything every time

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 14, 2015

Ok, I've been able to sync the "core" repository to /ipns/local using this script, and it works wonderfully on my Arch machine with ipfs 0.3.10:
https://gist.github.com/robcat/3dbbafee096269b6843a

But me and @fazo96 were trying to make it work on a Ubuntu 15.10 VM (same versions of ipfs and fuse), and rsync doesn't detect the unchanged files (it overwrites every file, triggering the re-hash and re-publish by ipfs). This is of course too slow.
Unfortunately this problem is not straightforward to debug (it seems to be a timeout issue in the interaction between rsync and fuse): if anyone want to try, please contact me or @fazo96 in order to get access to the VM.

@jbenet The mountable files api you described can be useful in a lot of situations.
But in the "mirror" use case, we are relying on the auto-publishing feature of /ipns/local. I believe we would have no real advantage in copying the packages in a different folder.

robcat commented Dec 14, 2015

Ok, I've been able to sync the "core" repository to /ipns/local using this script, and it works wonderfully on my Arch machine with ipfs 0.3.10:
https://gist.github.com/robcat/3dbbafee096269b6843a

But me and @fazo96 were trying to make it work on a Ubuntu 15.10 VM (same versions of ipfs and fuse), and rsync doesn't detect the unchanged files (it overwrites every file, triggering the re-hash and re-publish by ipfs). This is of course too slow.
Unfortunately this problem is not straightforward to debug (it seems to be a timeout issue in the interaction between rsync and fuse): if anyone want to try, please contact me or @fazo96 in order to get access to the VM.

@jbenet The mountable files api you described can be useful in a lot of situations.
But in the "mirror" use case, we are relying on the auto-publishing feature of /ipns/local. I believe we would have no real advantage in copying the packages in a different folder.

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Dec 14, 2015

Member

@robcat i think I know the issue. the Atime and Ctime of the files in /ipns arent set. Let me work on a patch for ya

Member

whyrusleeping commented Dec 14, 2015

@robcat i think I know the issue. the Atime and Ctime of the files in /ipns arent set. Let me work on a patch for ya

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Dec 14, 2015

Member

hrmm... after a bit more thought its not going to be easy to fix. I'm working modtimes into the mfs code on 0.4.0 (the code behind the fuse interface). You could try the -c option on rsync as a workaround for now

Member

whyrusleeping commented Dec 14, 2015

hrmm... after a bit more thought its not going to be easy to fix. I'm working modtimes into the mfs code on 0.4.0 (the code behind the fuse interface). You could try the -c option on rsync as a workaround for now

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 14, 2015

@whyrusleeping

hrmm... after a bit more thought its not going to be easy to fix. I'm working modtimes into the mfs code on 0.4.0 (the code behind the fuse interface). You could try the -c option on rsync as a workaround for now

Uhm, I double checked and in fact there is something seriously wrong with my Ubuntu machine.
Rsync doesn't give any errors, but the resulting files accessed via ls have zero sizes (of course the -c option doesn't help):

# ls -l /ipns/local/foo
total 0
-rw-rw-rw- 0 root root 0 Aug 30  1754 acl-2.2.52-2-i686.pkg.tar.xz
-rw-rw-rw- 0 root root 0 Aug 30  1754 acl-2.2.52-2-i686.pkg.tar.xz.sig
[...]

But the file objects are stored correctly by ipfs:

# ipfs/ipfs object links QmcCShzUmGvMjbWyC1jT1fG9y4LFYt7Cxwms7Pm82vg72d
Hash                                           Size   Name                                            
QmaKpUC352G5raBe1YMhKAWEGR4zbHVh4GgWG4sbVuEdFo 133358 acl-2.2.52-2-i686.pkg.tar.xz                    
QmaobnVG1xfHcYEF4DiLQYQFGGyJaZ32rYNUCeAchGqXe5 607    acl-2.2.52-2-i686.pkg.tar.xz.sig    

To ask for a third opinion, I will try soon with osx.

robcat commented Dec 14, 2015

@whyrusleeping

hrmm... after a bit more thought its not going to be easy to fix. I'm working modtimes into the mfs code on 0.4.0 (the code behind the fuse interface). You could try the -c option on rsync as a workaround for now

Uhm, I double checked and in fact there is something seriously wrong with my Ubuntu machine.
Rsync doesn't give any errors, but the resulting files accessed via ls have zero sizes (of course the -c option doesn't help):

# ls -l /ipns/local/foo
total 0
-rw-rw-rw- 0 root root 0 Aug 30  1754 acl-2.2.52-2-i686.pkg.tar.xz
-rw-rw-rw- 0 root root 0 Aug 30  1754 acl-2.2.52-2-i686.pkg.tar.xz.sig
[...]

But the file objects are stored correctly by ipfs:

# ipfs/ipfs object links QmcCShzUmGvMjbWyC1jT1fG9y4LFYt7Cxwms7Pm82vg72d
Hash                                           Size   Name                                            
QmaKpUC352G5raBe1YMhKAWEGR4zbHVh4GgWG4sbVuEdFo 133358 acl-2.2.52-2-i686.pkg.tar.xz                    
QmaobnVG1xfHcYEF4DiLQYQFGGyJaZ32rYNUCeAchGqXe5 607    acl-2.2.52-2-i686.pkg.tar.xz.sig    

To ask for a third opinion, I will try soon with osx.

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Dec 14, 2015

@robcat that's very nice!

I'll give it a try too on my machine if I have the time and network bandwidth. We could try setting up a dockerfile based on the go-ipfs one that also mounts fuse and keeps an updated arch mirror using your script, so that we can run it on any platform. I'm not sure it's possible to mount IPFS with fuse inside a docker container...

Then, only the "root" arch mirror using IPFS needs to actually sync with the other mirrors using rsync. The other IPFS based mirrors will just need to pin the IPNS name of the "root" mirror once pub/sub is implemented. As a temporary workaround, a very small script could poll for IPNS updates and pin the new hash.

fazo96 commented Dec 14, 2015

@robcat that's very nice!

I'll give it a try too on my machine if I have the time and network bandwidth. We could try setting up a dockerfile based on the go-ipfs one that also mounts fuse and keeps an updated arch mirror using your script, so that we can run it on any platform. I'm not sure it's possible to mount IPFS with fuse inside a docker container...

Then, only the "root" arch mirror using IPFS needs to actually sync with the other mirrors using rsync. The other IPFS based mirrors will just need to pin the IPNS name of the "root" mirror once pub/sub is implemented. As a temporary workaround, a very small script could poll for IPNS updates and pin the new hash.

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Dec 14, 2015

Member

@robcat you could try enabling the fuse debug logs, it currently need to be added in and recompiled like so:

diff --git a/fuse/ipns/ipns_unix.go b/fuse/ipns/ipns_unix.go
index bd4b861..5f634d4 100644
--- a/fuse/ipns/ipns_unix.go
+++ b/fuse/ipns/ipns_unix.go
@@ -23,6 +23,12 @@ import (
        ft "github.com/ipfs/go-ipfs/unixfs"
 )

+func init() {
+       fuse.Debug = func(i interface{}) {
+               fmt.Println(i)
+       }
+}
+
 var log = logging.Logger("fuse/ipns")

 // FileSystem is the readwrite IPNS Fuse Filesystem.

but that should give you some more insight

Member

whyrusleeping commented Dec 14, 2015

@robcat you could try enabling the fuse debug logs, it currently need to be added in and recompiled like so:

diff --git a/fuse/ipns/ipns_unix.go b/fuse/ipns/ipns_unix.go
index bd4b861..5f634d4 100644
--- a/fuse/ipns/ipns_unix.go
+++ b/fuse/ipns/ipns_unix.go
@@ -23,6 +23,12 @@ import (
        ft "github.com/ipfs/go-ipfs/unixfs"
 )

+func init() {
+       fuse.Debug = func(i interface{}) {
+               fmt.Println(i)
+       }
+}
+
 var log = logging.Logger("fuse/ipns")

 // FileSystem is the readwrite IPNS Fuse Filesystem.

but that should give you some more insight

@jbenet jbenet referenced this issue Dec 14, 2015

Closed

Sprint Dec 7 #67

@eminence eminence referenced this issue Dec 15, 2015

Closed

Weekly Roundup #73

@atondwal

This comment has been minimized.

Show comment
Hide comment
@atondwal

atondwal Mar 8, 2016

So I tried @robcat's script on my machine and it works for [core], but the bigger packages from [community] (in my case 0ad, it's really big, but it's also alphabetically first) run into ipfs/go-ipfs#1456 and crashes violently, complaining about too many open files.

When it does work, though, it works beautifully. I'm playing with it at /ipns/ani.sh/, but that's my laptop, so don't expect much uptime.

atondwal commented Mar 8, 2016

So I tried @robcat's script on my machine and it works for [core], but the bigger packages from [community] (in my case 0ad, it's really big, but it's also alphabetically first) run into ipfs/go-ipfs#1456 and crashes violently, complaining about too many open files.

When it does work, though, it works beautifully. I'm playing with it at /ipns/ani.sh/, but that's my laptop, so don't expect much uptime.

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Mar 8, 2016

Member

@atondwal thanks for the report! could you try upping your ulimit before starting the daemon and seeing how things go for you then?

On linux thats: ulimit -n 2048 (or any number you like,really)

Member

whyrusleeping commented Mar 8, 2016

@atondwal thanks for the report! could you try upping your ulimit before starting the daemon and seeing how things go for you then?

On linux thats: ulimit -n 2048 (or any number you like,really)

@atondwal

This comment has been minimized.

Show comment
Hide comment
@atondwal

atondwal Mar 8, 2016

Huh, that makes it fail to mount

~ ❯❯❯ ipfs daemon --mount                                                                                                                                                                                                                  ⏎
Initializing daemon...
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.0.9/tcp/4001
Swarm listening on /ip4/68.100.248.158/tcp/20207
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
01:53:18.061 ERROR    bitswap: failed to find any peer in table workers.go:92
01:53:18.144 ERROR core/comma: error mounting: %!s(<nil>) failed to find any peer in table mount_unix.go:219
Error: failed to find any peer in table

(without --mount it starts fine though...)

atondwal commented Mar 8, 2016

Huh, that makes it fail to mount

~ ❯❯❯ ipfs daemon --mount                                                                                                                                                                                                                  ⏎
Initializing daemon...
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/192.168.0.9/tcp/4001
Swarm listening on /ip4/68.100.248.158/tcp/20207
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
01:53:18.061 ERROR    bitswap: failed to find any peer in table workers.go:92
01:53:18.144 ERROR core/comma: error mounting: %!s(<nil>) failed to find any peer in table mount_unix.go:219
Error: failed to find any peer in table

(without --mount it starts fine though...)

@atondwal

This comment has been minimized.

Show comment
Hide comment
@atondwal

atondwal Mar 8, 2016

Oh, I just had to umount it by hand before remounting it, because I had a shell open at that directory when I restarted ipfs daemon.

Also after playing around with it, the problem wasn't the filesize, it was the fact that I was using cp instead of rsync. Which is sort of a strange thing to be the problem.

EDIT: No, the only difference is that rync fails gracefully. See all the very small files in /ipfs/QmQpNqAn44qsxLKABB5G2DGBKrtkmR5NGWXKS6iYWmrsZi ? Any idea what's causing that? Also the fact that there are only 82 entries, vs the 2672 packages I tried to dump in that directory.

atondwal commented Mar 8, 2016

Oh, I just had to umount it by hand before remounting it, because I had a shell open at that directory when I restarted ipfs daemon.

Also after playing around with it, the problem wasn't the filesize, it was the fact that I was using cp instead of rsync. Which is sort of a strange thing to be the problem.

EDIT: No, the only difference is that rync fails gracefully. See all the very small files in /ipfs/QmQpNqAn44qsxLKABB5G2DGBKrtkmR5NGWXKS6iYWmrsZi ? Any idea what's causing that? Also the fact that there are only 82 entries, vs the 2672 packages I tried to dump in that directory.

@atondwal

This comment has been minimized.

Show comment
Hide comment
@atondwal

atondwal Apr 15, 2016

Ah, the problem was with FUSE. Just doing ipfs add -r works like a charm.

atondwal commented Apr 15, 2016

Ah, the problem was with FUSE. Just doing ipfs add -r works like a charm.

@Kubuxu

This comment has been minimized.

Show comment
Hide comment
@Kubuxu

Kubuxu Apr 16, 2016

Member

@atondwal yes FUSE part has its quirks, it is better to use CLI for the most part.

Member

Kubuxu commented Apr 16, 2016

@atondwal yes FUSE part has its quirks, it is better to use CLI for the most part.

@Split7fire

This comment has been minimized.

Show comment
Hide comment
@Split7fire

Split7fire Apr 24, 2016

Have read an entire thread, but dont understand:

  • ipns is not an vialable solution?
  • someone say to use mfs instead, where can I read about it?
  • any chance for official support in Archlinux?

Split7fire commented Apr 24, 2016

Have read an entire thread, but dont understand:

  • ipns is not an vialable solution?
  • someone say to use mfs instead, where can I read about it?
  • any chance for official support in Archlinux?
@Kubuxu

This comment has been minimized.

Show comment
Hide comment
@Kubuxu

Kubuxu Nov 7, 2016

Member

I would love to help, as I have arch mirror in my local network already (it is 61GiB), but unfortunately go-ipfs is not able to punch the NAT I am behind.

Member

Kubuxu commented Nov 7, 2016

I would love to help, as I have arch mirror in my local network already (it is 61GiB), but unfortunately go-ipfs is not able to punch the NAT I am behind.

@chpio

This comment has been minimized.

Show comment
Hide comment
@chpio

chpio Nov 7, 2016

ok, so i did an rsync copy from an TIER2-Arch-mirror and added the dir into ipfs (with the default ipfs add -r . command), but im getting errors now from pacman and also when i try to open a file directly:

internalWebError: cannot currently read symlinks

https://ipfs.io/ipfs/QmdhPp1TSS2pLfNbAsTMPbnHtqA8i9oo5CypwhvKkar6Qp/core/os/x86_64/acl-2.2.52-2-x86_64.pkg.tar.xz.sig

What's the best way to add all the files and dirs as they are added right now, but resolve the symlinks, so they are added just like normal files? It seems like there is no flag for that in the ipfs add-cmd.

ps: 100GB shouldn't be a real problem in the modern days where you can get a 5TB HDD for a few $/€/$whatever

chpio commented Nov 7, 2016

ok, so i did an rsync copy from an TIER2-Arch-mirror and added the dir into ipfs (with the default ipfs add -r . command), but im getting errors now from pacman and also when i try to open a file directly:

internalWebError: cannot currently read symlinks

https://ipfs.io/ipfs/QmdhPp1TSS2pLfNbAsTMPbnHtqA8i9oo5CypwhvKkar6Qp/core/os/x86_64/acl-2.2.52-2-x86_64.pkg.tar.xz.sig

What's the best way to add all the files and dirs as they are added right now, but resolve the symlinks, so they are added just like normal files? It seems like there is no flag for that in the ipfs add-cmd.

ps: 100GB shouldn't be a real problem in the modern days where you can get a 5TB HDD for a few $/€/$whatever

@Kubuxu

This comment has been minimized.

Show comment
Hide comment
@Kubuxu

Kubuxu Nov 7, 2016

Member

Gateway is not supporting symlinks, which will be a problem in this case, there are three solutions for that, add symlink resolving to gateway, add symlink resolving to adder or write a adder script that would resolve symlink and use files api to build directory structure.

Member

Kubuxu commented Nov 7, 2016

Gateway is not supporting symlinks, which will be a problem in this case, there are three solutions for that, add symlink resolving to gateway, add symlink resolving to adder or write a adder script that would resolve symlink and use files api to build directory structure.

@Kubuxu Kubuxu closed this Nov 7, 2016

@Kubuxu Kubuxu reopened this Nov 7, 2016

@Split7fire

This comment has been minimized.

Show comment
Hide comment
@Split7fire

Split7fire Nov 7, 2016

you can replace symlinks by their referral during rsync

Split7fire commented Nov 7, 2016

you can replace symlinks by their referral during rsync

@chpio

This comment has been minimized.

Show comment
Hide comment
@chpio

chpio Nov 7, 2016

@Split7fire yeah, the disk usage went from 55GB to 126GB (just for the rsync copy). rsync is missing an option to convert symlinks to reflinks D: (https://bugzilla.samba.org/show_bug.cgi?id=10170) the ipfs add time went up also. a symlink-aware adder script (which would cache symlinks, so they wouldn't be added twice to ipfs, resulting in faster adds...) would be much nicer

edit: it's working now: https://ipfs.io/ipfs/QmfBVGtA5aW6baQB6ZPiBAHbUusMhWAnGDSULmsj4awSGx/

chpio commented Nov 7, 2016

@Split7fire yeah, the disk usage went from 55GB to 126GB (just for the rsync copy). rsync is missing an option to convert symlinks to reflinks D: (https://bugzilla.samba.org/show_bug.cgi?id=10170) the ipfs add time went up also. a symlink-aware adder script (which would cache symlinks, so they wouldn't be added twice to ipfs, resulting in faster adds...) would be much nicer

edit: it's working now: https://ipfs.io/ipfs/QmfBVGtA5aW6baQB6ZPiBAHbUusMhWAnGDSULmsj4awSGx/

@fazo96

This comment has been minimized.

Show comment
Hide comment
@fazo96

fazo96 Nov 7, 2016

Wow @chpio, that's great! I'll try it as soon as I can!

fazo96 commented Nov 7, 2016

Wow @chpio, that's great! I'll try it as soon as I can!

@chpio

This comment has been minimized.

Show comment
Hide comment
@chpio

chpio Nov 7, 2016

nice, but take in mind that this was just a test and im not updating it (it's already outdated)

chpio commented Nov 7, 2016

nice, but take in mind that this was just a test and im not updating it (it's already outdated)

@whoamin

This comment has been minimized.

Show comment
Hide comment
@whoamin

whoamin Feb 19, 2017

@chpio works for me only with port 8080. I have clean installation ipfs-0.4.5 and ipfs init, I mean:

# /etc/pacman.d/mirrorlist
Server = http://127.0.0.1:8080/ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/$repo/os/$arch

strange behavior for wget, idk real speed --.-KB/s

whoamin commented Feb 19, 2017

@chpio works for me only with port 8080. I have clean installation ipfs-0.4.5 and ipfs init, I mean:

# /etc/pacman.d/mirrorlist
Server = http://127.0.0.1:8080/ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/$repo/os/$arch

strange behavior for wget, idk real speed --.-KB/s

@wanderer

This comment has been minimized.

Show comment
Hide comment
@wanderer

wanderer Apr 25, 2017

Member

@chpio do you intend to keep /ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/ updated for the foreseeable future?

Member

wanderer commented Apr 25, 2017

@chpio do you intend to keep /ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/ updated for the foreseeable future?

@jruere

This comment has been minimized.

Show comment
Hide comment
@jruere

jruere Nov 13, 2017

AFAIU, https://github.com/ipfs-filestore/go-ipfs/tree/master/filestore could help a lot avoiding duplicating storage requirements.

PS: I could not find a peer with an up-to-date repository.

jruere commented Nov 13, 2017

AFAIU, https://github.com/ipfs-filestore/go-ipfs/tree/master/filestore could help a lot avoiding duplicating storage requirements.

PS: I could not find a peer with an up-to-date repository.

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@jruere

This comment has been minimized.

Show comment
Hide comment
@jruere

jruere Nov 13, 2017

@whyrusleeping that's very neat! Sorry I missed it.

Do I understand correctly that this would simplify adding IPFS to an existing mirror?

jruere commented Nov 13, 2017

@whyrusleeping that's very neat! Sorry I missed it.

Do I understand correctly that this would simplify adding IPFS to an existing mirror?

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Nov 13, 2017

Member

@jruere Yeah, it should make things a bit simpler. The caveat is around updating things, cc @kevina for discussion around updating files that are in the filestore.

Member

whyrusleeping commented Nov 13, 2017

@jruere Yeah, it should make things a bit simpler. The caveat is around updating things, cc @kevina for discussion around updating files that are in the filestore.

@jcaesar

This comment has been minimized.

Show comment
Hide comment
@jcaesar

jcaesar Dec 10, 2017

I've experimented around a bit with @chpio's repo and found it to be unbearably slow. I set up a cronjob that resolves /ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/ hourly and pushes the result to _dnslink.arch-chpio.cameo.liftm.de. Using http://arch-chpio.cameo.liftm.de:8080/$repo/os/$arch (depends on ipfs running on localhost:8080) is a bit faster and practically bearable, imho.

PS: (I know it's been years, but I feel like this should be mentioned)
@robcat:

having to recompress the package at installation time to verify the signature (at a great CPU cost)

From what I hear, recompressing files and getting the exact same binary is fragile at best. This may not be an option.

jcaesar commented Dec 10, 2017

I've experimented around a bit with @chpio's repo and found it to be unbearably slow. I set up a cronjob that resolves /ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/ hourly and pushes the result to _dnslink.arch-chpio.cameo.liftm.de. Using http://arch-chpio.cameo.liftm.de:8080/$repo/os/$arch (depends on ipfs running on localhost:8080) is a bit faster and practically bearable, imho.

PS: (I know it's been years, but I feel like this should be mentioned)
@robcat:

having to recompress the package at installation time to verify the signature (at a great CPU cost)

From what I hear, recompressing files and getting the exact same binary is fragile at best. This may not be an option.

@robcat

This comment has been minimized.

Show comment
Hide comment
@robcat

robcat Dec 10, 2017

@jcaesar

From what I hear, recompressing files and getting the exact same binary is fragile at best. This may not be an option.

There is an ongoing effort in deterministic builds on Arch Linux (77% done), and the compressing part of the project is the easiest one (compiling deterministically is much harder).

robcat commented Dec 10, 2017

@jcaesar

From what I hear, recompressing files and getting the exact same binary is fragile at best. This may not be an option.

There is an ongoing effort in deterministic builds on Arch Linux (77% done), and the compressing part of the project is the easiest one (compiling deterministically is much harder).

@chpio

This comment has been minimized.

Show comment
Hide comment
@chpio

chpio Feb 6, 2018

@chpio do you intend to keep /ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/ updated for the foreseeable future?

@wanderer Nope, im going to shut it down in a few days.

chpio commented Feb 6, 2018

@chpio do you intend to keep /ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/ updated for the foreseeable future?

@wanderer Nope, im going to shut it down in a few days.

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen Feb 6, 2018

So there is no resolution on this, except some unmaintained defunct mirrors that at one point has been working?

I have several laptops and other computers running arch linux, and a thin uplink - it's a shame to download the same packages over and over again. Also, some non-standard packages aren't available at the standard mirrors and may be really slow to download even when having a decent uplink, IPFS is just perfect for distributing such content.

I tried putting the /var/cache/pacman/pkg download directory under NFS for a while before learning about IPFS, but it was painful - remembering to mount it up when the laptop is at home and to unmount before leaving home, etc, not giving any benefits i.e. when being at vacation having to upgrade two laptops over mobile networks, etc. IPFS seems to be the perfect solution.

I think the "quick and dirty" solution would be to just ensure /var/cache/pacman/pkg being a ipfs mount - though, I'm having problems mounting up ipfs currently, so I haven't gone that route yet. I believe the proper solution is native support from arch linux, making sure ipfs hashes for the uncompressed tar files are available and that pacman can download it.

tobixen commented Feb 6, 2018

So there is no resolution on this, except some unmaintained defunct mirrors that at one point has been working?

I have several laptops and other computers running arch linux, and a thin uplink - it's a shame to download the same packages over and over again. Also, some non-standard packages aren't available at the standard mirrors and may be really slow to download even when having a decent uplink, IPFS is just perfect for distributing such content.

I tried putting the /var/cache/pacman/pkg download directory under NFS for a while before learning about IPFS, but it was painful - remembering to mount it up when the laptop is at home and to unmount before leaving home, etc, not giving any benefits i.e. when being at vacation having to upgrade two laptops over mobile networks, etc. IPFS seems to be the perfect solution.

I think the "quick and dirty" solution would be to just ensure /var/cache/pacman/pkg being a ipfs mount - though, I'm having problems mounting up ipfs currently, so I haven't gone that route yet. I believe the proper solution is native support from arch linux, making sure ipfs hashes for the uncompressed tar files are available and that pacman can download it.

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Feb 6, 2018

Member

No resolution yet, And this is something I would personally love to have too.

If anyone is interested in spending some time developing a proper solution here, let me know.

Member

whyrusleeping commented Feb 6, 2018

No resolution yet, And this is something I would personally love to have too.

If anyone is interested in spending some time developing a proper solution here, let me know.

@simonbru

This comment has been minimized.

Show comment
Hide comment
@simonbru

simonbru Feb 6, 2018

@tobixen I also think IPFS (or another P2P distribution mechanism) for linux packages is a cool solution. I hope people will continue to work on it.
However if your main use case is to share packages in your LAN, I suggest you to try pacserve in the meantime. It does not require much configuration of maintenance and works pretty well in my experience.

simonbru commented Feb 6, 2018

@tobixen I also think IPFS (or another P2P distribution mechanism) for linux packages is a cool solution. I hope people will continue to work on it.
However if your main use case is to share packages in your LAN, I suggest you to try pacserve in the meantime. It does not require much configuration of maintenance and works pretty well in my experience.

@jcaesar

This comment has been minimized.

Show comment
Hide comment
@jcaesar

jcaesar Feb 14, 2018

@chpio: I don't know if I have the resources to run something like this, but could I ask you to share whatever scripts or configuration you have that made the mirror work?

jcaesar commented Feb 14, 2018

@chpio: I don't know if I have the resources to run something like this, but could I ask you to share whatever scripts or configuration you have that made the mirror work?

@VictorBjelkholm

This comment has been minimized.

Show comment
Hide comment
@VictorBjelkholm

VictorBjelkholm Mar 5, 2018

Member

I've made some reusable scripts for pulling down a repository and adding it to IPFS. https://github.com/VictorBjelkholm/arch-mirror

Still, IPFS doesn't handle symlinks and this cannot be used for pacman just yet. But now we have some scripts to work from.

Member

VictorBjelkholm commented Mar 5, 2018

I've made some reusable scripts for pulling down a repository and adding it to IPFS. https://github.com/VictorBjelkholm/arch-mirror

Still, IPFS doesn't handle symlinks and this cannot be used for pacman just yet. But now we have some scripts to work from.

@Kubuxu

This comment has been minimized.

Show comment
Hide comment
@Kubuxu

Kubuxu Mar 5, 2018

Member

@VictorBjelkholm You can try initializing the repo with badgerds profile (ipfs init --profile=badgerds) it should be even faster then.

Member

Kubuxu commented Mar 5, 2018

@VictorBjelkholm You can try initializing the repo with badgerds profile (ipfs init --profile=badgerds) it should be even faster then.

@VictorBjelkholm

This comment has been minimized.

Show comment
Hide comment
@VictorBjelkholm

VictorBjelkholm Mar 8, 2018

Member

Thanks @Kubuxu, I'll try that in a future update.

Update

I've successfully managed to download, add to IPFS and use it for a full system upgrade already 🎉

The full instructions for usage and hosting your own mirror is over here: https://github.com/VictorBjelkholm/arch-mirror

TLDR of usage is: Add Server = https://ipfs.io/ipns/arch.victor.earth/$repo/os/$arch in the top of /etc/pacman.d/mirrorlist

This DNS record gets updated at least once per day, I have the sync setup to happen automatic twice a day. It's currently hosted on at least three different machines, my home server, a digitalocean droplet specifically for this arch-mirror and also pollux which is part of our standard infrastructure. I'll also pin this on ipfs-cluster once we've done a bit more testing.

If you want to help out now, please try the server above and report any issues either here on on https://github.com/VictorBjelkholm/arch-mirror/issues/new

If people are stepping up and want to help out maintaining the scripts, we can move the repository over to ipfs-shipyard instead of within my own Github scope.

Member

VictorBjelkholm commented Mar 8, 2018

Thanks @Kubuxu, I'll try that in a future update.

Update

I've successfully managed to download, add to IPFS and use it for a full system upgrade already 🎉

The full instructions for usage and hosting your own mirror is over here: https://github.com/VictorBjelkholm/arch-mirror

TLDR of usage is: Add Server = https://ipfs.io/ipns/arch.victor.earth/$repo/os/$arch in the top of /etc/pacman.d/mirrorlist

This DNS record gets updated at least once per day, I have the sync setup to happen automatic twice a day. It's currently hosted on at least three different machines, my home server, a digitalocean droplet specifically for this arch-mirror and also pollux which is part of our standard infrastructure. I'll also pin this on ipfs-cluster once we've done a bit more testing.

If you want to help out now, please try the server above and report any issues either here on on https://github.com/VictorBjelkholm/arch-mirror/issues/new

If people are stepping up and want to help out maintaining the scripts, we can move the repository over to ipfs-shipyard instead of within my own Github scope.

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen Mar 8, 2018

I'm a bit confused here ... sorry if the questions are dumb, I haven't really taken the time to understand ipns yet.

  • I set up http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch at the top of the mirror list - this option seems not to be covered in your README? Seems to work, except frequent fallbacks to the other mirrors due to timeouts - I should probably see if I can adjust up the timeout.
  • Wouldn't /ipns/arch.victor.earth/$repo/os/$arch work out when ipfs-mount is used? (I haven't had much luck with ipfs mount - Error: fuse failed to access mountpoint /ipfs despite ownership and permissions apparently being right - but that's a digression)
  • Will one gain any benefits at all using https://ipfs.io/ as compared to a conventional mirror?
  • IPFS hashes is immutable as far as I've understood (and that's the reason why we need ipns?) - so with the suggestion to use /ipfs/QmQgfsgwenh3tiCmgbhtxr9sAVCaxbBRFyYu1GjAp84rgo you'll get a permanently frozen "mirror"? ("harder for you to update" seems like an understatement?)

tobixen commented Mar 8, 2018

I'm a bit confused here ... sorry if the questions are dumb, I haven't really taken the time to understand ipns yet.

  • I set up http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch at the top of the mirror list - this option seems not to be covered in your README? Seems to work, except frequent fallbacks to the other mirrors due to timeouts - I should probably see if I can adjust up the timeout.
  • Wouldn't /ipns/arch.victor.earth/$repo/os/$arch work out when ipfs-mount is used? (I haven't had much luck with ipfs mount - Error: fuse failed to access mountpoint /ipfs despite ownership and permissions apparently being right - but that's a digression)
  • Will one gain any benefits at all using https://ipfs.io/ as compared to a conventional mirror?
  • IPFS hashes is immutable as far as I've understood (and that's the reason why we need ipns?) - so with the suggestion to use /ipfs/QmQgfsgwenh3tiCmgbhtxr9sAVCaxbBRFyYu1GjAp84rgo you'll get a permanently frozen "mirror"? ("harder for you to update" seems like an understatement?)
@VictorBjelkholm

This comment has been minimized.

Show comment
Hide comment
@VictorBjelkholm

VictorBjelkholm Mar 8, 2018

Member

I set up http://127.0.0.1:8080/ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/$repo/os/$arch at the top of the mirror list - this option seems not to be covered in your README? Seems to work, except frequent fallbacks to the other mirrors due to timeouts - I should probably see if I can adjust up the timeout.

Yeah, it's mentioned in the readme here: https://github.com/VictorBjelkholm/arch-mirror/blob/master/README.md#usage-with-local-daemon

Let me know if you figure out how to change the timeout, as it would be good to include in the readme.

Wouldn't /ipns/arch.victor.earth/$repo/os/$arch work out when ipfs-mount is used? (I haven't had much luck with ipfs mount - Error: fuse failed to access mountpoint /ipfs despite ownership and permissions apparently being right - but that's a digression)

Hm, I haven't had much luck with fuse mounts lately either, which is why I left that out. Fuse support in go-ipfs is notably in need of some love:

Will one gain any benefits at all using https://ipfs.io/ as compared to a conventional mirror?

Yeah, better caching than traditional mirrors (assumption, haven't actually looked into how other Tier 2 mirrors does caching yet) and verifiable packages. It's also part of our migration process, so we can easily move to something else in the future. For example, ipfs.io could resolve to localhost once every computer runs IPFS itself. But best is of course to already run your own go-ipfs instance locally.

IPFS hashes is immutable as far as I've understood (and that's the reason why we need ipns?) - so with the suggestion to use /ipfs/QmQgfsgwenh3tiCmgbhtxr9sAVCaxbBRFyYu1GjAp84rgo you'll get a permanently frozen "mirror"? ("harder for you to update" seems like an understatement?)

Yes, they are immutable and why we need IPNS or DNS. The suggestion means (but maybe not clear enough) that you would check either the readme of arch-mirror (I update that hash everytime I run the update) or resolve the IPNS/DNS record and replace it manually in your mirrorlist. If you keep the same hash forever, it'll be permanently frozen, that is correct. And you would have to manually update it to point to the new hash once you see it.

Member

VictorBjelkholm commented Mar 8, 2018

I set up http://127.0.0.1:8080/ipns/QmaeXrsLHWm4gbjyEUJ4NtPsF3d36mXVzY5eTBQHLdMQ19/$repo/os/$arch at the top of the mirror list - this option seems not to be covered in your README? Seems to work, except frequent fallbacks to the other mirrors due to timeouts - I should probably see if I can adjust up the timeout.

Yeah, it's mentioned in the readme here: https://github.com/VictorBjelkholm/arch-mirror/blob/master/README.md#usage-with-local-daemon

Let me know if you figure out how to change the timeout, as it would be good to include in the readme.

Wouldn't /ipns/arch.victor.earth/$repo/os/$arch work out when ipfs-mount is used? (I haven't had much luck with ipfs mount - Error: fuse failed to access mountpoint /ipfs despite ownership and permissions apparently being right - but that's a digression)

Hm, I haven't had much luck with fuse mounts lately either, which is why I left that out. Fuse support in go-ipfs is notably in need of some love:

Will one gain any benefits at all using https://ipfs.io/ as compared to a conventional mirror?

Yeah, better caching than traditional mirrors (assumption, haven't actually looked into how other Tier 2 mirrors does caching yet) and verifiable packages. It's also part of our migration process, so we can easily move to something else in the future. For example, ipfs.io could resolve to localhost once every computer runs IPFS itself. But best is of course to already run your own go-ipfs instance locally.

IPFS hashes is immutable as far as I've understood (and that's the reason why we need ipns?) - so with the suggestion to use /ipfs/QmQgfsgwenh3tiCmgbhtxr9sAVCaxbBRFyYu1GjAp84rgo you'll get a permanently frozen "mirror"? ("harder for you to update" seems like an understatement?)

Yes, they are immutable and why we need IPNS or DNS. The suggestion means (but maybe not clear enough) that you would check either the readme of arch-mirror (I update that hash everytime I run the update) or resolve the IPNS/DNS record and replace it manually in your mirrorlist. If you keep the same hash forever, it'll be permanently frozen, that is correct. And you would have to manually update it to point to the new hash once you see it.

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen Mar 8, 2018

I spent some time searching, but did not find any simple/obvious way to increase the timeout.

tobixen commented Mar 8, 2018

I spent some time searching, but did not find any simple/obvious way to increase the timeout.

@jruere

This comment has been minimized.

Show comment
Hide comment
@jruere

jruere Mar 8, 2018

Is this not enough?

XferCommand = /usr/bin/wget --timeout=180 -c -O %o %u

jruere commented Mar 8, 2018

Is this not enough?

XferCommand = /usr/bin/wget --timeout=180 -c -O %o %u

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen Mar 18, 2018

Usage of the XferCommand breaks the "TotalDownload"-feature in pacman, I had hoped there were some simple/obvious way to increase the timeout without resorting to that one.

Another workaround: It seems like the same URL can be specified several times in the pacman mirror list ...

$ grep 127.0.0.1 /etc/pacman.d/mirrorlist
Server = http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch
Server = http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch
Server = http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch

This doesn't seem to help me though, if it times out at the first one, it also times out at the two others.

Maybe it's just good to skip IPFS and go to some other mirror when IPFS times out. To help making the packages available at IPFS without going full-scale with running a full mirror, I've thrown up this cronjob as root:

*/6 * * * * sudo -u ipfs ipfs add -r --pin=false /var/cache/pkgfile /var/cache/pacman/pkg/ > /dev/null 2>&1 ; find /var/cache/pacman/pkg -mmin +30 -delete

I wish ipfs mount had been working ... having /var/cache/pacman/pkg working directly from ipfs would be super.

tobixen commented Mar 18, 2018

Usage of the XferCommand breaks the "TotalDownload"-feature in pacman, I had hoped there were some simple/obvious way to increase the timeout without resorting to that one.

Another workaround: It seems like the same URL can be specified several times in the pacman mirror list ...

$ grep 127.0.0.1 /etc/pacman.d/mirrorlist
Server = http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch
Server = http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch
Server = http://127.0.0.1:8080/ipns/arch.victor.earth/$repo/os/$arch

This doesn't seem to help me though, if it times out at the first one, it also times out at the two others.

Maybe it's just good to skip IPFS and go to some other mirror when IPFS times out. To help making the packages available at IPFS without going full-scale with running a full mirror, I've thrown up this cronjob as root:

*/6 * * * * sudo -u ipfs ipfs add -r --pin=false /var/cache/pkgfile /var/cache/pacman/pkg/ > /dev/null 2>&1 ; find /var/cache/pacman/pkg -mmin +30 -delete

I wish ipfs mount had been working ... having /var/cache/pacman/pkg working directly from ipfs would be super.

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen May 2, 2018

@VictorBjelkholm is your mirror still functioning? I get timeouts all the time now. Could I have the "swarm address" to the mirror so I can add it to my peer lists?

tobixen commented May 2, 2018

@VictorBjelkholm is your mirror still functioning? I get timeouts all the time now. Could I have the "swarm address" to the mirror so I can add it to my peer lists?

@VictorBjelkholm

This comment has been minimized.

Show comment
Hide comment
@VictorBjelkholm

VictorBjelkholm May 2, 2018

Member

@tobixen hm, that's weird. It's currently hosted at two nodes, one I have running at home and one at Digital Ocean. The addresses:

/ip4/88.18.104.235/tcp/7000/ipfs/QmUWA5FD6otih6aCqnSTSy4cQbcttwCYpA6mYMPJ9qAyKi
/ip4/167.99.95.100/tcp/4001/ipfs/QmX1WWwNwfEsuzsqWFVstnEkzVFN5i4kefqJQMb5S4umAT

Probably need to setup some integration with our ipfs cluster we have running, and pin it there on every update. I've checked the logs and have been running fine since the inception.

Member

VictorBjelkholm commented May 2, 2018

@tobixen hm, that's weird. It's currently hosted at two nodes, one I have running at home and one at Digital Ocean. The addresses:

/ip4/88.18.104.235/tcp/7000/ipfs/QmUWA5FD6otih6aCqnSTSy4cQbcttwCYpA6mYMPJ9qAyKi
/ip4/167.99.95.100/tcp/4001/ipfs/QmX1WWwNwfEsuzsqWFVstnEkzVFN5i4kefqJQMb5S4umAT

Probably need to setup some integration with our ipfs cluster we have running, and pin it there on every update. I've checked the logs and have been running fine since the inception.

@jruere

This comment has been minimized.

Show comment
Hide comment
@jruere

jruere May 2, 2018

jruere commented May 2, 2018

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen May 3, 2018

For me, ipfs on ipv6 works out quite well sometimes. ipfs on NAT and ipv4 sometimes doesn't work at all. I just changed ISP at home, the new ISP is supposedly supporting ipv6, but in practice my ipv6 setup at home has been down since I changed provider. I will try to do some research on this.

Another thing I found is that it's the network discovery that takes time. Manually connecting to the "right" peers either in the bootstrap configuration or through the "ipfs swarm connect"-command works out wonders.

I wonder if it's any way to mirror all the metadata without running a full package mirror? Due to the cronjob I set up above, I often have all the packages I need available locally on ipfs nodes in my house, but since I don't know the hash for them (due to ipfs timeouts) it's still needed to download all the packages from some official mirror.

It seems to me that ipfs is currently not very well suited for "the long tail" of content, when being the only node searching for some content it may be impossible to find it. If many nodes are searching for the same content, ipfs probably works great. Meaning that the more people that uses the mirror, the better the performance will be.

tobixen commented May 3, 2018

For me, ipfs on ipv6 works out quite well sometimes. ipfs on NAT and ipv4 sometimes doesn't work at all. I just changed ISP at home, the new ISP is supposedly supporting ipv6, but in practice my ipv6 setup at home has been down since I changed provider. I will try to do some research on this.

Another thing I found is that it's the network discovery that takes time. Manually connecting to the "right" peers either in the bootstrap configuration or through the "ipfs swarm connect"-command works out wonders.

I wonder if it's any way to mirror all the metadata without running a full package mirror? Due to the cronjob I set up above, I often have all the packages I need available locally on ipfs nodes in my house, but since I don't know the hash for them (due to ipfs timeouts) it's still needed to download all the packages from some official mirror.

It seems to me that ipfs is currently not very well suited for "the long tail" of content, when being the only node searching for some content it may be impossible to find it. If many nodes are searching for the same content, ipfs probably works great. Meaning that the more people that uses the mirror, the better the performance will be.

@tobixen

This comment has been minimized.

Show comment
Hide comment
@tobixen

tobixen May 3, 2018

For the reference, I did this:

ipfs swarm connect /ip4/88.18.104.235/tcp/7000/ipfs/QmUWA5FD6otih6aCqnSTSy4cQbcttwCYpA6mYMPJ9qAyKi 
ipfs swarm connect /ip4/167.99.95.100/tcp/4001/ipfs/QmX1WWwNwfEsuzsqWFVstnEkzVFN5i4kefqJQMb5S4umAT

and then upgraded the system - packages were downloaded from ipfs without any problems whatsoever!

Maybe an idea to add a hint about this in the README.

tobixen commented May 3, 2018

For the reference, I did this:

ipfs swarm connect /ip4/88.18.104.235/tcp/7000/ipfs/QmUWA5FD6otih6aCqnSTSy4cQbcttwCYpA6mYMPJ9qAyKi 
ipfs swarm connect /ip4/167.99.95.100/tcp/4001/ipfs/QmX1WWwNwfEsuzsqWFVstnEkzVFN5i4kefqJQMb5S4umAT

and then upgraded the system - packages were downloaded from ipfs without any problems whatsoever!

Maybe an idea to add a hint about this in the README.

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping May 3, 2018

Member

poor content discovery in the past ~3mo is very likely due to recent degradation in DHT performance. We're working on improvements here, see this issue for details.

Member

whyrusleeping commented May 3, 2018

poor content discovery in the past ~3mo is very likely due to recent degradation in DHT performance. We're working on improvements here, see this issue for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment