Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Why can't I access the ipfs mirror? #20

Closed
RubenKelevra opened this issue Apr 21, 2020 · 15 comments
Closed

Why can't I access the ipfs mirror? #20

RubenKelevra opened this issue Apr 21, 2020 · 15 comments
Labels
bug Something isn't working documentation Improvements or additions to documentation

Comments

@RubenKelevra
Copy link
Owner

RubenKelevra commented Apr 21, 2020 •
edited

The cluster is already running the latest 0.5.0-Software which has major speed improvements for resolution of content-addresses as well as transfer speeds.

The last maintenance on 2020-04-21 UTC changed the CID's of the mirror to CIDv1.

This allows the cluster to be future proof, even when some packages might not be updated in a long timeframe.

But this breaks the compatibility with the current 0.4.23 version.

TL;DR: Update your system to the 0.5.0-version before accessing the IPFS-mirror
@RubenKelevra RubenKelevra added bug Something isn't working documentation Improvements or additions to documentation labels Apr 21, 2020
@RubenKelevra RubenKelevra pinned this issue Apr 21, 2020
@RubenKelevra RubenKelevra added the wontfix This will not be worked on label Apr 21, 2020
@rpodgorny
Copy link

just to be clean (i'm not really familiar with ipfs, yet) -> does this explain why i see this?

radek@prdell ~> sudo pacman -Syu --disable-download-timeout (+00:00:23) 2020-05-01 14:46:35
:: Synchronizing package databases...
core is up to date 0.0 B 0.00 B/s 00:00 [---------------------------------------------------------------------] 0%
extra is up to date 0.0 B 0.00 B/s 00:00 [---------------------------------------------------------------------] 0%
community is up to date 0.0 B 0.00 B/s 00:00 [---------------------------------------------------------------------] 0%
multilib is up to date 0.0 B 0.00 B/s 00:00 [---------------------------------------------------------------------] 0%
rpodgorny is up to date
:: Starting full system upgrade...
warning: ghc-libs: local (8.10.1-2) is newer than community (8.8.3-2)
warning: haskell-aeson: local (1.4.7.1-7) is newer than community (1.4.7.1-6)
warning: haskell-aeson-pretty: local (0.8.8-36) is newer than community (0.8.8-35)
warning: haskell-ansi-terminal: local (0.10.3-17) is newer than community (0.10.3-16)
warning: haskell-asn1-encoding: local (0.9.6-25) is newer than community (0.9.6-24)
warning: haskell-asn1-parse: local (0.9.5-25) is newer than community (0.9.5-24)
warning: haskell-asn1-types: local (0.3.4-4) is newer than community (0.3.4-3)
warning: haskell-async: local (2.2.2-7) is newer than community (2.2.2-6)
warning: haskell-attoparsec: local (0.13.2.4-3) is newer than community (0.13.2.4-2)
warning: haskell-base-compat: local (0.11.1-7) is newer than community (0.11.1-4)
warning: haskell-base-compat-batteries: local (0.11.1-25) is newer than community (0.11.1-24)
...

@rpodgorny
Copy link

...or is your mirror just slightly behind? and if so, is that to be expected? thanks!

@RubenKelevra
Copy link
Owner Author

RubenKelevra commented May 1, 2020 •
edited

Well, I'm not sure you really used IPFS... since IPFS cannot provide the "last changed" tag on HTTP, so the databases will be always fetched from the local node.

Please make sure that IPFS is the first mirror on your mirrorlist :)

It should look like this:

$ sudo /usr/bin/pacman -Sy
:: Synchronizing package databases...
 core                                                             134,4 KiB  65,6 MiB/s 00:00 [#######################################################] 100%
 extra                                                           1641,2 KiB   401 MiB/s 00:00 [#######################################################] 100%
 community                                                          4,9 MiB   409 MiB/s 00:00 [#######################################################] 100%
 multilib                                                         159,6 KiB   156 MiB/s 00:00 [#######################################################] 100%
[ruben@i3 ~]$

@RubenKelevra
Copy link
Owner Author

RubenKelevra commented May 1, 2020 •
edited

...or is your mirror just slightly behind? and if so, is that to be expected? thanks!

The mirror checks minutely for new updates and fetch them. Download, processing, publishing takes a while. Since an update is atomic, large updates on the mirror I'm copying might take up to say 10 minutes to process. But that should be the maximum delay.

But I don't mirror from the tier 0 mirror, but a tier 1. So there's an additionally 10-15 minute delay.

@RubenKelevra
Copy link
Owner Author

I took a look at the server which imports the packages and the IPFS has an unexpected high CPU load, which slowed down the import process.

I gonna update the server to the latest version of IPFS and restart the import process.

Thanks for the report!

@rpodgorny
Copy link

no problem ;-) ...please post here when everything is fixed and i can test it.

@RubenKelevra
Copy link
Owner Author

Looks like the server hit two bugs at once:

Import speed is extremely low thru the ipfs-cluster-ctl-command, when running ipfs 0.5.0 (bug one).

And IPFS itself used way too much memory and CPU while basically idling. Which slowed everything down even more. (bug two)

The first one has now a workaround, the second one seems to happen very irregularly - I keep an eye on that.

The server is somewhat behind now and needs some time to catch up. I keep you posted.

Sorry for the inconveniences

@RubenKelevra
Copy link
Owner Author

@rpodgorny should work fine now, make sure to use pacman -Syy on the first sync, to avoid that some databases won't be updated.

@rpodgorny
Copy link

how about now?

it used to work fine after your fix but it seems to be taking forever again for the last two days or so... :-(

@mohe2015
Copy link

Do you have enabled experimental settings of ipfs? (I don't know if they are still needed as of v0.5 but they may help, --enable-namesys-pubsub and quic may help). Which version of ipfs are you using and in which country do you live?

@RubenKelevra
Copy link
Owner Author

@mohe2015 QUIC isn't enabled on the servers I maintain, because of issues with memory usage (there are some memory leaks). See ipfs/kubo#7263

@rpodgorny The importing server is currently the only one providing the directorys due to a ipfs-cluster restriction it impacts the whole performance.

This has been fixed and will be soon included in the setup. See ipfs-cluster/ipfs-cluster#1009

The importing server had some trouble with hitting the connection limit at that time you reported it. I fixed that and it shouldn't happen anymore in the future.

If you still experiencing issues with using this mirror you might hitting an issue in IPFS itself. IPFS has currently a hard time distinguishing WiFi disconnects or other network interruptions and recovering from them.

The easiest solution for this is currently to restart the IPFS daemon and wait a minute for it to bootstrap.

See ipfs/kubo#7211

@mohe2015
Copy link

The importing server is currently the only one providing the directorys due to a ipfs-cluster restriction it impacts the whole performance.

I updated https://aur.archlinux.org/packages/ipfs-cluster-bin/ to version 0.13.0.

Would this improve the situation if you changed your import script?

@RubenKelevra
Copy link
Owner Author

The importing server is currently the only one providing the directorys due to a ipfs-cluster restriction it impacts the whole performance.

I updated https://aur.archlinux.org/packages/ipfs-cluster-bin/ to version 0.13.0.

Would this improve the situation if you changed your import script?

Yes, that's the release I've been waiting for.

I need to traverse all directories before an update is processed and remove them from the cluster if they have been changed by an update and add the new directories non-recursive to the cluster afterward.

That's some major changes, not sure when I'm able to implement, test it.

@RubenKelevra
Copy link
Owner Author

To improve the communication on this setup I've started a maintenance log, with status:

https://github.com/RubenKelevra/pacman.store/wiki/Status,-Announcements-&-Maintenance

@RubenKelevra RubenKelevra removed the wontfix This will not be worked on label May 20, 2020
@RubenKelevra
Copy link
Owner Author

I've added an FAQ to replace this ticket:

https://github.com/RubenKelevra/pacman.store/wiki/FAQ

@RubenKelevra RubenKelevra unpinned this issue May 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@RubenKelevra @rpodgorny @mohe2015