best practices to run a big ipfs node

[johnongit]

Hello.

Version information:

0.4.14

Type

Question.

I want to know best practices in order to run a big ipfs node (pin > 50 GB)

Description:

I'm hosting a an ipfs node (0.4.14) on a VPS server (Ubuntu 16.04).

I'm currently pinning more than 20GB and I regularly have the following errors :

• too many open files
• failed to bind to anycast
  09:15:14.085 ERROR mdns: mdns lookup error: failed to bind to any unicast udp port asm_amd64.s:2361
  09:15:24.085 ERROR mdns: mdns lookup error: failed to bind to any unicast udp port asm_amd64.s:2361
  09:15:34.085 ERROR mdns: mdns lookup error: failed to bind to any multicast udp port asm_amd64.s:2361
  09:15:44.085 ERROR mdns: mdns lookup error: failed to bind to any unicast udp port asm_amd64.s:2361
• takes huge RAM even using zram and swap (> 12 GB)
• ipfs repo gc is very slow (more than 20 minues)

I'm currently running the deamon like this

sudo IPFS_FD_MAX=50240 IPFS_PATH=/var/ipfs/ ipfs daemon --routing=dhtclient

I would like to know if I need to upgrade my server, enhance max number of opened files or configure another thing.
My goal is to pin more than 50GB.

[magik6k]

• You want to run the daemon with the server profile enabled - do ipfs init --profile=server or ipfs config profile apply server. This won't affect performance much but is a good practice when running ipfs on a internet-facing server
• Using the --offline flag for daemon will speedup the adding process significantly, but may affect reachability of the data.
• Using the Badger datastore will usually help a lot when adding big data sets. Either do ipfs init --profile=server,badgerds or
  • make sure the daemon is not running
  • get ipfs-ds-convert from https://dist.ipfs.io/#ipfs-ds-convert or build from source - https://github.com/ipfs/ipfs-ds-convert
  • make sure you have enough disk space (conversion process will make your repo about 2x larger)
  • ipfs config profile apply badgerds
  • ipfs-ds-convert convert --keep
  • start ipfs daemon to see if it works (and try running command like ipfs pin ls -t recursive to see if the data is still there)
  • run ipfs-ds-convert cleanup to remove backup data
• After the data is added you may want to tell reprovider to only announce root pin objects - run ipfs config Reprovider.Strategy roots

Note that if you need ipfs repo gc to work well, using badger may not be the best idea for now - see https://github.com/ipfs/go-ipfs/issues/4824

[johnongit]

Wow big thx.

It’s working well now. And I’m able to fetch content pinned on my ipfs node.

[flyingzumwalt]

I'm glad @magik6k's advice sorted you out. You also might want to look into using ipfs-cluster

[johnongit]

Hmmm I think I still have memory consummation problems.

sudo IPFS_FD_MAX=50240 IPFS_PATH=/var/ipfs/ ipfs daemon

17:46:52.080 ERROR  cmds/http: write tcp4 127.0.0.1:5001->127.0.0.1:39288: write: broken pipe response.go:131
17:46:52.083 ERROR   cmds/lgc: write tcp4 127.0.0.1:5001->127.0.0.1:39288: write: broken pipe command.go:107
17:47:32.273 ERROR  cmds/http: write tcp4 127.0.0.1:5001->127.0.0.1:40600: write: broken pipe response.go:131
17:47:32.358 ERROR   cmds/lgc: write tcp4 127.0.0.1:5001->127.0.0.1:40600: write: broken pipe command.go:107
17:50:03.409 ERROR  cmds/http: write tcp4 127.0.0.1:5001->127.0.0.1:45200: write: broken pipe response.go:131
17:50:03.436 ERROR   cmds/lgc: write tcp4 127.0.0.1:5001->127.0.0.1:45200: write: broken pipe command.go:107
17:50:16.359 ERROR  cmds/http: write tcp4 127.0.0.1:5001->127.0.0.1:45620: write: broken pipe response.go:131
17:50:16.364 ERROR   cmds/lgc: write tcp4 127.0.0.1:5001->127.0.0.1:45620: write: broken pipe command.go:107

Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)
17:51:29.305 ERROR ipns-repub: Republisher failed to republish: process closing core.go:518
^CReceived another interrupt before graceful shutdown, terminating...

I've killed the daemon because IPFS was using too much memory (8 GB memory + 8GB swap).
I've readjusted connection manager (hight 600, low 300).

During this time, the number of peers was 600.

after restarting, my node has 600 peers and consumes 1GB.

[magik6k]

Can You get heap/goroutine dump? https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md

Were you adding data when this happened or was it just a daemon running? If you were adding something, what was the structure of the data (roughly, e.g. 'big directories, many small files', 'deep directory tree with some big files', etc.)

[johnongit]

Hi @magik6k I've added additionnal informations in #5046

The following behavior appears when running the daemon (no add no ls).
My node has currently pinned 28GB (for 375 recursive objects). All these (except few of them) has been pinned.

The dump
https://ipfs.io/ipfs/QmRhYCJcwy1G1gfCQtBtEDDWNY7ns4B9aGobT5ujeEEti8