New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many open files on add with daemon #3792

Closed
Kubuxu opened this Issue Mar 17, 2017 · 18 comments

Comments

Projects
None yet
10 participants
@Kubuxu
Member

Kubuxu commented Mar 17, 2017

Version: 0.4.7

Just running: dd bs=128M if=/dev/urandom count=1 | pv -a | ipfs add causes too many files error in 80% of cases.

 23.00 MB / ? [-------------------------------------------------------------=--------------------] 0+1 records in
0+1 records out
33554431 bytes (34 MB, 32 MiB) copied, 0.785311 s, 42.7 MB/s
[40.8MiB/s]
 32.00 MB / ? [-------------------------------------------------------------=--------------------] 12:00:54.217 ERROR commands/h: open /home/kubuxu/.ipfs/blocks/WP/put-927501522: too many open files client.go:247
Error: open /home/kubuxu/.ipfs/blocks/WP/put-927501522: too many open files
@NeoTeo

This comment has been minimized.

NeoTeo commented Mar 18, 2017

Also on
go-ipfs version: 0.4.7-
Repo version: 5
System version: amd64/darwin
Golang version: go1.8

I'm also getting a bunch (133 in a row last time) of
flatfs: too many open files, retrying in 500ms flatfs.go:180

It seemed to be caused by loading some new hash into my browser from outside (not already in my node) but cannot reproduce.

@Kubuxu

This comment has been minimized.

Member

Kubuxu commented Mar 18, 2017

The flatfs warning is not a problem (it is working) but in this case it fails the add process.

@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Mar 18, 2017

Lets go ahead and raise the default fd limit to 2048 for now. The number of nodes on the network is getting larger.

Longer term solutions will be adding QUIC support, and implementing connection closing and limiting strategies

@whyrusleeping whyrusleeping added this to the Ipfs 0.4.8 milestone Mar 18, 2017

@ghost

This comment has been minimized.

ghost commented Mar 20, 2017

Running the daemon with --offline should avoid this, since it's the same root cause as many other open bugs (flood of outgoing network traffic overloading something or other).

@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Mar 24, 2017

This should be temporarily resolved by #3828

We're working on several different solutions that will all help with this:

  • QUIC, a udp based transport, will use only a single file descriptor for all connections
  • sbs, a new block storage engine, will keep and use only a single file descriptor for all put operations
  • connection closing, in the next few months we will be working on strategies for selecting open connections to close out. This will reduce the overall load on the daemon, including reducing the number of open file descriptors
@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Mar 24, 2017

We can either keep this open until other solutions are implemented, or close this once #3828 gets in.

Thoughts @Kubuxu ?

@Kubuxu

This comment has been minimized.

Member

Kubuxu commented Mar 24, 2017

I would keep it.

@whyrusleeping whyrusleeping modified the milestones: Ipfs 0.4.9, Ipfs 0.4.8 Mar 25, 2017

@ligi

This comment has been minimized.

ligi commented Apr 5, 2017

also stumbling into this right now with a ci-server I wrote that writes reports to IPFS:

14:40:14.196 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
14:40:14.302 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
14:40:14.502 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
14:40:14.802 ERROR     flatfs: too many open files, retrying in 400ms flatfs.go:180
14:40:15.272 ERROR     flatfs: too many open files, retrying in 100ms flatfs.go:180
14:40:15.373 ERROR     flatfs: too many open files, retrying in 200ms flatfs.go:180
14:40:15.573 ERROR     flatfs: too many open files, retrying in 300ms flatfs.go:180
14:40:16.109 ERROR commands/h: err: open /home/kontinuum/.ipfs/blocks/QK/put-044922444: too many open files handler.go:288

unfortunately "--offline" is not an option as far as I see - is there another workaround?

EDIT: just saw 0.4.8 is now out and I am on 0.4.7 - will try if the problem vanishes with 0.4.8

EDIT#2: Dam it also happens with 0.4.8:

15:03:57.094 ERROR commands/h: err: open /home/kontinuum/.ipfs/blocks/LP/put-915734068: too many open files handler.go:288
15:04:03.748 ERROR       mdns: mdns lookup error: failed to bind to any multicast udp port mdns.go:135
^C
Received interrupt signal, shutting down...
(Hit ctrl-c again to force-shutdown the daemon.)
kontinuum@ligi-tp ~> ipfs version
ipfs version 0.4.8

@whyrusleeping

This comment has been minimized.

Member

whyrusleeping commented Apr 5, 2017

@ligi when trying 0.4.8, does the daemon output the message saying its raising the file descriptor limit to 2048 ?

@ligi

This comment has been minimized.

ligi commented Apr 5, 2017

@Kubuxu Kubuxu added the bug label Apr 17, 2017

@whyrusleeping whyrusleeping modified the milestones: Ipfs 0.4.10, Ipfs 0.4.9 May 2, 2017

@htor

This comment has been minimized.

htor commented Jul 5, 2017

Getting the same error here on os x 10.11

client

ipfs version 0.4.10
...

 72.00 MB / 875.45 MB [========>------------------------------------------------------------------------------------------]   8.22% 14s22:14:29.876 ERROR commands/h: open /Users/herman/.ipfs/blocks/6R/put-228389154: too many open files client.go:247
Error: open /Users/herman/.ipfs/blocks/6R/put-228389154: too many open files

daemon:

...
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
21:53:22.811 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
22:14:29.876 ERROR commands/h: err: open /Users/herman/.ipfs/blocks/6R/put-228389154: too many open files handler.go:285
22:14:32.811 ERROR       mdns: mdns lookup error: failed to bind to any unicast udp port mdns.go:135
@alphaCTzo7G

This comment has been minimized.

alphaCTzo7G commented Jul 22, 2017

Got this as well today.. was following the tutorial here..

Probably related to the number of files added... happened while I was running the ipns example here..

http://www.atnnn.com/p/ipfs-hosting/

@Kubuxu Kubuxu marked this as a duplicate of #4096 Jul 27, 2017

@magik6k

This comment has been minimized.

Member

magik6k commented Jul 27, 2017

Gave it a shoot, connection closing should help. Using this script to monitor deamon FD usage:

export DPID=26024; watch -n0 'printf "sockets: %s\nleveldb: %s\nflatfs: %s\n" $(ls /proc/${DPID}/fd/ -l | grep "socket:" | wc -l) $(ls /proc/${DPID}/fd/ -l | grep "\\/datastore\\/" | wc -l) $(ls /proc/${DPID}/fd/ -l | grep "\\/blocks\\/" | wc -l)'

where DPID is daemon PID, output looks like this:

sockets: 448
leveldb: 8
flatfs: 1

When adding a large file(dd bs=1M if=/dev/urandom count=1000 | ipfs add --pin=false), flatfs rarely gets up to 30, sockets sometimes 'explode' to 2k, though they can't be seen in ipfs swarm peers which is constantly at ~500 peers.

@magik6k magik6k marked this as a duplicate of #4101 Jul 28, 2017

@ETeissonniere

This comment has been minimized.

ETeissonniere commented Jul 28, 2017

Got this on 0.4.10 today:

Version information:

go-ipfs version: 0.4.10-
Repo version: 5
System version: amd64/linux
Golang version: go1.8.3

Description:

Trying to add a big (70MB) file to ipfs make it crash.

Here is the output of the command:

$ ipfs add backup.tar.gz 
 72.00 MB / 78.10 MB [============================================================================================================================================================================================>----------------]  92.19% 1s18:55:08.136 ERROR commands/h: open /home/eliott/.ipfs/blocks/MX/put-110547071: too many open files client.go:247
Error: open /home/user/.ipfs/blocks/MX/put-110547071: too many open files

On the daemon side:

$ ipfs daemon
Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
Swarm listening on /ip4/10.1.1.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/172.17.0.1/tcp/4001
Swarm listening on /ip4/172.18.0.1/tcp/4001
Swarm listening on /ip4/192.168.42.175/tcp/4001
Swarm listening on /ip4/78.233.22.41/tcp/4001
Swarm listening on /ip6/::1/tcp/4001
API server listening on /ip4/127.0.0.1/tcp/5001
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready
18:55:08.136 ERROR commands/h: err: open /home/user/.ipfs/blocks/MX/put-110547071: too many open files handler.go:285

Some more infos:

$ ipfs diag sys
{"diskinfo":{"free_space":249762070528,"fstype":"61267","total_space":231902359552},"environment":{"GOPATH":"/home/user/Documents/GoLang","IPFS_PATH":""},"ipfs_commit":"","ipfs_version":"0.4.10","memory":{"swap":0,"virt":762644000},"net":{"interface_addresses":["/ip4/127.0.0.1","/ip4/10.1.1.1","/ip4/172.18.0.1","/ip4/172.17.0.1","/ip4/192.168.42.175","/ip6/::1","/ip6/fe80::4ef4:6811:8297:44d0"],"online":true},"runtime":{"arch":"amd64","compiler":"gc","gomaxprocs":4,"numcpu":4,"numgoroutines":2564,"os":"linux","version":"go1.8.3"}}
@magik6k

This comment has been minimized.

Member

magik6k commented Jul 28, 2017

As explained above the problem is caused by creating way too many new connections when a big file is added. Workaround is to run daemon with IPFS_FD_MAX env variable set to 4k.

IPFS_FD_MAX=4096 ipfs daemon

@magik6k

This comment has been minimized.

Member

magik6k commented Aug 18, 2017

Seems to have been fixed by fixing #4102, haven't ran into it since the fix was merged.

@magik6k magik6k closed this Aug 18, 2017

@kamaci

This comment has been minimized.

kamaci commented Mar 6, 2018

Whenever I try to start a private network I get ulimit error:

Initializing daemon...
Adjusting current ulimit to 2048...
Successfully raised file descriptor limit to 2048.
00:10:32.457 ERROR   cmd/ipfs: error from node construction: EOF daemon.go:320
Error: EOF

I've increased ulimit size via IPFS_FD_MAX, tried fresh install with declaring IPFS_PATH but didn't work. When I remove swarm.key (https://github.com/Kubuxu/go-ipfs-swarm-key-gen) I get no error.

@Stebalien

This comment has been minimized.

Contributor

Stebalien commented Mar 6, 2018

I don't see any ulimit errors there. It looks like go-ipfs is prematurely hitting the end of a file (probably your swarm key) when starting. Let's open another issue, this should have a better error message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment