Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many open files #366

Closed
reb0rn21 opened this issue Oct 5, 2020 · 3 comments
Closed

Too many open files #366

reb0rn21 opened this issue Oct 5, 2020 · 3 comments

Comments

@reb0rn21
Copy link

reb0rn21 commented Oct 5, 2020

version: Nym 0.8.1
system: Ubuntu 18.04
running precompiled bin with systemd

log file show after few days:
failed to send presence - reqwest::Error { kind: Request, url: "https://directory.nymtech.net/api/presence/mixnodes", source: hyper::Error(Connect, ConnectError("dns error", Os { code: 24, kind: Other, message: "Too many open files" })) }

mixing is still working but dashboard.nymtech.net do not show node anymore

after restart node is again show on site

@futurechimp
Copy link
Contributor

This is due to the fact that the network has grown about 20x in the past few weeks to ~500 nodes, and we still have one centralized component left over from when we had 6 nodes - the directory server that provides an overview of what nodes exist. The directory server has now been merged into our new (decentralized) validators and we're working towards a release probably end of this week or beginning of next week which will start to fix this problem.

The other part of the problem here is that a lot of people are running nodes which are misconfigured. So your node is probably trying to connect to other nodes which show up in the directory listing, but which are not actually accepting mixnet traffic Maybe they're firewalled off, maybe they node operator reported the wrong IP address during node init.

This will be completely fixed over the next few weeks as we are currently building a system of mixnode staking, to reward people running properly configured nodes and penalize node operators with misconfigured nodes. Nodes that don't mix will be knocked out of the network as people won't stake on them, and the network should then only rarely have nodes that don't work.

@futurechimp futurechimp added this to Backlog in Core systems via automation Oct 12, 2020
@futurechimp futurechimp added this to the 0.9.0 milestone Oct 12, 2020
@futurechimp futurechimp modified the milestones: 0.9.0, 0.9.1 Nov 5, 2020
@gyrusdentatus
Copy link

gyrusdentatus commented Nov 5, 2020

This is more a system configuration/Unix issue if I understand it correctly. Can be solved with ulimit -n 4096 . This will set the limit for "open files" to 4096, when default is 1024.

I had the same issue, node was not accessible from the outside although it was still mixing somehow.

@futurechimp futurechimp moved this from Backlog to Cold Storage in Core systems Nov 11, 2020
@futurechimp
Copy link
Contributor

I had the same issue, node was not accessible from the outside although it was still mixing somehow.

Yeah you would see this kind of behaviour - previously established connections would continue to function, but new connections would be disallowed by the operating system.

The new 0.9.1 release (out today) has fixes for ulimit-related networking problems. People should also set their default ulimits higher as noted at https://nymtech.net/docs/run-nym-nodes/mixnodes/ in order to stop the "too many connections" error and resume normal service.

Core systems automation moved this from Cold Storage to Done Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Core systems
  
Done
Development

No branches or pull requests

3 participants