Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.10/master: "too many open files" during web-UI activity #1618

Closed
onlyjob opened this issue Nov 14, 2020 · 4 comments
Closed

2.10/master: "too many open files" during web-UI activity #1618

onlyjob opened this issue Nov 14, 2020 · 4 comments

Comments

@onlyjob
Copy link

onlyjob commented Nov 14, 2020 

+ ulimit -Sn 16384                                                                                                                                           
+ ulimit -n                                                                                                                                                  
16384                                                                                                                                                        
+ weed master -ip=192.168.0.204 -port=9333 -peers=server1:9333,server2:9333 -defaultReplication=001
[...]
I1115 10:31:22 76619 master_server.go:192] adminScripts:                                                                                                     
  lock                                                                                                                                                       
  volume.balance -force                                                                                                                                      
  volume.fix.replication                                                                                                                                     
  unlock                                                                                                                                                     
I1115 10:31:22 76619 master.go:122] Start Seaweed Master 8000GB 2.10  at 0.0.0.0:9333                                                                        
I1115 10:31:22 76619 raft_server.go:70] Starting RaftServer with 192.168.0.204:9333                                                                          
I1115 10:31:22 76619 raft_server.go:129] current cluster leader:                                                                                             
I1115 10:31:22 76619 master.go:146] Start Seaweed Master 8000GB 2.10  grpc server at 0.0.0.0:19333                                                           
I1115 10:31:23 76619 masterclient.go:74] existing leader is 192.168.0.3:9333                                                                                 
I1115 10:31:41 76619 masterclient.go:120] master masterClient failed to receive from 192.168.0.204:9333: EOF                                                 
I1115 10:31:41 76619 masterclient.go:120] adminShell masterClient failed to receive from 192.168.0.204:9333: EOF                                             
2020-11-15 10:31:49.426416 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 5ms                                       
2020-11-15 10:31:49.431561 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 10ms                                      
2020-11-15 10:31:49.441789 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 20ms                                      
2020-11-15 10:31:49.461925 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 40ms                                      
2020-11-15 10:31:49.502054 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 80ms                                      
2020-11-15 10:31:49.582186 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 160ms
2020-11-15 10:31:49.742291 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 320ms
2020-11-15 10:31:50.062433 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 640ms
2020-11-15 10:31:50.702589 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:51.702726 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:52.702802 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:53.702942 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:54.703096 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:55.703252 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:56.703346 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:57.704009 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:58.704168 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:31:59.704321 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
I1115 10:32:00 76619 masterclient.go:120] adminShell masterClient failed to receive from 192.168.0.204:9333: EOF
2020-11-15 10:32:00.704509 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:01.704658 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:02.704792 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:03.704949 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:04.705086 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:05.705232 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:06.705346 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:07.705486 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:08.705621 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:09.705813 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:10.705973 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:11.706270 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:12.706451 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:13.706619 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:14.706780 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:15.706941 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:16.707079 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:17.707296 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
2020-11-15 10:32:18.707450 I | http: Accept error: accept tcp [::]:9333: accept4: too many open files; retrying in 1s
I1115 10:32:19 76619 masterclient.go:120] adminShell masterClient failed to receive from 192.168.0.204:9333: EOF

To reproduce refresh http://localhost:9333/ui/index.html few times in browser.
chrislusf added a commit that referenced this issue Nov 15, 2020
@chrislusf
raft: grpc wait for ready
7cfa616 
related to #1618

https://github.com/grpc/grpc/blob/master/doc/wait-for-ready.md

seaweedfs/raft@7c64624
@chrislusf
Copy link
Collaborator

I tried but could not reproduce it. Is it on some docker/k8s? If so, reproducing could be easier.

Added a possible fix.

Reference:

@onlyjob
Copy link
Author

onlyjob commented Nov 15, 2020

Just a normal Debian system x86-64. No Docker, no k8s... Binary built from source.

Thanks for quick fix. I will test it when a new release will be made.

@LazyDBA247-Anyvision
Copy link
Contributor

@onlyjob did 2.13 solved it for you?
if not, what are your Hard limits?
ulimit -aH

@onlyjob
Copy link
Author

onlyjob commented Dec 9, 2020

I have not seen this issue on 2.14.
Hard limit for nofiles is 1048576.
Thanks.

@onlyjob onlyjob closed this as completed Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@onlyjob @chrislusf @LazyDBA247-Anyvision