Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Weedfs won't work without any reason #2146

Closed
hamidreza-hosseini opened this issue Jun 22, 2021 · 9 comments
Closed

Weedfs won't work without any reason #2146

hamidreza-hosseini opened this issue Jun 22, 2021 · 9 comments

Comments

@hamidreza-hosseini
Copy link

hamidreza-hosseini commented Jun 22, 2021 •
edited

Hi,
Suddenly today I dont know , my weedfs cluster became unhealthy,
There is some Errors that I would attach but I dont know how should I fix this issue.
I would appreciate if you could help me to solve the problem: @chrislusf
I have restarted master but it didn't have any effect
weed version:
version 8000GB 2.40 e983f91 linux amd64

Jun 22 08:53:51 weed-master-1 seaweedfs-master[21217]: I0622 08:53:51 21217 common.go:53] response method:GET URL:/dir/lookup?volumeId=286 with httpStatus:404 and JSON:{"volumeId":"286","error":"volume id 286 not found"} Jun 22 08:53:51 weed-master-1 seaweedfs-master[21217]: I0622 08:53:51 21217 common.go:53] response method:GET URL:/dir/lookup?volumeId=926 with httpStatus:404 and JSON:{"volumeId":"926","error":"volume id 926 not found"} Jun 22 08:53:51 weed-master-1 seaweedfs-master[21217]: I0622 08:53:51 21217 common.go:53] response method:GET URL:/dir/lookup?volumeId=22 with httpStatus:404 and JSON:{"volumeId":"22","error":"volume id 22 not found"}
@ijunaid8989
Copy link

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.

how did you start your weedfs instance?

@hamidreza-hosseini
Copy link
Author

hamidreza-hosseini commented Jun 22, 2021 •
edited

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.

how did you start your weedfs instance?

They were working correctly for 3 months, today suddenly weed baceame unhealthy,
and even I cant see weed-ui

@ijunaid8989
Copy link

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.
how did you start your weedfs instance?

They were working correctly for 3 months, today suddenly weed baceame unhealthy,
and even I cant see weed-ui

Its okay don't worry be patient.

how did you start your seaweedfs instance? the command?

are you in docker? are you in systemctl job?

can you see the logs?

@chrislusf
Copy link
Collaborator

restart the volume server(s)?

@hamidreza-hosseini
Copy link
Author

@chrislusf

These are my log:
Master continuously changed over and over till I restart all masters and then weed cluster became healthy,
I think there is a bug and problem with master election in weed:

Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:25:49 weed-master-1 seaweedfs-master[1157]: I0622 06:25:49 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 Jun 22 06:25:51 weed-master-1 seaweedfs-master[1157]: I0622 06:25:51 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 Jun 22 06:25:52 weed-master-1 seaweedfs-master[1157]: I0622 06:25:52 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:249] + client master@[::1]:5294 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 master_grpc_server.go:265] - client master@[::1]:5294 Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026

and

`
Jun 22 06:29:17 weed-master-1 seaweedfs-master[1157]: I0622 06:29:17 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400
Jun 22 06:29:48 weed-master-1 seaweedfs-master[1157]: I0622 06:29:48 1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333

`

and some part of logs said: response method:GET :

`
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:265] - client master@192.168.200.100:50960
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:249] + client master@192.168.200.100:50960
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=830 with httpStatus:404 and JSON:{"volumeId":"830","error":"volume id 830 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=21 with httpStatus:404 and JSON:{"volumeId":"21","error":"volume id 21 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=867 with httpStatus:404 and JSON:{"volumeId":"867","error":"volume id 867 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=869 with httpStatus:404 and JSON:{"volumeId":"869","error":"volume id 869 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=927 with httpStatus:404 and JSON:{"volumeId":"927","error":"volume id 927 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=35 with httpStatus:404 and JSON:{"volumeId":"35","error":"volume id 35 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=740 with httpStatus:404 and JSON:{"volumeId":"740","error":"volume id 740 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=84 with httpStatus:404 and JSON:{"volumeId":"84","error":"volume id 84 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=237 with httpStatus:404 and JSON:{"volumeId":"237","error":"volume id 237 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=562 with httpStatus:404 and JSON:{"volumeId":"562","error":"volume id 562 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=20 with httpStatus:404 and JSON:{"volumeId":"20","error":"volume id 20 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=80 with httpStatus:404 and JSON:{"volumeId":"80","error":"volume id 80 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=956 with httpStatus:404 and JSON:{"volumeId":"956","error":"volume id 956 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=236 with httpStatus:404 and JSON:{"volumeId":"236","error":"volume id 236 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=371 with httpStatus:404 and JSON:{"volumeId":"371","error":"volume id 371 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}

`

@hamidreza-hosseini
Copy link
Author

restart the volume server(s)?

No I restart all Masters

@hamidreza-hosseini
Copy link
Author

hamidreza-hosseini commented Jun 22, 2021 •
edited

{"volumeId":"286","error":"volume id 286 not found"}

seems like the volume file is not present on the filer directory.
how did you start your weedfs instance?

They were working correctly for 3 months, today suddenly weed baceame unhealthy,
and even I cant see weed-ui

Its okay don't worry be patient.

how did you start your seaweedfs instance? the command?

are you in docker? are you in systemctl job?

can you see the logs?

I'm using systemctl job

Command to run weed cluster in systemd unit file:
ExecStart=/usr/local/bin/weed master -mdir="/etc/master" -ip=weed-master-1.company.local -port=9333 -peers="weed-master-1.company.local:9333,weed-master-2.company.local:9333,weed-master-3.company.local:9333" -defaultReplication=010 -volumeSizeLimitMB=102400 -metrics.address=weed-monitoring:9091

weed version:
version 8000GB 2.40 e983f91 linux amd64

@hamidreza-hosseini
Copy link
Author

hamidreza-hosseini commented Jun 22, 2021 •
edited by chrislusf

@chrislusf

These are my log:
Master continuously changed over and over till I restart all masters and then weed cluster became healthy,
I think there is a bug and problem with master election in weed:

Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 
Jun 22 06:25:02 weed-master-1 seaweedfs-master[1157]: I0622 06:25:02 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 
Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 
Jun 22 06:25:20 weed-master-1 seaweedfs-master[1157]: I0622 06:25:20 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. 
Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 master_grpc_server.go:265] - client master@[::1]:5294 
Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 
Jun 22 06:25:22 weed-master-1 seaweedfs-master[1157]: I0622 06:25:22 1157 master_grpc_server.go:249] + client master@[::1]:5294 
Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 
Jun 22 06:25:23 weed-master-1 seaweedfs-master[1157]: I0622 06:25:23 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 
Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 
Jun 22 06:25:26 weed-master-1 seaweedfs-master[1157]: I0622 06:25:26 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 
Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 
Jun 22 06:25:48 weed-master-1 seaweedfs-master[1157]: I0622 06:25:48 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. 
Jun 22 06:25:49 weed-master-1 seaweedfs-master[1157]: I0622 06:25:49 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 
Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:265] - client master@[::1]:5294 
Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF 
Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 
Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:249] + client master@[::1]:5294 
Jun 22 06:25:50 weed-master-1 seaweedfs-master[1157]: I0622 06:25:50 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 
Jun 22 06:25:51 weed-master-1 seaweedfs-master[1157]: I0622 06:25:51 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 
Jun 22 06:25:52 weed-master-1 seaweedfs-master[1157]: I0622 06:25:52 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 
Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 
Jun 22 06:26:17 weed-master-1 seaweedfs-master[1157]: I0622 06:26:17 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. 
Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:265] - client master@[::1]:5294 
Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF 
Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 
Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:249] + client master@[::1]:5294 
Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 
Jun 22 06:26:18 weed-master-1 seaweedfs-master[1157]: I0622 06:26:18 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 
Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 
Jun 22 06:26:20 weed-master-1 seaweedfs-master[1157]: I0622 06:26:20 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 
Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 
Jun 22 06:26:38 weed-master-1 seaweedfs-master[1157]: I0622 06:26:38 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. 
Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:265] - client master@[::1]:5294 
Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333 
Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:249] + client master@[::1]:5294 
Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026 
Jun 22 06:26:39 weed-master-1 seaweedfs-master[1157]: I0622 06:26:39 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026 
Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400 
Jun 22 06:26:41 weed-master-1 seaweedfs-master[1157]: I0622 06:26:41 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400 
Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333 
Jun 22 06:27:06 weed-master-1 seaweedfs-master[1157]: I0622 06:27:06 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader. 
Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 master_grpc_server.go:265] - client master@[::1]:5294 
Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF 
Jun 22 06:27:07 weed-master-1 seaweedfs-master[1157]: I0622 06:27:07 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026

and

`
Jun 22 06:29:17 weed-master-1 seaweedfs-master[1157]: I0622 06:29:17 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400
Jun 22 06:29:19 weed-master-1 seaweedfs-master[1157]: I0622 06:29:19 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 06:29:24 weed-master-1 seaweedfs-master[1157]: I0622 06:29:24 1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333
Jun 22 06:29:42 weed-master-1 seaweedfs-master[1157]: I0622 06:29:42 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45 1157 master_grpc_server.go:265] - client master@192.168.200.99:46026
Jun 22 06:29:45 weed-master-1 seaweedfs-master[1157]: I0622 06:29:45 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400
Jun 22 06:29:47 weed-master-1 seaweedfs-master[1157]: I0622 06:29:47 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400
Jun 22 06:29:48 weed-master-1 seaweedfs-master[1157]: I0622 06:29:48 1157 master_grpc_server.go:249] + client master@[::1]:5294
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06 1157 master_server.go:141] leader change event: => weed-master-1.company.local:9333
Jun 22 06:30:06 weed-master-1 seaweedfs-master[1157]: I0622 06:30:06 1157 master_server.go:143] [ weed-master-1.company.local:9333 ] weed-master-1.company.local:9333 becomes leader.
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10 1157 master_grpc_server.go:265] - client master@192.168.200.100:50400
Jun 22 06:30:10 weed-master-1 seaweedfs-master[1157]: I0622 06:30:10 1157 master_grpc_server.go:249] + client master@192.168.200.100:50400
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 06:30:11 weed-master-1 seaweedfs-master[1157]: I0622 06:30:11 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333

`

and some part of logs said: response method:GET :

`
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:249] + client master@192.168.200.99:46026
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:265] - client master@192.168.200.100:50960
Jun 22 08:40:42 weed-master-1 seaweedfs-master[1157]: I0622 08:40:42 1157 master_grpc_server.go:249] + client master@192.168.200.100:50960
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 master_grpc_server.go:265] - client master@[::1]:5294
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 masterclient.go:120] master masterClient failed to receive from weed-master-1.company.local:9333: EOF
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 masterclient.go:126] redirected to leader weed-master-1.company.local:9333
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=830 with httpStatus:404 and JSON:{"volumeId":"830","error":"volume id 830 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=21 with httpStatus:404 and JSON:{"volumeId":"21","error":"volume id 21 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=867 with httpStatus:404 and JSON:{"volumeId":"867","error":"volume id 867 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=869 with httpStatus:404 and JSON:{"volumeId":"869","error":"volume id 869 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=12 with httpStatus:404 and JSON:{"volumeId":"12","error":"volume id 12 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=927 with httpStatus:404 and JSON:{"volumeId":"927","error":"volume id 927 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=35 with httpStatus:404 and JSON:{"volumeId":"35","error":"volume id 35 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=740 with httpStatus:404 and JSON:{"volumeId":"740","error":"volume id 740 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=84 with httpStatus:404 and JSON:{"volumeId":"84","error":"volume id 84 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=237 with httpStatus:404 and JSON:{"volumeId":"237","error":"volume id 237 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=562 with httpStatus:404 and JSON:{"volumeId":"562","error":"volume id 562 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=20 with httpStatus:404 and JSON:{"volumeId":"20","error":"volume id 20 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=80 with httpStatus:404 and JSON:{"volumeId":"80","error":"volume id 80 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=956 with httpStatus:404 and JSON:{"volumeId":"956","error":"volume id 956 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=19 with httpStatus:404 and JSON:{"volumeId":"19","error":"volume id 19 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=236 with httpStatus:404 and JSON:{"volumeId":"236","error":"volume id 236 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=371 with httpStatus:404 and JSON:{"volumeId":"371","error":"volume id 371 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=960 with httpStatus:404 and JSON:{"volumeId":"960","error":"volume id 960 not found"}
Jun 22 08:41:04 weed-master-1 seaweedfs-master[1157]: I0622 08:41:04 1157 common.go:53] response method:GET URL:/dir/lookup?volumeId=328 with httpStatus:404 and JSON:{"volumeId":"328","error":"volume id 328 not found"}

`

These are just part of my weed cluster logs , because they were repetitive so I just copied the main part

@chrislusf
Copy link
Collaborator

Added a possible fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@chrislusf @ijunaid8989 @hamidreza-hosseini