Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WeedFS can't start normally caused by Raft Server fault. #1118

Closed
TsingJyujing opened this issue Nov 14, 2019 · 15 comments
Closed

WeedFS can't start normally caused by Raft Server fault. #1118

TsingJyujing opened this issue Nov 14, 2019 · 15 comments

Comments

@TsingJyujing
Copy link

Describe the bug
Some time server cannot start normally because of raft server may down?

System Setup
I started this server with docker-compose

version: '3'

# weed/redis/...'s exposed port is only for debug

services:
  master:
    image: chrislusf/seaweedfs
    ports:
      - 9333:9333 # todo remove this port expose
    command: "master -mdir /data"
    volumes:
      - "./data/meta/:/data/"
  volume:
    image: chrislusf/seaweedfs
    ports:
      - 9301:9301 # todo remove this port expose
    command: 'volume -max=30 -mserver="master:9333" -port=9301'
    volumes:
      - "./data/volume/:/data/"
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs
    ports:
      - 9401:9401 # todo remove this port expose
    command: 'filer -master="master:9333" -port=9401 -dirListLimit 1000000'
    volumes:
      - "./data/filer/:/data/filerldb2/"
    depends_on:
      - master
      - volume
  redis:
    image: redis:5.0
    ports:
      - 6379:6379 # todo remove this port expose

I'm using default filer.toml, but I think this have no relationship with filer.

Expected behavior
Start normally

Screenshots

These are logs repeately printed:

.....
volume_1  | I1114 08:21:11     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
master_1  | I1114 08:21:11     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:72] added volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 1 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 2 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 6 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 3 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 5 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 7 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:102] master see new volume 4 from 172.22.0.4:9301
master_1  | I1114 08:21:11     1 master_grpc_server.go:24] unregister disconnected volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1  | I1114 08:21:11     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.22.0.4:9301
master_1  | I1114 08:21:11     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
volume_1  | I1114 08:21:11     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
master_1  | I1114 08:21:11     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1  | I1114 08:21:11     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.22.0.4:9301
master_1  | E1114 08:21:12     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:12     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:12     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:12     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1114 08:21:12     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:12     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:13     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:13     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:13     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:13     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1114 08:21:13     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:13     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:14     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:14     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:14     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:14     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1114 08:21:14     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:14     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:15     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1114 08:21:15     1 masterclient.go:88] master failed to receive from 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1114 08:21:15     1 masterclient.go:116] master failed to connect with master 172.22.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:15     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1114 08:21:15     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1114 08:21:15     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
......

Additional context
None for now
@chrislusf
Copy link
Collaborator

chrislusf commented Nov 14, 2019 •
edited

Has it ever worked?

Need to expose gRPC ports also.

@TsingJyujing
Copy link
Author

@chrislusf

Thank you for your reply.

After I restart the cluster, it recovered, but sometime this bug will appear and very annoying.
I don't think it's the problem of gRPC, because in docker-compose it will sharing same network environment and can connect each other freely.

(Or I have to specify gRPC port in command?)

@xiaoweihong
Copy link

I use same docker-compose file but add grpc port ,when restart the cluster,same error

master_1  | I1115 16:48:02     1 master_grpc_server.go:72] added volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1  | I1115 16:48:02     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9301
master_1  | I1115 16:48:02     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1  | I1115 16:48:02     1 volume_layout.go:237] Volume 7 becomes unwritable

@chrislusf
Copy link
Collaborator

works for me. the errors should be transient.

tmp$ docker-compose -f 1118.yaml -p seaweedfs up
ERROR: Couldn't connect to Docker daemon. You might need to start Docker for Mac.
tmp$ docker-compose -f 1118.yaml -p seaweedfs up
Creating network "seaweedfs_default" with the default driver
Pulling redis (redis:5.0)...
5.0: Pulling from library/redis
8d691f585fa8: Pull complete
8ccd02d17190: Pull complete
4719eb1815f2: Pull complete
200531706a7d: Pull complete
eed7c26916cf: Pull complete
e1285fcc6a46: Pull complete
Digest: sha256:fe80393a67c7058590ca6b6903f64e35b50fa411b0496f604a85c526fb5bd2d2
Status: Downloaded newer image for redis:5.0
Creating seaweedfs_redis_1  ... done
Creating seaweedfs_master_1 ... done
Creating seaweedfs_volume_1 ... done
Creating seaweedfs_filer_1  ... done
Attaching to seaweedfs_redis_1, seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_filer_1
master_1  | I1115 17:41:53     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1  | I1115 17:41:53     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1   | 1:C 15 Nov 2019 17:41:52.883 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1   | 1:C 15 Nov 2019 17:41:52.883 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1   | 1:C 15 Nov 2019 17:41:52.883 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
master_1  | I1115 17:41:53     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1   | 1:M 15 Nov 2019 17:41:52.884 * Running mode=standalone, port=6379.
redis_1   | 1:M 15 Nov 2019 17:41:52.884 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1   | 1:M 15 Nov 2019 17:41:52.884 # Server initialized
redis_1   | 1:M 15 Nov 2019 17:41:52.884 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1   | 1:M 15 Nov 2019 17:41:52.884 * Ready to accept connections
master_1  | I1115 17:41:53     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1  | I1115 17:41:53     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1  | I1115 17:41:53     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1  | I1115 17:41:53     1 master_server.go:180] adminScripts:
volume_1  | I1115 17:41:53     1 disk_location.go:109] Store started on dir: /data with 0 volumes max 30
volume_1  | I1115 17:41:53     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
volume_1  | I1115 17:41:53     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9301
master_1  | I1115 17:41:53     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1  | I1115 17:41:53     1 raft_server.go:44] Starting RaftServer with 172.18.0.3:9333
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
master_1  | I1115 17:41:53     1 raft_server.go:48] Peers Change: [] => [172.18.0.3:9333]
master_1  | I1115 17:41:53     1 raft_server.go:71] Initializing new cluster
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.3:9333 instead of master:9333
master_1  | I1115 17:41:53     1 raft_server.go:84] current cluster leader: 172.18.0.3:9333
master_1  | I1115 17:41:53     1 master_server.go:134] [ 172.18.0.3:9333 ] I am the leader!
master_1  | I1115 17:41:53     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.3:9333
master_1  | I1115 17:41:53     1 master_grpc_server.go:196] + client master172.18.0.3:58202
master_1  | I1115 17:41:53     1 node.go:241] topo adds child DefaultDataCenter
master_1  | I1115 17:41:53     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1  | I1115 17:41:53     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:41:53     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:41:53     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:41:53     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | I1115 17:41:53     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:41:53     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
filer_1   | I1115 17:41:54     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1   | I1115 17:41:54     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1   | I1115 17:41:54     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1   | I1115 17:41:54     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
master_1  | I1115 17:41:54     1 master_grpc_server.go:196] + client filer172.18.0.5:50684
filer_1   | I1115 17:41:54     1 configuration.go:26] Configure filer for leveldb2
filer_1   | I1115 17:41:54     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :9401
^CGracefully stopping... (press Ctrl+C again to force)
Stopping seaweedfs_filer_1  ... done
Stopping seaweedfs_volume_1 ... done
Stopping seaweedfs_redis_1  ... done
Stopping seaweedfs_master_1 ... done
tmp$ docker-compose -f 1118.yaml -p seaweedfs up
Starting seaweedfs_master_1 ... done
Starting seaweedfs_redis_1  ... done
Starting seaweedfs_volume_1 ... done
Starting seaweedfs_filer_1  ... done
Attaching to seaweedfs_redis_1, seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_filer_1
redis_1   | 1:C 15 Nov 2019 17:42:56.320 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1   | 1:C 15 Nov 2019 17:42:56.320 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1   | 1:C 15 Nov 2019 17:42:56.320 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
master_1  | I1115 17:42:56     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1  | I1115 17:42:56     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1   | 1:M 15 Nov 2019 17:42:56.321 * Running mode=standalone, port=6379.
redis_1   | 1:M 15 Nov 2019 17:42:56.321 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1   | 1:M 15 Nov 2019 17:42:56.321 # Server initialized
redis_1   | 1:M 15 Nov 2019 17:42:56.321 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1   | 1:M 15 Nov 2019 17:42:56.321 * DB loaded from disk: 0.000 seconds
redis_1   | 1:M 15 Nov 2019 17:42:56.321 * Ready to accept connections
master_1  | I1115 17:42:56     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1  | I1115 17:42:56     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1  | I1115 17:42:56     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1  | I1115 17:42:56     1 disk_location.go:109] Store started on dir: /data with 0 volumes max 30
master_1  | I1115 17:42:56     1 master_server.go:93] Volume Size Limit is 30000 MB
volume_1  | I1115 17:42:56     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
volume_1  | I1115 17:42:56     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
master_1  | I1115 17:42:56     1 master_server.go:180] adminScripts:
master_1  | I1115 17:42:56     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1  | I1115 17:42:56     1 raft_server.go:44] Starting RaftServer with 172.18.0.3:9333
volume_1  | I1115 17:42:56     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9301
volume_1  | I1115 17:42:56     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
master_1  | I1115 17:42:56     1 raft_server.go:84] current cluster leader:
master_1  | I1115 17:42:56     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1  | E1115 17:42:56     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
volume_1  | I1115 17:42:56     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
master_1  | I1115 17:42:56     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:56     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:56     1 node.go:241] topo adds child DefaultDataCenter
master_1  | I1115 17:42:56     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1  | I1115 17:42:56     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:42:56     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:42:56     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:42:56     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | E1115 17:42:57     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:57     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:57     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1115 17:42:57     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1   | I1115 17:42:57     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1   | I1115 17:42:57     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1   | I1115 17:42:57     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
master_1  | E1115 17:42:57     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1115 17:42:57     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1115 17:42:57     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1115 17:42:57     1 configuration.go:26] Configure filer for leveldb2
filer_1   | I1115 17:42:57     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :9401
master_1  | E1115 17:42:58     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:58     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:58     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:42:58     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1115 17:42:58     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1115 17:42:58     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:42:59     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:59     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:59     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:42:59     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1115 17:42:59     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1115 17:42:59     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:43:00     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:43:00     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:43:00     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:43:00     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1   | I1115 17:43:00     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1   | I1115 17:43:00     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:43:00     1 master_server.go:131] state change: &{typ:stateChange source:0xc0001f8ea0 value:candidate prevValue:follower}
master_1  | I1115 17:43:00     1 master_server.go:131] state change: &{typ:stateChange source:0xc0001f8ea0 value:leader prevValue:candidate}
master_1  | I1115 17:43:00     1 master_server.go:125] event: &{typ:leaderChange source:0xc0001f8ea0 value:172.18.0.3:9333 prevValue:}
master_1  | I1115 17:43:00     1 master_server.go:127] [ 172.18.0.3:9333 ] 172.18.0.3:9333 becomes leader.
master_1  | I1115 17:43:01     1 master_grpc_server.go:196] + client master172.18.0.3:58230
master_1  | I1115 17:43:01     1 master_grpc_server.go:196] + client filer172.18.0.5:50698
volume_1  | I1115 17:43:01     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
master_1  | I1115 17:43:01     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:43:01     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
volume_1  | I1115 17:43:01     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.3:9333 instead of master:9333
master_1  | I1115 17:43:01     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:43:01     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | I1115 17:43:01     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:43:01     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
volume_1  | I1115 17:43:01     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.3:9333








volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:1 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/1.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 1
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 1 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:1
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:1
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:2 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/2.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 2
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 2 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:2
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:2
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:3 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/3.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 3
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 3 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:3
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:3
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:4 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/4.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 4
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 4 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:4
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:4
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:5 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/5.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 5
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 5 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:5
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:5
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:6 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/6.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 6
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:6
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:6
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 6 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:7 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/7.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 7
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 7 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:7
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:7
^CGracefully stopping... (press Ctrl+C again to force)
Stopping seaweedfs_filer_1  ...
Stopping seaweedfs_volume_1 ...
Stopping seaweedfs_redis_1  ...
Stopping seaweedfs_master_1 ...
^C^CERROR: Aborting.
tmp$
tmp$
tmp$
tmp$
tmp$
tmp$
tmp$
tmp$
tmp$
tmp$ docker-compose -f 1118.yaml -p seaweedfs up
seaweedfs_master_1 is up-to-date
Starting seaweedfs_redis_1 ...
Starting seaweedfs_redis_1 ... done
Starting seaweedfs_filer_1 ... done
Attaching to seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_filer_1, seaweedfs_redis_1
filer_1   | I1115 17:44:13     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1   | 1:C 15 Nov 2019 17:44:13.112 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1   | 1:C 15 Nov 2019 17:44:13.112 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1   | 1:C 15 Nov 2019 17:44:13.112 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1   | 1:M 15 Nov 2019 17:44:13.114 * Running mode=standalone, port=6379.
redis_1   | 1:M 15 Nov 2019 17:44:13.114 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
filer_1   | I1115 17:44:13     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1   | I1115 17:44:13     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1   | I1115 17:44:13     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
redis_1   | 1:M 15 Nov 2019 17:44:13.114 # Server initialized
redis_1   | 1:M 15 Nov 2019 17:44:13.115 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1   | 1:M 15 Nov 2019 17:44:13.115 * DB loaded from disk: 0.000 seconds
redis_1   | 1:M 15 Nov 2019 17:44:13.115 * Ready to accept connections
master_1  | I1115 17:41:53     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1  | I1115 17:41:53     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1  | I1115 17:41:53     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1  | I1115 17:41:53     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1  | I1115 17:41:53     1 master_server.go:180] adminScripts:
master_1  | I1115 17:41:53     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1  | I1115 17:41:53     1 raft_server.go:44] Starting RaftServer with 172.18.0.3:9333
master_1  | I1115 17:41:53     1 raft_server.go:48] Peers Change: [] => [172.18.0.3:9333]
master_1  | I1115 17:41:53     1 raft_server.go:71] Initializing new cluster
master_1  | I1115 17:41:53     1 raft_server.go:84] current cluster leader: 172.18.0.3:9333
master_1  | I1115 17:41:53     1 master_server.go:134] [ 172.18.0.3:9333 ] I am the leader!
master_1  | I1115 17:41:53     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1  | I1115 17:41:53     1 master_grpc_server.go:196] + client master172.18.0.3:58202
master_1  | I1115 17:41:53     1 node.go:241] topo adds child DefaultDataCenter
master_1  | I1115 17:41:53     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1  | I1115 17:41:53     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:41:53     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:41:53     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:41:53     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | I1115 17:41:53     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:41:53     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:41:54     1 master_grpc_server.go:196] + client filer172.18.0.5:50684
master_1  | I1115 17:42:50     1 master_grpc_server.go:206] - client filer172.18.0.5:50684
master_1  | I1115 17:42:50     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:42:50     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | I1115 17:42:56     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1  | I1115 17:42:56     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1  | I1115 17:42:56     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1  | I1115 17:42:56     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1  | I1115 17:42:56     1 master_server.go:180] adminScripts:
master_1  | I1115 17:42:56     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1  | I1115 17:42:56     1 raft_server.go:44] Starting RaftServer with 172.18.0.3:9333
master_1  | I1115 17:42:56     1 raft_server.go:84] current cluster leader:
master_1  | I1115 17:42:56     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1  | E1115 17:42:56     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:56     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:56     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:56     1 node.go:241] topo adds child DefaultDataCenter
master_1  | I1115 17:42:56     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1  | I1115 17:42:56     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:42:56     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:42:56     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:42:56     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | E1115 17:42:57     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:57     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:57     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:42:57     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | E1115 17:42:58     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:58     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:58     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:42:58     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | E1115 17:42:59     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:42:59     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:42:59     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:42:59     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | E1115 17:43:00     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:43:00     1 masterclient.go:88] master failed to receive from 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | I1115 17:43:00     1 masterclient.go:116] master failed to connect with master 172.18.0.3:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1  | E1115 17:43:00     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1  | I1115 17:43:00     1 master_server.go:131] state change: &{typ:stateChange source:0xc0001f8ea0 value:candidate prevValue:follower}
master_1  | I1115 17:43:00     1 master_server.go:131] state change: &{typ:stateChange source:0xc0001f8ea0 value:leader prevValue:candidate}
master_1  | I1115 17:43:00     1 master_server.go:125] event: &{typ:leaderChange source:0xc0001f8ea0 value:172.18.0.3:9333 prevValue:}
master_1  | I1115 17:43:00     1 master_server.go:127] [ 172.18.0.3:9333 ] 172.18.0.3:9333 becomes leader.
master_1  | I1115 17:43:01     1 master_grpc_server.go:196] + client master172.18.0.3:58230
master_1  | I1115 17:43:01     1 master_grpc_server.go:196] + client filer172.18.0.5:50698
master_1  | I1115 17:43:01     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:43:01     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:43:01     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.4:9301
master_1  | I1115 17:43:01     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.4:9301
master_1  | I1115 17:43:01     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.4:9301
master_1  | I1115 17:43:01     1 master_grpc_server.go:72] added volume server 172.18.0.4:9301
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 1 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:1
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:1
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 2 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:2
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:2
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 3 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:3
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:3
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 4 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:4
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:4
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 5 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:5
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:5
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:6
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:6
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 6 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 volume_growth.go:211] Created Volume 7 on topo:DefaultDataCenter:DefaultRack:172.18.0.4:9301
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to master172.18.0.3:58230: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:7
master_1  | I1115 17:44:00     1 master_grpc_server.go:148] master send to filer172.18.0.5:50698: url:"172.18.0.4:9301" public_url:"172.18.0.4:9301" new_vids:7
master_1  | I1115 17:44:07     1 master_grpc_server.go:206] - client filer172.18.0.5:50698
master_1  | I1115 17:44:13     1 master_grpc_server.go:196] + client filer172.18.0.5:50726
volume_1  | I1115 17:41:53     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1  | I1115 17:41:53     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1  | I1115 17:41:53     1 disk_location.go:109] Store started on dir: /data with 0 volumes max 30
volume_1  | I1115 17:41:53     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
volume_1  | I1115 17:41:53     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9301
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.3:9333 instead of master:9333
volume_1  | I1115 17:41:53     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.3:9333
volume_1  | I1115 17:42:50     1 volume_server.go:98] Shutting down volume server...
volume_1  | I1115 17:42:50     1 volume_server.go:100] Shut down successfully!
volume_1  | I1115 17:42:56     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1  | I1115 17:42:56     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1  | I1115 17:42:56     1 disk_location.go:109] Store started on dir: /data with 0 volumes max 30
volume_1  | I1115 17:42:56     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
volume_1  | I1115 17:42:56     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
volume_1  | I1115 17:42:56     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9301
volume_1  | I1115 17:42:56     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1  | I1115 17:42:56     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
volume_1  | I1115 17:43:01     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1  | I1115 17:43:01     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.3:9333 instead of master:9333
volume_1  | I1115 17:43:01     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.3:9333
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:1 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/1.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 1
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:2 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/2.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 2
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:3 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/3.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 3
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:4 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/4.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 4
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:5 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/5.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 5
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:6 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/6.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 6
volume_1  | I1115 17:44:00     1 store.go:109] In dir /data adds volume:7 collection: replicaPlacement:000 ttl:
volume_1  | I1115 17:44:00     1 volume_loading.go:86] loading index /data/7.idx to memory readonly false
volume_1  | I1115 17:44:00     1 store.go:113] add volume 7
filer_1   | I1115 17:44:13     1 configuration.go:26] Configure filer for leveldb2
filer_1   | I1115 17:44:13     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :9401

@xiaoweihong
Copy link

e,that's strange.

my full result

root@ubuntu:~/seaweedfs# docker-compose -f seaweedfs.yml up
Creating seaweedfs_master_1 ... 
Creating seaweedfs_master_1 ... done
Creating seaweedfs_volume_1 ... 
Creating seaweedfs_volume_1 ... done
Creating seaweedfs_filer_1 ... 
Creating seaweedfs_cronjob_1 ... 
Creating seaweedfs_filer_1
Creating seaweedfs_filer_1 ... done
Attaching to seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_cronjob_1, seaweedfs_filer_1
master_1   | I1116 15:56:42     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:56:42     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:56:42     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1   | I1116 15:56:42     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1   | I1116 15:56:42     1 master_server.go:180] adminScripts:
master_1   | I1116 15:56:42     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1   | I1116 15:56:42     1 raft_server.go:44] Starting RaftServer with 172.18.0.2:9333
master_1   | I1116 15:56:42     1 raft_server.go:48] Peers Change: [] => [172.18.0.2:9333]
master_1   | I1116 15:56:42     1 raft_server.go:71] Initializing new cluster
master_1   | I1116 15:56:42     1 raft_server.go:84] current cluster leader: 172.18.0.2:9333
master_1   | I1116 15:56:42     1 master_server.go:134] [ 172.18.0.2:9333 ] I am the leader!
master_1   | I1116 15:56:42     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1   | I1116 15:56:42     1 master_grpc_server.go:196] + client master172.18.0.2:42584
master_1   | I1116 15:56:43     1 node.go:241] topo adds child DefaultDataCenter
master_1   | I1116 15:56:43     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1   | I1116 15:56:43     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1   | I1116 15:56:43     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1   | I1116 15:56:43     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/1.idx to memory readonly false
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/1.dat, replicaPlacement=000 v=3 size=65215832 ttl=
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/2.idx to memory readonly false
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/2.dat, replicaPlacement=000 v=3 size=63397064 ttl=
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/4.idx to memory readonly false
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/4.dat, replicaPlacement=000 v=3 size=70672136 ttl=
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/5.idx to memory readonly false
master_1   | I1116 15:56:43     1 master_grpc_server.go:148] master send to master172.18.0.2:42584: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:3 new_vids:1 new_vids:2 new_vids:4 new_vids:5 new_vids:6 new_vids:7 
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/5.dat, replicaPlacement=000 v=3 size=74569496 ttl=
master_1   | I1116 15:56:43     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/6.idx to memory readonly false
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/6.dat, replicaPlacement=000 v=3 size=61838120 ttl=
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/7.idx to memory readonly false
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/7.dat, replicaPlacement=000 v=3 size=77427560 ttl=
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 7 becomes unwritable
volume_1   | I1116 15:56:43     1 volume_loading.go:86] loading index /data/3.idx to memory readonly false
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
volume_1   | I1116 15:56:43     1 disk_location.go:68] data file /data/3.dat, replicaPlacement=000 v=3 size=73010552 ttl=
cronjob_1  | Running Crontab:
cronjob_1  | */2 * * * * * echo "volume.fix.replication" | weed shell -master=master:9333
cronjob_1  | */2 * * * * * echo "volume.balance -c ALL -force" | weed shell -master=master:9333
volume_1   | I1116 15:56:43     1 disk_location.go:109] Store started on dir: /data with 7 volumes max 15
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 3 becomes unwritable
cronjob_1  | time="2019-11-16T07:56:44Z" level=info msg="read crontab: /crontab"
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
volume_1   | I1116 15:56:43     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9300
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 1 becomes unwritable
volume_1   | I1116 15:56:43     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
volume_1   | I1116 15:56:43     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1   | I1116 15:56:43     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.2:9333 instead of master:9333
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 2 becomes unwritable
volume_1   | I1116 15:56:43     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.2:9333
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:56:43     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:43     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:56:43     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:56:43     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
master_1   | I1116 15:56:43     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
filer_1    | I1116 15:56:44     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:56:44     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:56:44     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1    | I1116 15:56:44     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:56:43     1 master_grpc_server.go:148] master send to master172.18.0.2:42584: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:3 new_vids:1 new_vids:2 new_vids:4 new_vids:5 new_vids:6 new_vids:7 
master_1   | I1116 15:56:44     1 master_grpc_server.go:196] + client filer172.18.0.4:43850
filer_1    | I1116 15:56:44     1 configuration.go:26] Configure filer for leveldb2
filer_1    | I1116 15:56:44     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :8888
^CGracefully stopping... (press Ctrl+C again to force)
Stopping seaweedfs_cronjob_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
root@ubuntu:~/seaweedfs# docker-compose -f seaweedfs.yml up
Starting seaweedfs_master_1 ... 
Starting seaweedfs_master_1 ... done
Starting seaweedfs_volume_1 ... 
Starting seaweedfs_volume_1 ... done
Starting seaweedfs_filer_1 ... 
Starting seaweedfs_cronjob_1 ... 
Starting seaweedfs_filer_1
Starting seaweedfs_cronjob_1 ... done
Attaching to seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_filer_1, seaweedfs_cronjob_1
master_1   | I1116 15:56:53     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:56:53     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:56:53     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1   | I1116 15:56:53     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1   | I1116 15:56:53     1 master_server.go:180] adminScripts:
master_1   | I1116 15:56:53     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1   | I1116 15:56:53     1 raft_server.go:44] Starting RaftServer with 172.18.0.2:9333
master_1   | I1116 15:56:53     1 raft_server.go:84] current cluster leader: 
master_1   | I1116 15:56:53     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1   | E1116 15:56:53     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:56:53     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:56:53     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:56:54     1 node.go:241] topo adds child DefaultDataCenter
master_1   | I1116 15:56:54     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1   | I1116 15:56:54     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:56:54     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:56:54     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:54     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:56:54     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:56:54     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
master_1   | E1116 15:56:54     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:56:54     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:56:54     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1   | I1116 15:56:54     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1   | I1116 15:56:54     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/1.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/1.dat, replicaPlacement=000 v=3 size=65215832 ttl=
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/2.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/2.dat, replicaPlacement=000 v=3 size=63397064 ttl=
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/4.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/4.dat, replicaPlacement=000 v=3 size=70672136 ttl=
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/5.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/5.dat, replicaPlacement=000 v=3 size=74569496 ttl=
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/6.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/6.dat, replicaPlacement=000 v=3 size=61838120 ttl=
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/7.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/7.dat, replicaPlacement=000 v=3 size=77427560 ttl=
volume_1   | I1116 15:56:54     1 volume_loading.go:86] loading index /data/3.idx to memory readonly false
volume_1   | I1116 15:56:54     1 disk_location.go:68] data file /data/3.dat, replicaPlacement=000 v=3 size=73010552 ttl=
volume_1   | I1116 15:56:54     1 disk_location.go:109] Store started on dir: /data with 7 volumes max 15
filer_1    | I1116 15:56:55     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:56:55     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:56:55     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1    | I1116 15:56:55     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
volume_1   | I1116 15:56:54     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
volume_1   | I1116 15:56:54     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9300
volume_1   | I1116 15:56:54     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
volume_1   | I1116 15:56:54     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1   | I1116 15:56:54     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
cronjob_1  | Running Crontab:
cronjob_1  | */2 * * * * * echo "volume.fix.replication" | weed shell -master=master:9333
cronjob_1  | */2 * * * * * echo "volume.balance -c ALL -force" | weed shell -master=master:9333
cronjob_1  | time="2019-11-16T07:56:55Z" level=info msg="read crontab: /crontab"
filer_1    | I1116 15:56:55     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:56:55     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | E1116 15:56:55     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1    | I1116 15:56:55     1 configuration.go:26] Configure filer for leveldb2
filer_1    | I1116 15:56:55     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :8888
master_1   | E1116 15:56:55     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:56:55     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:56:55     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | E1116 15:56:56     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1    | I1116 15:56:56     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:56:56     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | E1116 15:56:56     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:56:56     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:56:56     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:56:57     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:56:57     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | E1116 15:56:57     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:56:57     1 master_server.go:131] state change: &{typ:stateChange source:0xc00036d8c0 value:candidate prevValue:follower}
master_1   | I1116 15:56:57     1 master_server.go:131] state change: &{typ:stateChange source:0xc00036d8c0 value:leader prevValue:candidate}
master_1   | I1116 15:56:57     1 master_server.go:125] event: &{typ:leaderChange source:0xc00036d8c0 value:172.18.0.2:9333 prevValue:}
master_1   | I1116 15:56:57     1 master_server.go:127] [ 172.18.0.2:9333 ] 172.18.0.2:9333 becomes leader.
master_1   | I1116 15:56:57     1 master_grpc_server.go:196] + client master172.18.0.2:42606
master_1   | I1116 15:56:58     1 master_grpc_server.go:196] + client filer172.18.0.4:43860
master_1   | I1116 15:56:59     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
volume_1   | I1116 15:56:59     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1   | I1116 15:56:59     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.2:9333 instead of master:9333
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
volume_1   | I1116 15:56:59     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.2:9333
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:148] master send to filer172.18.0.4:43860: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:2 new_vids:4 new_vids:5 new_vids:6 new_vids:7 new_vids:3 new_vids:1 
master_1   | I1116 15:56:59     1 master_grpc_server.go:148] master send to master172.18.0.2:42606: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:2 new_vids:4 new_vids:5 new_vids:6 new_vids:7 new_vids:3 new_vids:1 
master_1   | I1116 15:56:59     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1   | I1116 15:56:59     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:56:59     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:56:59     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1   | I1116 15:56:59     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
^CGracefully stopping... (press Ctrl+C again to force)
Stopping seaweedfs_cronjob_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
root@ubuntu:~/seaweedfs# docker-compose -f seaweedfs.yml up
Starting seaweedfs_master_1 ... 
Starting seaweedfs_master_1 ... done
Starting seaweedfs_volume_1 ... 
Starting seaweedfs_volume_1 ... done
Starting seaweedfs_filer_1 ... 
Starting seaweedfs_cronjob_1 ... 
Starting seaweedfs_filer_1
Starting seaweedfs_cronjob_1 ... done
Attaching to seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_filer_1, seaweedfs_cronjob_1
master_1   | I1116 15:57:04     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:57:04     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:57:04     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1   | I1116 15:57:04     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1   | I1116 15:57:04     1 master_server.go:180] adminScripts:
master_1   | I1116 15:57:04     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1   | I1116 15:57:04     1 raft_server.go:44] Starting RaftServer with 172.18.0.2:9333
master_1   | I1116 15:57:04     1 raft_server.go:84] current cluster leader: 
master_1   | I1116 15:57:04     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1   | E1116 15:57:04     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:04     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:04     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:05     1 node.go:241] topo adds child DefaultDataCenter
master_1   | I1116 15:57:05     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1   | I1116 15:57:05     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:57:05     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:57:05     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:05     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:57:05     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:57:05     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
master_1   | E1116 15:57:05     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:05     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:05     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1   | I1116 15:57:05     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1   | I1116 15:57:05     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/1.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/1.dat, replicaPlacement=000 v=3 size=65215832 ttl=
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/2.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/2.dat, replicaPlacement=000 v=3 size=63397064 ttl=
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/4.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/4.dat, replicaPlacement=000 v=3 size=70672136 ttl=
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/5.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/5.dat, replicaPlacement=000 v=3 size=74569496 ttl=
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/6.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/6.dat, replicaPlacement=000 v=3 size=61838120 ttl=
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/7.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/7.dat, replicaPlacement=000 v=3 size=77427560 ttl=
volume_1   | I1116 15:57:05     1 volume_loading.go:86] loading index /data/3.idx to memory readonly false
volume_1   | I1116 15:57:05     1 disk_location.go:68] data file /data/3.dat, replicaPlacement=000 v=3 size=73010552 ttl=
volume_1   | I1116 15:57:05     1 disk_location.go:109] Store started on dir: /data with 7 volumes max 15
volume_1   | I1116 15:57:05     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
filer_1    | I1116 15:57:06     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:57:06     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:57:06     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1    | I1116 15:57:06     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
volume_1   | I1116 15:57:05     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9300
volume_1   | I1116 15:57:05     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
volume_1   | I1116 15:57:05     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1   | I1116 15:57:05     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
cronjob_1  | Running Crontab:
cronjob_1  | */2 * * * * * echo "volume.fix.replication" | weed shell -master=master:9333
cronjob_1  | */2 * * * * * echo "volume.balance -c ALL -force" | weed shell -master=master:9333
cronjob_1  | time="2019-11-16T07:57:06Z" level=info msg="read crontab: /crontab"
master_1   | E1116 15:57:06     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1    | I1116 15:57:06     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:06     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:06     1 configuration.go:26] Configure filer for leveldb2
filer_1    | I1116 15:57:06     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :8888
master_1   | E1116 15:57:06     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:06     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:06     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:07     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:07     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | E1116 15:57:07     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | E1116 15:57:07     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:07     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:07     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:08     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:08     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | E1116 15:57:08     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | E1116 15:57:08     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:08     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:08     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:09     1 master_server.go:131] state change: &{typ:stateChange source:0xc000500000 value:candidate prevValue:follower}
master_1   | I1116 15:57:09     1 master_server.go:131] state change: &{typ:stateChange source:0xc000500000 value:leader prevValue:candidate}
master_1   | I1116 15:57:09     1 master_server.go:125] event: &{typ:leaderChange source:0xc000500000 value:172.18.0.2:9333 prevValue:}
master_1   | I1116 15:57:09     1 master_server.go:127] [ 172.18.0.2:9333 ] 172.18.0.2:9333 becomes leader.
master_1   | I1116 15:57:09     1 master_grpc_server.go:196] + client filer172.18.0.4:43880
master_1   | I1116 15:57:09     1 master_grpc_server.go:196] + client master172.18.0.2:42628
master_1   | I1116 15:57:10     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
volume_1   | I1116 15:57:10     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1   | I1116 15:57:10     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.2:9333 instead of master:9333
volume_1   | I1116 15:57:10     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.2:9333
master_1   | I1116 15:57:10     1 master_grpc_server.go:148] master send to filer172.18.0.4:43880: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:7 new_vids:3 new_vids:1 new_vids:2 new_vids:4 new_vids:5 new_vids:6 
master_1   | I1116 15:57:10     1 master_grpc_server.go:148] master send to master172.18.0.2:42628: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:7 new_vids:3 new_vids:1 new_vids:2 new_vids:4 new_vids:5 new_vids:6 
master_1   | I1116 15:57:10     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1   | I1116 15:57:10     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:10     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:57:10     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1   | I1116 15:57:10     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
^CGracefully stopping... (press Ctrl+C again to force)
Stopping seaweedfs_cronjob_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
root@ubuntu:~/seaweedfs# docker-compose -f seaweedfs.yml up
Starting seaweedfs_master_1 ... 
Starting seaweedfs_master_1 ... done
Starting seaweedfs_volume_1 ... 
Starting seaweedfs_volume_1 ... done
Starting seaweedfs_filer_1 ... 
Starting seaweedfs_cronjob_1 ... 
Starting seaweedfs_filer_1
Starting seaweedfs_filer_1 ... done
Attaching to seaweedfs_master_1, seaweedfs_volume_1, seaweedfs_cronjob_1, seaweedfs_filer_1
master_1   | I1116 15:57:15     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:57:15     1 config.go:28] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
master_1   | I1116 15:57:15     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
master_1   | I1116 15:57:15     1 master_server.go:93] Volume Size Limit is 30000 MB
master_1   | I1116 15:57:15     1 master_server.go:180] adminScripts:
master_1   | I1116 15:57:15     1 master.go:109] Start Seaweed Master 30GB 1.44 at 0.0.0.0:9333
master_1   | I1116 15:57:15     1 raft_server.go:44] Starting RaftServer with 172.18.0.2:9333
master_1   | I1116 15:57:15     1 raft_server.go:84] current cluster leader: 
master_1   | I1116 15:57:15     1 master.go:133] Start Seaweed Master 30GB 1.44 grpc server at 0.0.0.0:19333
master_1   | E1116 15:57:15     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:15     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:15     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:16     1 node.go:241] topo adds child DefaultDataCenter
master_1   | I1116 15:57:16     1 node.go:241] topo:DefaultDataCenter adds child DefaultRack
master_1   | I1116 15:57:16     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:57:16     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
volume_1   | I1116 15:57:16     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
volume_1   | I1116 15:57:16     1 file_util.go:19] Folder /data Permission: -rwxr-xr-x
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/1.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/1.dat, replicaPlacement=000 v=3 size=65215832 ttl=
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/2.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/2.dat, replicaPlacement=000 v=3 size=63397064 ttl=
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/4.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/4.dat, replicaPlacement=000 v=3 size=70672136 ttl=
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/5.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/5.dat, replicaPlacement=000 v=3 size=74569496 ttl=
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/6.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/6.dat, replicaPlacement=000 v=3 size=61838120 ttl=
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/3.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/3.dat, replicaPlacement=000 v=3 size=73010552 ttl=
volume_1   | I1116 15:57:16     1 volume_loading.go:86] loading index /data/7.idx to memory readonly false
volume_1   | I1116 15:57:16     1 disk_location.go:68] data file /data/7.dat, replicaPlacement=000 v=3 size=77427560 ttl=
volume_1   | I1116 15:57:16     1 disk_location.go:109] Store started on dir: /data with 7 volumes max 15
volume_1   | I1116 15:57:16     1 disk_location.go:112] Store started on dir: /data with 0 ec shards
volume_1   | I1116 15:57:16     1 volume.go:162] Start Seaweed volume server 30GB 1.44 at 0.0.0.0:9300
volume_1   | I1116 15:57:16     1 volume_grpc_client_to_master.go:24] Volume server start with seed master nodes: [master:9333]
volume_1   | I1116 15:57:16     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
volume_1   | I1116 15:57:16     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:57:16     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:16     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1   | I1116 15:57:16     1 volume_layout.go:237] Volume 3 becomes unwritable
cronjob_1  | Running Crontab:
cronjob_1  | */2 * * * * * echo "volume.fix.replication" | weed shell -master=master:9333
cronjob_1  | */2 * * * * * echo "volume.balance -c ALL -force" | weed shell -master=master:9333
master_1   | I1116 15:57:16     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
master_1   | E1116 15:57:16     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:16     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:16     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
cronjob_1  | time="2019-11-16T07:57:17Z" level=info msg="read crontab: /crontab"
filer_1    | I1116 15:57:17     1 config.go:28] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:57:17     1 config.go:28] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
filer_1    | I1116 15:57:17     1 leveldb2_store.go:38] filer store leveldb2 dir: /data/filerldb2
filer_1    | I1116 15:57:17     1 file_util.go:19] Folder /data/filerldb2 Permission: -rwxr-xr-x
master_1   | E1116 15:57:17     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
filer_1    | I1116 15:57:17     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:17     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1    | I1116 15:57:17     1 configuration.go:26] Configure filer for leveldb2
filer_1    | I1116 15:57:17     1 filer.go:133] Start Seaweed Filer 30GB 1.44 at :8888
master_1   | E1116 15:57:17     1 master_grpc_server.go:252] topo leader: Raft Server not initialized!
master_1   | I1116 15:57:17     1 masterclient.go:88] master failed to receive from 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:17     1 masterclient.go:116] master failed to connect with master 172.18.0.2:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1   | I1116 15:57:18     1 master_server.go:131] state change: &{typ:stateChange source:0xc000369d40 value:candidate prevValue:follower}
master_1   | I1116 15:57:18     1 master_server.go:131] state change: &{typ:stateChange source:0xc000369d40 value:leader prevValue:candidate}
master_1   | I1116 15:57:18     1 master_server.go:125] event: &{typ:leaderChange source:0xc000369d40 value:172.18.0.2:9333 prevValue:}
master_1   | I1116 15:57:18     1 master_server.go:127] [ 172.18.0.2:9333 ] 172.18.0.2:9333 becomes leader.
master_1   | I1116 15:57:18     1 master_grpc_server.go:196] + client filer172.18.0.4:43902
master_1   | I1116 15:57:18     1 master_grpc_server.go:196] + client master172.18.0.2:42646
master_1   | I1116 15:57:21     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.18.0.3:9300
volume_1   | I1116 15:57:21     1 volume_grpc_client_to_master.go:68] Heartbeat to: master:9333
master_1   | I1116 15:57:21     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 4 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 5 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 6 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 3 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 7 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 1 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:102] master see new volume 2 from 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:148] master send to master172.18.0.2:42646: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:4 new_vids:5 new_vids:6 new_vids:3 new_vids:7 new_vids:1 new_vids:2 
master_1   | I1116 15:57:21     1 master_grpc_server.go:148] master send to filer172.18.0.4:43902: url:"172.18.0.3:9300" public_url:"192.168.2.111:9300" new_vids:4 new_vids:5 new_vids:6 new_vids:3 new_vids:7 new_vids:1 new_vids:2 
volume_1   | I1116 15:57:21     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.18.0.2:9333 instead of master:9333
volume_1   | I1116 15:57:21     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.18.0.2:9333
master_1   | I1116 15:57:21     1 master_grpc_server.go:72] added volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 master_grpc_server.go:24] unregister disconnected volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 4 becomes unwritable
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 5 becomes unwritable
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 6 becomes unwritable
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 3 becomes unwritable
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 7 becomes unwritable
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 1 becomes unwritable
master_1   | I1116 15:57:21     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.18.0.3:9300
master_1   | I1116 15:57:21     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
master_1   | I1116 15:57:21     1 volume_layout.go:237] Volume 2 becomes unwritable
master_1   | I1116 15:57:21     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.18.0.3:9300
^CGracefully stopping... (press Ctrl+C again to force)
Stopping seaweedfs_cronjob_1 ... done
Stopping seaweedfs_filer_1   ... done
Stopping seaweedfs_volume_1  ... done
Stopping seaweedfs_master_1  ... done
root@ubuntu:~/seaweedfs#

seaweedfs.yml

version: '2'

services:
  master:
    image: chrislusf/seaweedfs # use a remote image
    ports:
    - 9333:9333
    - 19333:19333
    command: "master -mdir /data"
    volumes:
      - /etc/localtime:/etc/localtime
  volume:
    image: chrislusf/seaweedfs # use a remote image
    ports:
    - 9301:9301
    - 19301:19301
    command: 'volume -max=15 -mserver="master:9333" -port=9300 -publicUrl=192.168.2.111:9300'
    volumes:
      - "./data/volume/:/data/"
      - /etc/localtime:/etc/localtime
    depends_on:
    - master
  filer:
    image: chrislusf/seaweedfs # use a remote image
    ports:
    - 8888:8888
    - 18888:18888
    command: 'filer -master="master:9333" -port=8888'
    volumes:
#      - "./data/filer/:/data/filerldb2/"
      - /etc/localtime:/etc/localtime
    tty: true
    stdin_open: true
    depends_on:
    - master
    - volume
  cronjob:
    image: chrislusf/seaweedfs # use a remote image
    command: 'cronjob'
    environment:
      # Run re-replication every 2 minutes
      CRON_SCHEDULE: '*/2 * * * * *' # Default: '*/5 * * * * *'
      WEED_MASTER: master:9333 # Default: localhost:9333
    depends_on:
    - master
    - volume

I have some data file

root@ubuntu:~/seaweedfs# tree 
.
├── data
│   └── volume
│       ├── 1.dat
│       ├── 1.idx
│       ├── 2.dat
│       ├── 2.idx
│       ├── 3.dat
│       ├── 3.idx
│       ├── 4.dat
│       ├── 4.idx
│       ├── 5.dat
│       ├── 5.idx
│       ├── 6.dat
│       ├── 6.idx
│       ├── 7.dat
│       └── 7.idx
├── docker-compose.yml -> /root/seaweedfs/seaweedfs/docker-compose.yml
├── filer.toml
├── run_postgres.sh
├── seaweedfs
│   ├── docker-compose.yml
│   └── README.md
├── seaweedfs.yml

@chrislusf
Copy link
Collaborator

seems you need to wait a bit more seconds

@TsingJyujing
Copy link
Author

seems you need to wait a bit more seconds

@chrislusf

For me: I can't reproduce this bug every time, but:

  • Every time I start service, raft server related error message will appear.
  • Some time even wait for minutes it still reporting error message.

@chrislusf
Copy link
Collaborator

which ones are "error message" you mentioned?

@TsingJyujing
Copy link
Author

which ones are "error message" you mentioned?

@chrislusf Yes! The error message looks like Raft server error.

@chrislusf
Copy link
Collaborator

those raft errors are transient

@chenxushuo
Copy link

I got the same error!
I start tow service master and volume used docker-compose,
but it got this error frequently.
Always It can be reseted sometimes late.
But sometime must restart master service :
pull logs:

weedfs-volume_1  | I1204 14:27:29     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:27:34     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:27:34     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:27:39     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:27:39     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:27:44     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:27:45     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:27:50     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:27:50     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:27:55     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:27:55     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:00     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:00     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:05     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:05     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:10     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:10     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:15     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:15     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:20     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:20     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:25     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:25     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:30     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:30     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:35     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:35     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:40     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:40     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:45     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:45     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:50     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:50     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:28:55     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:28:55     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:00     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:00     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:05     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:05     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:10     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:10     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:15     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:15     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:20     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:20     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:25     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:25     1 volume_grpc_client_to_master.go:45] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
weedfs-volume_1  | I1204 14:29:30     1 volume_grpc_client_to_master.go:68] Heartbeat to: weedfs-master:9333
weedfs-volume_1  | I1204 14:29:30     1 volume_grpc_client_to_master.go:84] Volume Server found a new master newLeader: 172.20.0.3:9333 instead of weedfs-master:9333
weedfs-volume_1  | I1204 14:29:30     1 volume_grpc_client_to_master.go:68] Heartbeat to: 172.20.0.3:9333
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 10 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 4 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 9 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 9 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 9 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 2 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 11 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 11 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 11 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 5 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 8 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 8 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 8 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 14 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 14 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 14 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 6 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 12 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 12 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 12 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 1 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 3 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 13 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 13 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:237] Volume 13 becomes unwritable
weedfs-master_1  | I1204 14:29:30     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 volume_layout.go:261] Volume 7 has 0 replica, less than required 1
weedfs-master_1  | I1204 14:29:30     1 node.go:256] topo:DefaultDataCenter:DefaultRack removes 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 node.go:241] topo:DefaultDataCenter:DefaultRack adds child 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:71] added volume server 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 1 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 14 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 4 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 9 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 8 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 13 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 7 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 2 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 11 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 12 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 3 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 5 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 6 from 172.20.0.5:8181
weedfs-master_1  | I1204 14:29:30     1 master_grpc_server.go:101] master see new volume 10 from 172.20.0.5:8181

@chrislusf
Copy link
Collaborator

@chenxushuo Thanks for the log. It shows the master is started correctly.

@chenxushuo
Copy link

Because I restart the master service. and it will get the some error soon.

@chrislusf
Copy link
Collaborator

When starting master, set the master -ip=master since this is the address within the docker cluster.

chrislusf added a commit that referenced this issue Dec 12, 2019
@chrislusf
set master ip in docker compose yaml file
06ff984 
fix #1118
@TsingJyujing
Copy link
Author

It still sometime happened that may cause my test case fail, please reopen it.

Verified on my MAC (maybe only happened on MAC?) , project repository here:

https://github.com/TsingJyujing/nemivir

docker-compose up --build --force-recreate --exit-code-from nemivir_unit_test
WARNING: using --exit-code-from implies --abort-on-container-exit
Building nemivir
Step 1/7 : FROM python:3.7
 ---> e497dabd8450
Step 2/7 : WORKDIR /app
 ---> Using cache
 ---> 68f0e54e0d2d
Step 3/7 : EXPOSE 8000
 ---> Using cache
 ---> 975976b02c99
Step 4/7 : RUN apt-get update &&     apt-get install -y libwebp-dev &&     apt-get clean
 ---> Using cache
 ---> fc3dfc2d14b9
Step 5/7 : RUN pip install --no-cache-dir     imagehash     redlock-py     uvicorn     fastapi     numpy     pillow     requests     prometheus_client     redis-lru     protobuf     python-multipart
 ---> Using cache
 ---> 15859bef3edf
Step 6/7 : COPY . .
 ---> Using cache
 ---> 2a902a4776fc
Step 7/7 : CMD ["uvicorn","nemivir.service.image_api:app","--host","0.0.0.0"]
 ---> Using cache
 ---> 5a991a4cc6d5
Successfully built 5a991a4cc6d5
Successfully tagged nemivir:latest
Building nemivir_unit_test
Step 1/7 : FROM python:3.7
 ---> e497dabd8450
Step 2/7 : WORKDIR /app
 ---> Using cache
 ---> 68f0e54e0d2d
Step 3/7 : EXPOSE 8000
 ---> Using cache
 ---> 975976b02c99
Step 4/7 : RUN apt-get update &&     apt-get install -y libwebp-dev &&     apt-get clean
 ---> Using cache
 ---> fc3dfc2d14b9
Step 5/7 : RUN pip install --no-cache-dir     imagehash     redlock-py     uvicorn     fastapi     numpy     pillow     requests     prometheus_client     redis-lru     protobuf     python-multipart
 ---> Using cache
 ---> 15859bef3edf
Step 6/7 : COPY . .
 ---> Using cache
 ---> 2a902a4776fc
Step 7/7 : CMD ["uvicorn","nemivir.service.image_api:app","--host","0.0.0.0"]
 ---> Using cache
 ---> 5a991a4cc6d5
Successfully built 5a991a4cc6d5
Successfully tagged nemivir:latest
Recreating nemivir_redis_1  ... done
Recreating nemivir_master_1 ... done
Recreating nemivir_volume_1 ... done
Recreating nemivir_filer_1  ... done
Recreating nemivir_nemivir_1 ... done
Recreating nemivir_nemivir_unit_test_1 ... done
Attaching to nemivir_master_1, nemivir_redis_1, nemivir_volume_1, nemivir_filer_1, nemivir_nemivir_1, nemivir_nemivir_unit_test_1
master_1             | I0110 07:26:23     1 config.go:27] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1              | 1:C 10 Jan 2020 07:26:23.936 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1              | 1:C 10 Jan 2020 07:26:23.936 # Redis version=5.0.6, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1              | 1:C 10 Jan 2020 07:26:23.936 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
volume_1             | I0110 07:26:24     1 config.go:27] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1              | 1:M 10 Jan 2020 07:26:23.937 * Running mode=standalone, port=6379.
redis_1              | 1:M 10 Jan 2020 07:26:23.937 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1              | 1:M 10 Jan 2020 07:26:23.938 # Server initialized
redis_1              | 1:M 10 Jan 2020 07:26:23.938 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
master_1             | I0110 07:26:23     1 config.go:27] Reading : Config File "master" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
redis_1              | 1:M 10 Jan 2020 07:26:23.942 * DB loaded from disk: 0.004 seconds
redis_1              | 1:M 10 Jan 2020 07:26:23.942 * Ready to accept connections
volume_1             | I0110 07:26:24     1 file_util.go:20] Folder /data Permission: -rwxr-xr-x
filer_1              | I0110 07:26:25     1 config.go:27] Reading : Config File "security" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
nemivir_1            | INFO:     email-validator not installed, email fields will be treated as str.
nemivir_1            | To install, run: pip install email-validator
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/7.idx to memory
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/6.idx to memory
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/1.idx to memory
master_1             | I0110 07:26:23     1 file_util.go:20] Folder /data Permission: -rwxr-xr-x
filer_1              | I0110 07:26:25     1 config.go:27] Reading : Config File "notification" Not Found in "[/ /root/.seaweedfs /etc/seaweedfs]"
nemivir_1            | INFO:     Started server process [1]
nemivir_1            | INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
nemivir_1            | INFO:     Waiting for application startup.
nemivir_1            | INFO:     Application startup complete.
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/5.idx to memory
filer_1              | I0110 07:26:25     1 leveldb2_store.go:39] filer store leveldb2 dir: /data/filerldb2
master_1             | I0110 07:26:23     1 master_server.go:102] Volume Size Limit is 30000 MB
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/2.idx to memory
filer_1              | I0110 07:26:25     1 file_util.go:20] Folder /data/filerldb2 Permission: -rwxr-xr-x
master_1             | I0110 07:26:23     1 master_server.go:188] adminScripts:
master_1             | I0110 07:26:23     1 master.go:112] Start Seaweed Master 30GB 1.50 at 0.0.0.0:9333
master_1             | I0110 07:26:23     1 raft_server.go:44] Starting RaftServer with master:9333
filer_1              | I0110 07:26:25     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:25     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/3.idx to memory
filer_1              | I0110 07:26:25     1 configuration.go:26] Configure filer for leveldb2
master_1             | I0110 07:26:23     1 raft_server.go:84] current cluster leader: 
master_1             | I0110 07:26:23     1 master.go:136] Start Seaweed Master 30GB 1.50 grpc server at 0.0.0.0:19333
master_1             | E0110 07:26:23     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:23     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:23     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/7.idx, replicaPlacement=000 v=3 size=7516704 ttl=
filer_1              | I0110 07:26:25     1 filer.go:133] Start Seaweed Filer 30GB 1.50 at :9401
master_1             | I0110 07:26:24     1 node.go:255] topo adds child DefaultDataCenter
master_1             | I0110 07:26:24     1 node.go:255] topo:DefaultDataCenter adds child DefaultRack
master_1             | I0110 07:26:24     1 node.go:255] topo:DefaultDataCenter:DefaultRack adds child 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:78] added volume server 172.19.0.4:9301
volume_1             | I0110 07:26:24     1 volume_loading.go:104] loading index /data/4.idx to memory
filer_1              | I0110 07:26:26     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:26     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 7 from 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 1 from 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 6 from 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 5 from 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 2 from 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 3 from 172.19.0.4:9301
master_1             | I0110 07:26:24     1 master_grpc_server.go:112] master see new volume 4 from 172.19.0.4:9301
master_1             | W0110 07:26:24     1 master_grpc_server.go:167] SendHeartbeat find leader: Raft Server not initialized!
master_1             | I0110 07:26:24     1 master_grpc_server.go:25] unregister disconnected volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 6 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 6 becomes unwritable
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 5 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 5 becomes unwritable
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 2 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 2 becomes unwritable
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 3 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 3 becomes unwritable
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 4 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 4 becomes unwritable
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 7 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 7 becomes unwritable
master_1             | I0110 07:26:24     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:24     1 volume_layout.go:262] Volume 1 has 0 replica, less than required 1
master_1             | I0110 07:26:24     1 volume_layout.go:238] Volume 1 becomes unwritable
master_1             | I0110 07:26:24     1 node.go:271] topo:DefaultDataCenter:DefaultRack removes 172.19.0.4:9301
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/1.idx, replicaPlacement=000 v=3 size=2266896 ttl=
master_1             | E0110 07:26:24     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:24     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:24     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/6.idx, replicaPlacement=000 v=3 size=2270072 ttl=
master_1             | E0110 07:26:25     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/5.idx, replicaPlacement=000 v=3 size=7506320 ttl=
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/2.idx, replicaPlacement=000 v=3 size=6020504 ttl=
master_1             | E0110 07:26:25     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/3.idx, replicaPlacement=000 v=3 size=5283768 ttl=
master_1             | I0110 07:26:25     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:25     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1             | I0110 07:26:24     1 disk_location.go:68] data file /data/4.idx, replicaPlacement=000 v=3 size=5265456 ttl=
volume_1             | I0110 07:26:24     1 disk_location.go:108] Store started on dir: /data with 7 volumes max 30
master_1             | E0110 07:26:26     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
volume_1             | I0110 07:26:24     1 disk_location.go:111] Store started on dir: /data with 0 ec shards
volume_1             | I0110 07:26:24     1 volume_grpc_client_to_master.go:25] Volume server start with seed master nodes: [master:9333]
volume_1             | I0110 07:26:24     1 volume.go:277] Start Seaweed volume server 30GB 1.50 at 0.0.0.0:9301
volume_1             | I0110 07:26:24     1 volume_grpc_client_to_master.go:69] Heartbeat to: master:9333
volume_1             | I0110 07:26:24     1 volume_grpc_client_to_master.go:46] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
master_1             | E0110 07:26:26     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:26     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:26     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
nemivir_unit_test_1  | INFO:unit_test.py:Waiting for service initialized.
filer_1              | I0110 07:26:27     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:27     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:27     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:27     1 master_server.go:140] state change: &{typ:stateChange source:0xc0002fe900 value:candidate prevValue:follower}
master_1             | E0110 07:26:27     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:27     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:27     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:28     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
filer_1              | I0110 07:26:28     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:28     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:28     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:28     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:28     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:29     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
filer_1              | I0110 07:26:29     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:29     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
volume_1             | I0110 07:26:29     1 volume_grpc_client_to_master.go:69] Heartbeat to: master:9333
master_1             | I0110 07:26:29     1 node.go:255] topo:DefaultDataCenter:DefaultRack adds child 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:78] added volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 6 from 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 5 from 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 2 from 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 3 from 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 4 from 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 7 from 172.19.0.4:9301
master_1             | I0110 07:26:29     1 master_grpc_server.go:112] master see new volume 1 from 172.19.0.4:9301
master_1             | W0110 07:26:29     1 master_grpc_server.go:167] SendHeartbeat find leader: Raft Server not initialized!
master_1             | I0110 07:26:29     1 master_grpc_server.go:25] unregister disconnected volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 6 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 6 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 6 becomes unwritable
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 5 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 5 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 5 becomes unwritable
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 2 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 2 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 2 becomes unwritable
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 3 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 3 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 3 becomes unwritable
volume_1             | I0110 07:26:29     1 volume_grpc_client_to_master.go:46] heartbeat error: rpc error: code = Unknown desc = Raft Server not initialized!
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 4 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 4 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 4 becomes unwritable
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 7 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 7 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 7 becomes unwritable
master_1             | I0110 07:26:29     1 topology_event_handling.go:57] Removing Volume 1 from the dead volume server 172.19.0.4:9301
master_1             | I0110 07:26:29     1 volume_layout.go:262] Volume 1 has 0 replica, less than required 1
master_1             | I0110 07:26:29     1 volume_layout.go:238] Volume 1 becomes unwritable
master_1             | I0110 07:26:29     1 node.go:271] topo:DefaultDataCenter:DefaultRack removes 172.19.0.4:9301
master_1             | E0110 07:26:29     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:29     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:29     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:30     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
filer_1              | I0110 07:26:30     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:30     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:30     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:30     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:30     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:31     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
filer_1              | I0110 07:26:31     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:31     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | E0110 07:26:31     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
master_1             | I0110 07:26:31     1 masterclient.go:88] master failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
master_1             | I0110 07:26:31     1 masterclient.go:116] master failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
nemivir_unit_test_1  | INFO:unit_test.py:Cleaning up
nemivir_1            | INFO:     172.19.0.7:44280 - "GET /list/hash HTTP/1.1" 200 OK
filer_1              | E0110 07:26:32     1 filer_server_handlers_write.go:67] failing to assign a file id: failed to parse master grpc : server should have hostname:port format: 
filer_1              | I0110 07:26:32     1 filer_server_handlers_write.go:103] fail to allocate volume for /66c92249fb24c9266d9b24892/4e85413533a8_59bc40b7b87f4_71vpl9UT.img, collection:, datacenter:
nemivir_1            | INFO:     172.19.0.7:44286 - "POST /upload HTTP/1.1" 500 Internal Server Error
nemivir_unit_test_1  | Traceback (most recent call last):
nemivir_unit_test_1  |   File "unit_test.py", line 134, in <module>
nemivir_unit_test_1  |     verify_image("/tmp/test.jpg")
nemivir_unit_test_1  |   File "unit_test.py", line 44, in verify_image
nemivir_unit_test_1  |     resp.raise_for_status()
nemivir_unit_test_1  |   File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
nemivir_unit_test_1  |     raise HTTPError(http_error_msg, response=self)
nemivir_unit_test_1  | requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://nemivir:8000/upload
master_1             | E0110 07:26:32     1 master_grpc_server.go:262] topo leader: Raft Server not initialized!
filer_1              | I0110 07:26:32     1 masterclient.go:88] filer failed to receive from master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
filer_1              | I0110 07:26:32     1 masterclient.go:116] filer failed to connect with master master:9333: rpc error: code = Unknown desc = raft.Server: Not current leader
nemivir_nemivir_unit_test_1 exited with code 1
Aborting on container exit...
Stopping nemivir_nemivir_1             ... done
Stopping nemivir_filer_1               ... done
Stopping nemivir_volume_1              ... done
Stopping nemivir_master_1              ... done
Stopping nemivir_redis_1               ... done

chrislusf added a commit that referenced this issue Jan 10, 2020
@chrislusf
treat it as a single node cluster if empty raft server name
d1ab16b 
possible fix for #1118
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@chrislusf @TsingJyujing @xiaoweihong @chenxushuo