Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[weed mount] write file error #1639

Closed
paochiang opened this issue Nov 26, 2020 · 19 comments
Closed

[weed mount] write file error #1639

paochiang opened this issue Nov 26, 2020 · 19 comments

Comments

@paochiang
Copy link
Contributor

Describe the bug
after weed mount, write file to filesystem, get error(filer‘s log):

{"log":"I1126 07:51:25     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:33492\n","stream":"stderr","time":"2020-11-26T07:51:25.871024505Z"}
{"log":"I1126 07:51:25     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:33492 starts to subscribe /registry01 from 2020-11-26 07:51:25.870030127 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:51:25.871089771Z"}
{"log":"I1126 07:55:00     1 filer_grpc_server_sub_meta.go:166] =\u003e client mount@172.25.0.1:33260: rpc error: code = Unavailable desc = transport is closing\n","stream":"stderr","time":"2020-11-26T07:55:00.619963419Z"}
{"log":"E1126 07:55:00     1 filer_grpc_server_sub_meta.go:49] processed to 2020-11-26 07:55:00.619624123 +0000 UTC: rpc error: code = Unavailable desc = transport is closing\n","stream":"stderr","time":"2020-11-26T07:55:00.62006021Z"}
{"log":"E1126 07:55:00     1 filer_delete_entry.go:79] deleting a folder /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T07:55:00.822087471Z"}
{"log":"I1126 07:55:00     1 filer_delete_entry.go:36] delete directory /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c: fail to delete non-empty folder: /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c\n","stream":"stderr","time":"2020-11-26T07:55:00.822130551Z"}
{"log":"E1126 07:55:00     1 filer_delete_entry.go:79] deleting a folder /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T07:55:00.822707754Z"}

The error in the last line above appears many many times

System Setup
version 2.12
seaweedfs-compose.yml:

version: '2'

services:
  master:
    image: chrislusf/seaweedfs:2.12
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:2.12
    ports:
      - 8080:8080
      - 18080:18080
    command: "volume -mserver=master:9333 -port=8080 -ip=volume -publicUrl=localhost:8080 -max=0"
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:2.12
    ports:
      - 8888:8888
      - 18888:18888
    command: 'filer -master="master:9333"'
    depends_on:
      - master
      - volume

docker-registry-config.yml:

version: 0.1
log:
  level: debug
storage:
  delete:
    enabled: true
  redirect:
    disable: true
  filesystem:
    rootdirectory: /var/lib/registry
    maxthreads: 100
validation:
  disabled: true
http:
  addr: 0.0.0.0:5001

run:

# set up seaweed
docker-compose -f seaweedfs-compose.yml -p seaweedfs up

# download source and build , get binary weed, omit...

# use weed mount to host path
sudo mkdir -p /mnt/registry
sudo ./weed mount -outsideContainerClusterMode -cacheCapacityMB=0 -filer=127.0.0.1:8888 -filer.path=/registry -dir=/mnt/registry

# setup docker registry
docker run -d -p 5001:5001 --name localregistry -v /home/orion/docker-registry-config.yml:/etc/docker/registry/config.yml -v /mnt/registry:/var/lib/registry registry:2

Expected behavior
write file success to filesystem from weed mount

Additional context
1、docker pull nginx:latest
2、docke tag nginx:latest 127.0.0.1:5001/nginx:latest
3、docker push 127.0.0.1:5001/nginx:latest

The push refers to repository [127.0.0.1:5001/nginx]
7e914612e366: Pushing [==================================================>]  3.584kB
f790aed835ee: Pushing [==================================================>]  4.096kB
850c2400ea4d: Pushing [==================================================>]  3.072kB
7ccabd267c9f: Pushing [===========================>                       ]  35.35MB/63.65MB
f5600c6330da: Pushing [===============>                                   ]  21.67MB/69.24MB

The upload process is stuck,can not finish push

This error just likes #1609, but this time docker registry backend use filesystem from weed mount

@paochiang
Copy link
Contributor Author

add more log
filer:

{"log":"I1126 07:32:15     1 leveldb2_store.go:40] filer store leveldb2 dir: /data/filerldb2\n","stream":"stderr","time":"2020-11-26T07:32:15.755518856Z"}
{"log":"I1126 07:32:15     1 file_util.go:23] Folder /data/filerldb2 Permission: -rwxr-xr-x\n","stream":"stderr","time":"2020-11-26T07:32:15.758926698Z"}
{"log":"I1126 07:32:15     1 filer.go:98] create filer.store.id to 1918813037\n","stream":"stderr","time":"2020-11-26T07:32:15.900919322Z"}
{"log":"I1126 07:32:15     1 configuration.go:25] Configure filer for leveldb2\n","stream":"stderr","time":"2020-11-26T07:32:15.900963501Z"}
{"log":"I1126 07:32:15     1 filer.go:172] Start Seaweed Filer 30GB 2.12 d3be2cee at 172.25.0.4:8888\n","stream":"stderr","time":"2020-11-26T07:32:15.90273815Z"}
{"log":"I1126 07:32:16     1 filer_grpc_server_sub_meta.go:175] + listener filer:172.25.0.4:8888@172.25.0.4:43426\n","stream":"stderr","time":"2020-11-26T07:32:16.907384679Z"}
{"log":"I1126 07:32:16     1 filer_grpc_server_sub_meta.go:70]  filer:172.25.0.4:8888@172.25.0.4:43426 local subscribe / from 2020-11-26 07:31:15.902195222 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:32:16.907905894Z"}
{"log":"I1126 07:32:16     1 filer_grpc_server_sub_meta.go:85] after local log reads, filer:172.25.0.4:8888@172.25.0.4:43426 local subscribe / from 2020-11-26 07:31:15.902195222 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:32:16.908726618Z"}
{"log":"I1126 07:36:21     1 leveldb2_store.go:40] filer store leveldb2 dir: /data/filerldb2\n","stream":"stderr","time":"2020-11-26T07:36:21.747831365Z"}
{"log":"I1126 07:36:21     1 file_util.go:23] Folder /data/filerldb2 Permission: -rwxr-xr-x\n","stream":"stderr","time":"2020-11-26T07:36:21.74869608Z"}
{"log":"I1126 07:36:21     1 filer.go:101] existing filer.store.id = 1918813037\n","stream":"stderr","time":"2020-11-26T07:36:21.872971759Z"}
{"log":"I1126 07:36:21     1 configuration.go:25] Configure filer for leveldb2\n","stream":"stderr","time":"2020-11-26T07:36:21.873057522Z"}
{"log":"I1126 07:36:21     1 filer.go:172] Start Seaweed Filer 30GB 2.12 d3be2cee at 172.25.0.4:8888\n","stream":"stderr","time":"2020-11-26T07:36:21.876656783Z"}
{"log":"I1126 07:36:22     1 filer_grpc_server_sub_meta.go:175] + listener filer:172.25.0.4:8888@172.25.0.4:44286\n","stream":"stderr","time":"2020-11-26T07:36:22.880885844Z"}
{"log":"I1126 07:36:22     1 filer_grpc_server_sub_meta.go:70]  filer:172.25.0.4:8888@172.25.0.4:44286 local subscribe / from 2020-11-26 07:35:21.876430065 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:36:22.880919319Z"}
{"log":"I1126 07:36:22     1 filer_grpc_server_sub_meta.go:85] after local log reads, filer:172.25.0.4:8888@172.25.0.4:44286 local subscribe / from 2020-11-26 07:35:21.876430065 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:36:22.880930134Z"}
{"log":"I1126 07:50:24     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:33260\n","stream":"stderr","time":"2020-11-26T07:50:24.343597335Z"}
{"log":"I1126 07:50:24     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:33260 starts to subscribe /registry01 from 2020-11-26 07:50:24.341251838 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:50:24.343678983Z"}
{"log":"I1126 07:51:25     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:33492\n","stream":"stderr","time":"2020-11-26T07:51:25.871024505Z"}
{"log":"I1126 07:51:25     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:33492 starts to subscribe /registry01 from 2020-11-26 07:51:25.870030127 +0000 UTC\n","stream":"stderr","time":"2020-11-26T07:51:25.871089771Z"}
{"log":"I1126 07:55:00     1 filer_grpc_server_sub_meta.go:166] =\u003e client mount@172.25.0.1:33260: rpc error: code = Unavailable desc = transport is closing\n","stream":"stderr","time":"2020-11-26T07:55:00.619963419Z"}
{"log":"E1126 07:55:00     1 filer_grpc_server_sub_meta.go:49] processed to 2020-11-26 07:55:00.619624123 +0000 UTC: rpc error: code = Unavailable desc = transport is closing\n","stream":"stderr","time":"2020-11-26T07:55:00.62006021Z"}
{"log":"E1126 07:55:00     1 filer_delete_entry.go:79] deleting a folder /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T07:55:00.822087471Z"}
{"log":"I1126 07:55:00     1 filer_delete_entry.go:36] delete directory /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c: fail to delete non-empty folder: /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c\n","stream":"stderr","time":"2020-11-26T07:55:00.822130551Z"}
{"log":"E1126 07:55:00     1 filer_delete_entry.go:79] deleting a folder /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T07:55:00.822707754Z"}
{"log":"I1126 07:55:00     1 filer_delete_entry.go:36] delete directory /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c: fail to delete non-empty folder: /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c\n","stream":"stderr","time":"2020-11-26T07:55:00.822730575Z"}
{"log":"E1126 07:55:00     1 filer_delete_entry.go:79] deleting a folder /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T07:55:00.82315072Z"}
{"log":"I1126 07:55:00     1 filer_delete_entry.go:36] delete directory /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c: fail to delete non-empty folder: /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c\n","stream":"stderr","time":"2020-11-26T07:55:00.823203244Z"}
{"log":"E1126 07:55:00     1 filer_delete_entry.go:79] deleting a folder /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T07:55:00.823621251Z"}
{"log":"I1126 07:55:00     1 filer_delete_entry.go:36] delete directory /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c: fail to delete non-empty folder: /registry01/docker/registry/v2/repositories/nginx/_uploads/a22c3818-57f5-4e04-bac4-107e0a0a999c\n","stream":"stderr","time":"2020-11-26T07:55:00.824127716Z"}
............ too many

volume:

I1126 07:32:14     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
I1126 07:32:19     1 disk_location.go:133] Store started on dir: /data with 0 volumes max 0
I1126 07:32:19     1 disk_location.go:136] Store started on dir: /data with 0 ec shards
I1126 07:32:19     1 volume_grpc_client_to_master.go:52] Volume server start with seed master nodes: [master:9333]
I1126 07:32:19     1 volume.go:331] Start Seaweed volume server 30GB 2.12 d3be2cee at 0.0.0.0:8080
I1126 07:32:19     1 volume_grpc_client_to_master.go:114] Heartbeat to: master:9333
volume server has be killed
I1126 07:33:38     1 volume.go:242] stop send heartbeat and wait 10 seconds until shutdown ...
I1126 07:33:48     1 volume.go:266] graceful stop cluster http server ...
I1126 07:33:48     1 volume.go:271] graceful stop gRPC ...
I1126 07:33:48     1 volume_server.go:106] Shutting down volume server...
I1126 07:33:48     1 volume_server.go:108] Shut down successfully!
I1126 07:36:20     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
I1126 07:36:25     1 disk_location.go:133] Store started on dir: /data with 0 volumes max 0
I1126 07:36:25     1 disk_location.go:136] Store started on dir: /data with 0 ec shards
I1126 07:36:25     1 volume.go:331] Start Seaweed volume server 30GB 2.12 d3be2cee at 0.0.0.0:8080
I1126 07:36:25     1 volume_grpc_client_to_master.go:52] Volume server start with seed master nodes: [master:9333]
I1126 07:36:25     1 volume_grpc_client_to_master.go:114] Heartbeat to: master:9333
I1126 07:51:02     1 store.go:123] In dir /data adds volume:1 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/1.idx to memory
I1126 07:51:02     1 store.go:127] add volume 1
I1126 07:51:02     1 store.go:123] In dir /data adds volume:2 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/2.idx to memory
I1126 07:51:02     1 store.go:127] add volume 2
I1126 07:51:02     1 store.go:123] In dir /data adds volume:3 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/3.idx to memory
I1126 07:51:02     1 store.go:127] add volume 3
I1126 07:51:02     1 store.go:123] In dir /data adds volume:4 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/4.idx to memory
I1126 07:51:02     1 store.go:127] add volume 4
I1126 07:51:02     1 store.go:123] In dir /data adds volume:5 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/5.idx to memory
I1126 07:51:02     1 store.go:127] add volume 5
I1126 07:51:02     1 store.go:123] In dir /data adds volume:6 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/6.idx to memory
I1126 07:51:02     1 store.go:127] add volume 6
I1126 07:51:02     1 store.go:123] In dir /data adds volume:7 collection: replicaPlacement:000 ttl:
I1126 07:51:02     1 volume_loading.go:104] loading index /data/7.idx to memory
I1126 07:51:02     1 store.go:127] add volume 7

master:

I1126 07:32:12     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
I1126 07:32:12     1 master.go:168] current: master:9333 peers:
I1126 07:32:12     1 master_server.go:107] Volume Size Limit is 1024 MB
I1126 07:32:12     1 master_server.go:192] adminScripts:
I1126 07:32:12     1 master.go:122] Start Seaweed Master 30GB 2.12 d3be2cee at 0.0.0.0:9333
I1126 07:32:12     1 raft_server.go:70] Starting RaftServer with master:9333
I1126 07:32:12     1 raft_server.go:129] current cluster leader:
I1126 07:32:12     1 master.go:146] Start Seaweed Master 30GB 2.12 d3be2cee grpc server at 0.0.0.0:19333
I1126 07:32:14     1 masterclient.go:78] No existing leader found!
I1126 07:32:14     1 raft_server.go:154] Initializing new cluster
I1126 07:32:14     1 master_server.go:141] leader change event:  => master:9333
I1126 07:32:14     1 master_server.go:143] [ master:9333 ] master:9333 becomes leader.
I1126 07:32:15     1 master_grpc_server.go:250] + client filer@172.25.0.4:8888
I1126 07:32:17     1 masterclient.go:126] redirected to leader master:9333
I1126 07:32:17     1 master_grpc_server.go:250] + client master@172.25.0.2:45890
I1126 07:32:19     1 node.go:278] topo adds child DefaultDataCenter
I1126 07:32:19     1 node.go:278] topo:DefaultDataCenter adds child DefaultRack
I1126 07:32:19     1 node.go:278] topo:DefaultDataCenter:DefaultRack adds child volume:8080
I1126 07:32:19     1 master_grpc_server.go:73] added volume server volume:8080
I1126 07:33:36     1 master_grpc_server.go:266] - client filer@172.25.0.4:8888
W1126 07:33:38     1 master_grpc_server.go:170] SendHeartbeat.Send response to to volume:8080 rpc error: code = Unavailable desc = transport is closing
I1126 07:33:38     1 node.go:294] topo:DefaultDataCenter:DefaultRack removes volume:8080
I1126 07:33:38     1 master_grpc_server.go:29] unregister disconnected volume server volume:8080
I1126 07:36:18     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
I1126 07:36:18     1 master.go:168] current: master:9333 peers:
I1126 07:36:19     1 master_server.go:107] Volume Size Limit is 1024 MB
I1126 07:36:19     1 master_server.go:192] adminScripts:
I1126 07:36:19     1 master.go:122] Start Seaweed Master 30GB 2.12 d3be2cee at 0.0.0.0:9333
I1126 07:36:19     1 raft_server.go:70] Starting RaftServer with master:9333
I1126 07:36:19     1 raft_server.go:129] current cluster leader:
I1126 07:36:19     1 master.go:146] Start Seaweed Master 30GB 2.12 d3be2cee grpc server at 0.0.0.0:19333
I1126 07:36:20     1 masterclient.go:78] No existing leader found!
I1126 07:36:20     1 raft_server.go:154] Initializing new cluster
I1126 07:36:20     1 master_server.go:141] leader change event:  => master:9333
I1126 07:36:20     1 master_server.go:143] [ master:9333 ] master:9333 becomes leader.
I1126 07:36:21     1 master_grpc_server.go:250] + client filer@172.25.0.4:8888
I1126 07:36:24     1 masterclient.go:126] redirected to leader master:9333
I1126 07:36:24     1 master_grpc_server.go:250] + client master@172.25.0.2:46760
I1126 07:36:25     1 node.go:278] topo adds child DefaultDataCenter
I1126 07:36:25     1 node.go:278] topo:DefaultDataCenter adds child DefaultRack
I1126 07:36:25     1 node.go:278] topo:DefaultDataCenter:DefaultRack adds child volume:8080
I1126 07:36:25     1 master_grpc_server.go:73] added volume server volume:8080
I1126 07:51:02     1 volume_layout.go:352] Volume 1 becomes writable
I1126 07:51:02     1 volume_growth.go:224] Created Volume 1 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:1  data_center:"DefaultDataCenter"
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:1  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_layout.go:352] Volume 2 becomes writable
I1126 07:51:02     1 volume_growth.go:224] Created Volume 2 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:2  data_center:"DefaultDataCenter"
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:2  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_layout.go:352] Volume 3 becomes writable
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:3  data_center:"DefaultDataCenter"
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:3  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_growth.go:224] Created Volume 3 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 volume_layout.go:352] Volume 4 becomes writable
I1126 07:51:02     1 volume_growth.go:224] Created Volume 4 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:4  data_center:"DefaultDataCenter"
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:4  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_layout.go:352] Volume 5 becomes writable
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:5  data_center:"DefaultDataCenter"
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:5  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_growth.go:224] Created Volume 5 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 volume_layout.go:352] Volume 6 becomes writable
I1126 07:51:02     1 volume_growth.go:224] Created Volume 6 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:6  data_center:"DefaultDataCenter"
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:6  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_layout.go:352] Volume 7 becomes writable
I1126 07:51:02     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:7  data_center:"DefaultDataCenter"
I1126 07:51:02     1 volume_growth.go:224] Created Volume 7 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 07:51:02     1 master_grpc_server.go:155] master send to master@172.25.0.2:46760: url:"volume:8080"  public_url:"localhost:8080"  new_vids:7  data_center:"DefaultDataCenter"

master and volume logs seem to be ok

@chrislusf
Copy link
Collaborator

can you try the config.yaml in this page? https://github.com/chrislusf/seaweedfs/wiki/Harbor-with-SeaweedFS

@paochiang
Copy link
Contributor Author

can you try the config.yaml in this page? https://github.com/chrislusf/seaweedfs/wiki/Harbor-with-SeaweedFS

still error:

{"log":"I1126 09:11:15     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:56540\n","stream":"stderr","time":"2020-11-26T09:11:15.684774144Z"}
{"log":"I1126 09:11:15     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:56540 starts to subscribe /registry02 from 2020-11-26 09:11:15.683627722 +0000 UTC\n","stream":"stderr","time":"2020-11-26T09:11:15.684815351Z"}
{"log":"E1126 09:16:03     1 filer_delete_entry.go:79] deleting a folder /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T09:16:03.860612784Z"}
{"log":"I1126 09:16:03     1 filer_delete_entry.go:36] delete directory /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63: fail to delete non-empty folder: /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63\n","stream":"stderr","time":"2020-11-26T09:16:03.861237957Z"}
{"log":"E1126 09:16:03     1 filer_delete_entry.go:79] deleting a folder /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T09:16:03.86195265Z"}
{"log":"I1126 09:16:03     1 filer_delete_entry.go:36] delete directory /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63: fail to delete non-empty folder: /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63\n","stream":"stderr","time":"2020-11-26T09:16:03.861993196Z"}
{"log":"E1126 09:16:03     1 filer_delete_entry.go:79] deleting a folder /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T09:16:03.866737271Z"}
{"log":"I1126 09:16:03     1 filer_delete_entry.go:36] delete directory /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63: fail to delete non-empty folder: /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63\n","stream":"stderr","time":"2020-11-26T09:16:03.866758714Z"}
{"log":"E1126 09:16:03     1 filer_delete_entry.go:79] deleting a folder /registry02/docker/registry/v2/repositories/nginx/_uploads/95c8f744-212e-4c89-9426-0132357e9a63 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T09:16:03.867857599Z"}

push process:

The push refers to repository [127.0.0.1:5000/nginx]
7e914612e366: Pushing [==================================================>]  3.584kB
f790aed835ee: Pushing [==================================================>]  4.096kB
850c2400ea4d: Pushing [==================================================>]  3.072kB
7ccabd267c9f: Pushing [=============>                                     ]  17.52MB/63.65MB
f5600c6330da: Pushing [=====================>                             ]  29.79MB/69.24MB

stucked, can't push success

@chrislusf
Copy link
Collaborator

Added a fix a102157

Please try maybe 15 minutes later with chrislusf/seaweedfs:dev

@paochiang
Copy link
Contributor Author

Added a fix a102157

Please try maybe 15 minutes later with chrislusf/seaweedfs:dev

seems doesn't work

{"log":"I1126 10:14:11     1 filer_grpc_server_sub_meta.go:85] after local log reads, filer:172.25.0.4:8888@172.25.0.4:55602 local subscribe / from 2020-11-26 10:13:11.871467884 +0000 UTC\n","stream":"stderr","time":"2020-11-26T10:14:11.875758848Z"}
{"log":"I1126 10:14:40     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:41770\n","stream":"stderr","time":"2020-11-26T10:14:40.040929345Z"}
{"log":"I1126 10:14:40     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:41770 starts to subscribe /registry03 from 2020-11-26 10:14:40.039584552 +0000 UTC\n","stream":"stderr","time":"2020-11-26T10:14:40.041069792Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.673881389Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.674103251Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.674805261Z"}

@chrislusf
Copy link
Collaborator

chrislusf commented Nov 26, 2020

please show the full logs.

@paochiang
Copy link
Contributor Author

wait, when weed mount, should i use the newly generated executable binary weed?

@paochiang
Copy link
Contributor Author

please show the full logs.

filer:

{"log":"I1126 10:14:11     1 leveldb2_store.go:40] filer store leveldb2 dir: /data/filerldb2\n","stream":"stderr","time":"2020-11-26T10:14:11.78925349Z"}
{"log":"I1126 10:14:11     1 file_util.go:23] Folder /data/filerldb2 Permission: -rwxr-xr-x\n","stream":"stderr","time":"2020-11-26T10:14:11.791174189Z"}
{"log":"I1126 10:14:11     1 filer.go:98] create filer.store.id to 352907018\n","stream":"stderr","time":"2020-11-26T10:14:11.871676933Z"}
{"log":"I1126 10:14:11     1 configuration.go:25] Configure filer for leveldb2\n","stream":"stderr","time":"2020-11-26T10:14:11.871716142Z"}
{"log":"I1126 10:14:11     1 filer.go:172] Start Seaweed Filer 30GB 2.12 d3be2cee at 172.25.0.4:8888\n","stream":"stderr","time":"2020-11-26T10:14:11.871728127Z"}
{"log":"I1126 10:14:11     1 filer_grpc_server_sub_meta.go:175] + listener filer:172.25.0.4:8888@172.25.0.4:55602\n","stream":"stderr","time":"2020-11-26T10:14:11.875016015Z"}
{"log":"I1126 10:14:11     1 filer_grpc_server_sub_meta.go:70]  filer:172.25.0.4:8888@172.25.0.4:55602 local subscribe / from 2020-11-26 10:13:11.871467884 +0000 UTC\n","stream":"stderr","time":"2020-11-26T10:14:11.875300524Z"}
{"log":"I1126 10:14:11     1 filer_grpc_server_sub_meta.go:85] after local log reads, filer:172.25.0.4:8888@172.25.0.4:55602 local subscribe / from 2020-11-26 10:13:11.871467884 +0000 UTC\n","stream":"stderr","time":"2020-11-26T10:14:11.875758848Z"}
{"log":"I1126 10:14:40     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:41770\n","stream":"stderr","time":"2020-11-26T10:14:40.040929345Z"}
{"log":"I1126 10:14:40     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:41770 starts to subscribe /registry03 from 2020-11-26 10:14:40.039584552 +0000 UTC\n","stream":"stderr","time":"2020-11-26T10:14:40.041069792Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.673881389Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.674103251Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.674805261Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.674826588Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.675401433Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.675412371Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.676008333Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.676030601Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.678182297Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.678198351Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.678202601Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.678206223Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.678209913Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.678213242Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.678217452Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.678220948Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.678679152Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.678688376Z"}
{"log":"E1126 10:15:27     1 filer_delete_entry.go:79] deleting a folder /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6 has children: hashstates ...\n","stream":"stderr","time":"2020-11-26T10:15:27.679396161Z"}
{"log":"I1126 10:15:27     1 filer_delete_entry.go:36] delete directory /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6: fail to delete non-empty folder: /registry03/docker/registry/v2/repositories/nginx/_uploads/e48548be-38a1-453e-b7c7-7ca2ccf31aa6\n","stream":"stderr","time":"2020-11-26T10:15:27.679412799Z"}
。。。too many

volume:

I1126 10:14:10     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
I1126 10:14:10     1 disk_location.go:133] Store started on dir: /data with 0 volumes max 0
I1126 10:14:10     1 disk_location.go:136] Store started on dir: /data with 0 ec shards
I1126 10:14:10     1 volume.go:331] Start Seaweed volume server 30GB 2.12 d3be2cee at 0.0.0.0:8080
I1126 10:14:10     1 volume_grpc_client_to_master.go:52] Volume server start with seed master nodes: [master:9333]
I1126 10:14:10     1 volume_grpc_client_to_master.go:114] Heartbeat to: master:9333
I1126 10:15:27     1 store.go:123] In dir /data adds volume:1 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/1.idx to memory
I1126 10:15:27     1 store.go:127] add volume 1
I1126 10:15:27     1 store.go:123] In dir /data adds volume:2 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/2.idx to memory
I1126 10:15:27     1 store.go:127] add volume 2
I1126 10:15:27     1 store.go:123] In dir /data adds volume:3 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/3.idx to memory
I1126 10:15:27     1 store.go:127] add volume 3
I1126 10:15:27     1 store.go:123] In dir /data adds volume:4 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/4.idx to memory
I1126 10:15:27     1 store.go:127] add volume 4
I1126 10:15:27     1 store.go:123] In dir /data adds volume:5 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/5.idx to memory
I1126 10:15:27     1 store.go:127] add volume 5
I1126 10:15:27     1 store.go:123] In dir /data adds volume:6 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/6.idx to memory
I1126 10:15:27     1 store.go:127] add volume 6
I1126 10:15:27     1 store.go:123] In dir /data adds volume:7 collection: replicaPlacement:000 ttl:
I1126 10:15:27     1 volume_loading.go:104] loading index /data/7.idx to memory
I1126 10:15:27     1 store.go:127] add volume 7
I1126 10:29:07     1 volume_vacuum.go:95] Committing volume 4 vacuuming...
I1126 10:29:07     1 volume_loading.go:104] loading index /data/4.idx to memory

master:

I1126 10:14:07     1 file_util.go:23] Folder /data Permission: -rwxr-xr-x
I1126 10:14:07     1 master.go:168] current: master:9333 peers:
I1126 10:14:07     1 master_server.go:107] Volume Size Limit is 1024 MB
I1126 10:14:07     1 master_server.go:192] adminScripts:
I1126 10:14:07     1 master.go:122] Start Seaweed Master 30GB 2.12 d3be2cee at 0.0.0.0:9333
I1126 10:14:07     1 raft_server.go:70] Starting RaftServer with master:9333
I1126 10:14:07     1 raft_server.go:129] current cluster leader:
I1126 10:14:07     1 master.go:146] Start Seaweed Master 30GB 2.12 d3be2cee grpc server at 0.0.0.0:19333
I1126 10:14:09     1 masterclient.go:78] No existing leader found!
I1126 10:14:09     1 raft_server.go:154] Initializing new cluster
I1126 10:14:09     1 master_server.go:141] leader change event:  => master:9333
I1126 10:14:09     1 master_server.go:143] [ master:9333 ] master:9333 becomes leader.
I1126 10:14:10     1 node.go:278] topo adds child DefaultDataCenter
I1126 10:14:10     1 node.go:278] topo:DefaultDataCenter adds child DefaultRack
I1126 10:14:10     1 node.go:278] topo:DefaultDataCenter:DefaultRack adds child volume:8080
I1126 10:14:10     1 master_grpc_server.go:73] added volume server volume:8080
I1126 10:14:11     1 master_grpc_server.go:250] + client filer@172.25.0.4:8888
I1126 10:14:12     1 masterclient.go:126] redirected to leader master:9333
I1126 10:14:12     1 master_grpc_server.go:250] + client master@172.25.0.2:58066
I1126 10:15:27     1 volume_layout.go:352] Volume 1 becomes writable
I1126 10:15:27     1 volume_growth.go:224] Created Volume 1 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:1  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:1  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_layout.go:352] Volume 2 becomes writable
I1126 10:15:27     1 volume_growth.go:224] Created Volume 2 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:2  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:2  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_layout.go:352] Volume 3 becomes writable
I1126 10:15:27     1 volume_growth.go:224] Created Volume 3 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:3  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:3  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_layout.go:352] Volume 4 becomes writable
I1126 10:15:27     1 volume_growth.go:224] Created Volume 4 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:4  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:4  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_layout.go:352] Volume 5 becomes writable
I1126 10:15:27     1 volume_growth.go:224] Created Volume 5 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:5  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:5  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_layout.go:352] Volume 6 becomes writable
I1126 10:15:27     1 volume_growth.go:224] Created Volume 6 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:6  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:6  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_layout.go:352] Volume 7 becomes writable
I1126 10:15:27     1 master_grpc_server.go:155] master send to filer@172.25.0.4:8888: url:"volume:8080"  public_url:"localhost:8080"  new_vids:7  data_center:"DefaultDataCenter"
I1126 10:15:27     1 master_grpc_server.go:155] master send to master@172.25.0.2:58066: url:"volume:8080"  public_url:"localhost:8080"  new_vids:7  data_center:"DefaultDataCenter"
I1126 10:15:27     1 volume_growth.go:224] Created Volume 7 on topo:DefaultDataCenter:DefaultRack:volume:8080
I1126 10:29:07     1 volume_layout.go:340] Volume 4 becomes unwritable
I1126 10:29:07     1 topology_vacuum.go:70] 0 Start vacuuming 4 on volume:8080
I1126 10:29:07     1 topology_vacuum.go:82] Complete vacuuming 4 on volume:8080
I1126 10:29:07     1 topology_vacuum.go:106] Start Committing vacuum 4 on volume:8080
I1126 10:29:07     1 topology_vacuum.go:120] Complete Committing vacuum 4 on volume:8080
I1126 10:29:07     1 volume_layout.go:352] Volume 4 becomes writable

weed mount(use 2.12 release version executable binary weed ):

This is SeaweedFS version 30GB 2.12  linux amd64
mount point owner uid=0 gid=0 mode=drwxr-xr-x
current uid=0 gid=0
I1126 18:14:40  4340 leveldb_store.go:40] filer store dir: /tmp/9d49/meta
I1126 18:14:40  4340 file_util.go:23] Folder /tmp/9d49/meta Permission: -rwxr-xr-x
I1126 18:14:40  4340 mount_std.go:207] mounted 127.0.0.1:8888/registry03 to /mnt/registry

@paochiang
Copy link
Contributor Author

paochiang commented Nov 26, 2020

Added a fix a102157

Please try maybe 15 minutes later with chrislusf/seaweedfs:dev

use this version build executable weed to weed mount, the problem was fixed . l can push the image successfully

@paochiang
Copy link
Contributor Author

I found that although I successfully pushed the image, there are some logs on filer and weed mount contained: fail to delete non-empty folder.

filer full log:

I1126 11:07:48     1 leveldb2_store.go:40] filer store leveldb2 dir: /data/filerldb2
I1126 11:07:48     1 file_util.go:23] Folder /data/filerldb2 Permission: -rwxr-xr-x
I1126 11:07:48     1 filer.go:98] create filer.store.id to -475906747
I1126 11:07:48     1 configuration.go:25] Configure filer for leveldb2
I1126 11:07:48     1 filer.go:172] Start Seaweed Filer 30GB 2.12 a1021570 at 172.25.0.4:8888
I1126 11:07:48     1 filer_grpc_server_sub_meta.go:175] + listener filer:172.25.0.4:8888@172.25.0.4:40096
I1126 11:07:48     1 filer_grpc_server_sub_meta.go:70]  filer:172.25.0.4:8888@172.25.0.4:40096 local subscribe / from 2020-11-26 11:06:48.860805685 +0000 UTC
I1126 11:07:48     1 filer_grpc_server_sub_meta.go:85] after local log reads, filer:172.25.0.4:8888@172.25.0.4:40096 local subscribe / from 2020-11-26 11:06:48.860805685 +0000 UTC
I1126 11:09:12     1 filer_grpc_server_sub_meta.go:175] + listener mount@172.25.0.1:54690
I1126 11:09:12     1 filer_grpc_server_sub_meta.go:26]  mount@172.25.0.1:54690 starts to subscribe /registry04 from 2020-11-26 11:09:12.443662809 +0000 UTC
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea has children: hashstates ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade has children: hashstates ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates has children: sha256 ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates has children: sha256 ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates/sha256 has children: 0 ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates/sha256
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates/sha256 has children: 0 ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates/sha256
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f has children: hashstates ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates has children: sha256 ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates
E1126 11:11:08     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates/sha256 has children: 0 ...
I1126 11:11:08     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates/sha256
E1126 11:11:16     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9 has children: hashstates ...
I1126 11:11:16     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9
E1126 11:11:16     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates has children: sha256 ...
I1126 11:11:16     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates
E1126 11:11:16     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates/sha256 has children: 0 ...
I1126 11:11:16     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates/sha256
E1126 11:11:16     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b has children: hashstates ...
I1126 11:11:16     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b
E1126 11:11:16     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates has children: sha256 ...
I1126 11:11:16     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates
E1126 11:11:16     1 filer_delete_entry.go:79] deleting a folder /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates/sha256 has children: 0 ...
I1126 11:11:16     1 filer_delete_entry.go:36] delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates/sha256

weed mount full log:

This is SeaweedFS version 30GB 2.12  linux amd64
mount point owner uid=0 gid=0 mode=drwxr-xr-x
current uid=0 gid=0
I1126 19:09:12 26636 leveldb_store.go:40] filer store dir: /tmp/5c34/meta
I1126 19:09:12 26636 file_util.go:23] Folder /tmp/5c34/meta Permission: -rwxr-xr-x
I1126 19:09:12 26636 mount_std.go:207] mounted 127.0.0.1:8888/registry04 to /mnt/registry
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates/sha256: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/b8f812f3-821c-40c8-8d3d-9a1a629620ea/hashstates/sha256
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates/sha256: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/7d523a6d-3dc6-4332-8790-31e221620ade/hashstates/sha256
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates
I1126 19:11:08 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates/sha256: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/ab16a368-65e9-4f50-9d0e-ba1c2776328f/hashstates/sha256
I1126 19:11:16 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9
I1126 19:11:16 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates
I1126 19:11:16 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates/sha256: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/f7e59b6e-622b-4a31-9d9a-58543acf69a9/hashstates/sha256
I1126 19:11:16 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b
I1126 19:11:16 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates
I1126 19:11:16 26636 dir.go:396] remove /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates/sha256: delete directory /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates/sha256: fail to delete non-empty folder: /registry04/docker/registry/v2/repositories/drone/drone/_uploads/13ff3b66-7063-4312-9751-295ba9fe3c6b/hashstates/sha256

I want to know whether the above logs, such as fail to delete non-empty folder, will affect the service?

@chrislusf
Copy link
Collaborator

No.

@paochiang
Copy link
Contributor Author

Added a fix a102157

Please try maybe 15 minutes later with chrislusf/seaweedfs:dev

when i setup seaweed in k8s,use this version. Pushing the image still get stuck.

when i push image, master、volume and weed mount did not output logs, only filer outputs the following log :

I1127 13:04:23     1 filer_grpc_server_sub_meta.go:166] => client mount@10.10.10.215:62635: rpc error: code = Unavailable desc = transport is closing
E1127 13:04:23     1 filer_grpc_server_sub_meta.go:49] processed to 2020-11-27 13:04:23.701602823 +0000 UTC: rpc error: code = Unavailable desc = transport is closing
I1127 13:04:26     1 filer_grpc_server_sub_meta.go:180] - listener mount@10.10.10.215:62635

@chrislusf
Copy link
Collaborator

Can not help without details. Do you want to share your k8s setup details?

@paochiang
Copy link
Contributor Author

totoal: 1 master pod、 1 filer pod、 1 volume pod、1 master svc、1 filer svc、 2 volume svc(1 headless and 1 nodeport)

# pod
seaweedfs-filer-0                                    1/1     Running   0          28m
seaweedfs-master-0                                   1/1     Running   0          28m
seaweedfs-volume-0                                   1/1     Running   0          28m

# svc
seaweedfs-filer                             NodePort       10.43.201.144   <none>        8888:32601/TCP,18888:32602/TCP                                            25m
seaweedfs-master                            ClusterIP      None            <none>        9333/TCP,19333/TCP                                                        25m
seaweedfs-volume                            ClusterIP      None            <none>        8080/TCP,18080/TCP                                                        25m
seaweedfs-volume-0-external                 NodePort       10.43.140.161   <none>        8181:32605/TCP                                                            25m

detail info
kubectl describe pod seaweedfs-filer-0:

Name:         seaweedfs-filer-0
Namespace:    default
Priority:     0
Node:         qa-5/10.10.10.215
Start Time:   Sat, 28 Nov 2020 10:55:21 +0800
Labels:       app=seaweedfs
              chart=seaweedfs-helm
              component=filer
              controller-revision-hash=seaweedfs-filer-54888565d7
              release=gemini-swfs
              statefulset.kubernetes.io/pod-name=seaweedfs-filer-0
Annotations:  cni.projectcalico.org/podIP: 10.42.0.216/32
              cni.projectcalico.org/podIPs: 10.42.0.216/32
Status:       Running
IP:           10.42.0.216
IPs:
  IP:           10.42.0.216
Controlled By:  StatefulSet/seaweedfs-filer
Containers:
  seaweedfs:
    Container ID:  docker://652116883e0ccddf9330c2b52334676aac4e76a8a0f7bdd2f1d312308f66b5f7
    Image:         chrislusf/seaweedfs:dev
    Image ID:      docker-pullable://chrislusf/seaweedfs@sha256:242b4235db3a4d324f898533977bab0b7f2790caec2b5bf47fdfeec85994b191
    Ports:         8888/TCP, 18888/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
      exec /usr/bin/weed \
      filer \
      -port=8888 \
      -ip=${POD_IP} \
      -master=${SEAWEEDFS_FULLNAME}-master-0.${SEAWEEDFS_FULLNAME}-master:9333

    State:          Running
      Started:      Sat, 28 Nov 2020 10:55:23 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8888/ delay=20s timeout=3s period=30s #success=1 #failure=5
    Readiness:      http-get http://:8888/ delay=10s timeout=3s period=15s #success=1 #failure=100
    Environment:
      POD_IP:                                (v1:status.podIP)
      POD_NAME:                             seaweedfs-filer-0 (v1:metadata.name)
      NAMESPACE:                            default (v1:metadata.namespace)
      WEED_MYSQL_USERNAME:                  <set to the key 'user' in secret 'secret-seaweedfs-db'>      Optional: false
      WEED_MYSQL_PASSWORD:                  <set to the key 'password' in secret 'secret-seaweedfs-db'>  Optional: false
      SEAWEEDFS_FULLNAME:                   seaweedfs
      WEED_FILER_BUCKETS_FOLDER:            /buckets
      WEED_FILER_OPTIONS_RECURSIVE_DELETE:  false
      WEED_LEVELDB2_ENABLED:                false
      WEED_MYSQL_CONNECTION_MAX_IDLE:       10
      WEED_MYSQL_CONNECTION_MAX_OPEN:       150
      WEED_MYSQL_DATABASE:                  sw_database
      WEED_MYSQL_ENABLED:                   true
      WEED_MYSQL_HOSTNAME:                  virtaitech-gemini-mysql
      WEED_MYSQL_INTERPOLATEPARAMS:         true
      WEED_MYSQL_PORT:                      3306
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from seaweefds-rw-sa-token-qxqcc (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  seaweedfs-filer-log-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /storage/logs/seaweedfs/filer
    HostPathType:  DirectoryOrCreate
  seaweefds-rw-sa-token-qxqcc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  seaweefds-rw-sa-token-qxqcc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  sw-backend=true
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  16m   default-scheduler  Successfully assigned default/seaweedfs-filer-0 to qa-5
  Normal  Pulled     16m   kubelet, qa-5      Container image "chrislusf/seaweedfs:dev" already present on machine
  Normal  Created    16m   kubelet, qa-5      Created container seaweedfs
  Normal  Started    16m   kubelet, qa-5      Started container seaweedfs

kubectl describe pod seaweedfs-master-0:

Name:         seaweedfs-master-0
Namespace:    default
Priority:     0
Node:         qa-5/10.10.10.215
Start Time:   Sat, 28 Nov 2020 10:55:21 +0800
Labels:       app=seaweedfs
              chart=seaweedfs-helm
              component=master
              controller-revision-hash=seaweedfs-master-85845d55b6
              release=gemini-swfs
              statefulset.kubernetes.io/pod-name=seaweedfs-master-0
Annotations:  cni.projectcalico.org/podIP: 10.42.0.217/32
              cni.projectcalico.org/podIPs: 10.42.0.217/32
Status:       Running
IP:           10.42.0.217
IPs:
  IP:           10.42.0.217
Controlled By:  StatefulSet/seaweedfs-master
Containers:
  seaweedfs:
    Container ID:  docker://f2d00fedbaa159d0a50f00508288c70f5bcb1401362362593c43f43d9de20ca6
    Image:         chrislusf/seaweedfs:dev
    Image ID:      docker-pullable://chrislusf/seaweedfs@sha256:242b4235db3a4d324f898533977bab0b7f2790caec2b5bf47fdfeec85994b191
    Ports:         9333/TCP, 19333/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
      exec /usr/bin/weed \
      master \
      -port=9333 \
      -mdir=/data \
      -volumePreallocate \
      -volumeSizeLimitMB=30000 \
      -ip=${POD_NAME}.${SEAWEEDFS_FULLNAME}-master \
      -peers=${SEAWEEDFS_FULLNAME}-master-0.${SEAWEEDFS_FULLNAME}-master:9333

    State:          Running
      Started:      Sat, 28 Nov 2020 10:55:23 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:9333/cluster/status delay=20s timeout=5s period=30s #success=1 #failure=4
    Readiness:      http-get http://:9333/cluster/status delay=10s timeout=5s period=45s #success=2 #failure=100
    Environment:
      POD_IP:                                 (v1:status.podIP)
      POD_NAME:                              seaweedfs-master-0 (v1:metadata.name)
      NAMESPACE:                             default (v1:metadata.namespace)
      SEAWEEDFS_FULLNAME:                    seaweedfs
      WEED_MASTER_VOLUME_GROWTH_COPY_1:      7
      WEED_MASTER_VOLUME_GROWTH_COPY_2:      6
      WEED_MASTER_VOLUME_GROWTH_COPY_3:      3
      WEED_MASTER_VOLUME_GROWTH_COPY_OTHER:  1
    Mounts:
      /data from data-default (rw)
      /logs/ from seaweedfs-master-log-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bs7vv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  seaweedfs-master-log-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /storage/logs/seaweedfs/master
    HostPathType:  DirectoryOrCreate
  data-default:
    Type:          HostPath (bare host directory volume)
    Path:          /ssd/seaweed-master/
    HostPathType:  DirectoryOrCreate
  default-token-bs7vv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bs7vv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  sw-backend=true
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  17m   default-scheduler  Successfully assigned default/seaweedfs-master-0 to qa-5
  Normal  Pulled     17m   kubelet, qa-5      Container image "chrislusf/seaweedfs:dev" already present on machine
  Normal  Created    17m   kubelet, qa-5      Created container seaweedfs
  Normal  Started    17m   kubelet, qa-5      Started container seaweedfs

kubectl describe pod seaweedfs-volume-0

Name:         seaweedfs-volume-0
Namespace:    default
Priority:     0
Node:         qa-2/10.10.10.212
Start Time:   Sat, 28 Nov 2020 10:55:21 +0800
Labels:       app=seaweedfs
              chart=seaweedfs-helm
              component=volume
              controller-revision-hash=seaweedfs-volume-59df44d7cf
              release=gemini-swfs
              statefulset.kubernetes.io/pod-name=seaweedfs-volume-0
Annotations:  cni.projectcalico.org/podIP: 10.42.1.252/32
              cni.projectcalico.org/podIPs: 10.42.1.252/32
Status:       Running
IP:           10.42.1.252
IPs:
  IP:           10.42.1.252
Controlled By:  StatefulSet/seaweedfs-volume
Containers:
  seaweedfs:
    Container ID:  docker://8ae25d5486e33fecb1e3fee198f099a2ab154f19827f869023c6a97a72d58002
    Image:         chrislusf/seaweedfs:dev
    Image ID:      docker://sha256:3128e0f4a26437e700b9657e3b2c6253dac0f099108f71d27d9900a87147242b
    Ports:         8080/TCP, 18080/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/sh
      -ec
      key=PUBLICURL$(env|grep HOSTNAME |awk -F- '{print $3}')
      publicurl=`eval echo '$'"$key"`
      echo $publicurl
      exec /usr/bin/weed \
        volume \
        -publicUrl=$publicurl \
        -port=8080 \
        -dir=/data \
        -max=0 \
        -ip=${POD_NAME}.${SEAWEEDFS_FULLNAME}-volume \
        -mserver=${SEAWEEDFS_FULLNAME}-master-0.${SEAWEEDFS_FULLNAME}-master:9333

    State:          Running
      Started:      Sat, 28 Nov 2020 10:55:23 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8080/status delay=20s timeout=5s period=90s #success=1 #failure=4
    Readiness:      http-get http://:8080/status delay=15s timeout=5s period=90s #success=1 #failure=100
    Environment:
      PUBLICURL0:          10.10.10.215:32605
      POD_NAME:            seaweedfs-volume-0 (v1:metadata.name)
      NAMESPACE:           default (v1:metadata.namespace)
      HOST_IP:              (v1:status.hostIP)
      SEAWEEDFS_FULLNAME:  seaweedfs
    Mounts:
      /data/ from seaweedfs-volume-storage (rw)
      /logs/ from seaweedfs-volume-log-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-bs7vv (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  seaweedfs-volume-log-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /data/storage/logs/seaweedfs/volume
    HostPathType:  DirectoryOrCreate
  seaweedfs-volume-storage:
    Type:          HostPath (bare host directory volume)
    Path:          /data/storage/object_store/
    HostPathType:  DirectoryOrCreate
  default-token-bs7vv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-bs7vv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  sw-volume=true
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  18m   default-scheduler  Successfully assigned default/seaweedfs-volume-0 to qa-2
  Normal  Pulled     18m   kubelet, qa-2      Container image "chrislusf/seaweedfs:dev" already present on machine
  Normal  Created    18m   kubelet, qa-2      Created container seaweedfs
  Normal  Started    18m   kubelet, qa-2      Started container seaweedfs

$publicurl=10.10.10.215:32605 (10.10.10.215 is k8s master node's ip, 32605 is port exposed by nodeport volume svc )

kubectl describe svc seaweedfs-filer:

Name:                     seaweedfs-filer
Namespace:                default
Labels:                   app=seaweedfs
                          app.kubernetes.io/managed-by=Helm
                          component=filer
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["10.10.10.212"],"port":32601,"protocol":"TCP","serviceName":"default:seaweedfs-filer","allNodes":true},{"addresses":["10.10...
                          meta.helm.sh/release-name: gemini-swfs
                          meta.helm.sh/release-namespace: default
Selector:                 app=seaweedfs,component=filer
Type:                     NodePort
IP:                       10.43.201.144
Port:                     swfs-filer  8888/TCP
TargetPort:               8888/TCP
NodePort:                 swfs-filer  32601/TCP
Endpoints:                10.42.0.216:8888
Port:                     swfs-filer-grpc  18888/TCP
TargetPort:               18888/TCP
NodePort:                 swfs-filer-grpc  32602/TCP
Endpoints:                10.42.0.216:18888
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

kubectl describe svc seaweedfs-master

Name:              seaweedfs-master
Namespace:         default
Labels:            app=seaweedfs
                   app.kubernetes.io/managed-by=Helm
                   component=master
Annotations:       meta.helm.sh/release-name: gemini-swfs
                   meta.helm.sh/release-namespace: default
                   service.alpha.kubernetes.io/tolerate-unready-endpoints: true
Selector:          app=seaweedfs,component=master
Type:              ClusterIP
IP:                None
Port:              swfs-master  9333/TCP
TargetPort:        9333/TCP
Endpoints:         10.42.0.217:9333
Port:              swfs-master-grpc  19333/TCP
TargetPort:        19333/TCP
Endpoints:         10.42.0.217:19333
Session Affinity:  None
Events:            <none>

kubectl describe svc seaweedfs-volume

Name:              seaweedfs-volume
Namespace:         default
Labels:            app=seaweedfs
                   app.kubernetes.io/managed-by=Helm
                   component=volume
Annotations:       meta.helm.sh/release-name: gemini-swfs
                   meta.helm.sh/release-namespace: default
Selector:          app=seaweedfs,component=volume
Type:              ClusterIP
IP:                None
Port:              swfs-volume  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.42.1.252:8080
Port:              swfs-volume-18080  18080/TCP
TargetPort:        18080/TCP
Endpoints:         10.42.1.252:18080
Session Affinity:  None
Events:            <none>

kubectl describe svc seaweedfs-volume-0-external

Name:                     seaweedfs-volume-0-external
Namespace:                default
Labels:                   app=seaweedfs
                          app.kubernetes.io/managed-by=Helm
                          component=volume
                          pod=seaweedfs-volume-0
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["10.10.10.212"],"port":32605,"protocol":"TCP","serviceName":"default:seaweedfs-volume-0-external","allNodes":true}]
                          meta.helm.sh/release-name: gemini-swfs
                          meta.helm.sh/release-namespace: default
Selector:                 app=seaweedfs,component=volume,statefulset.kubernetes.io/pod-name=seaweedfs-volume-0
Type:                     NodePort
IP:                       10.43.140.161
Port:                     swfs-volume-external  8181/TCP
TargetPort:               8080/TCP
NodePort:                 swfs-volume-external  32605/TCP
Endpoints:                10.42.1.252:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

l try again, after setup seaweedfs in k8s , only the filer outputs the logs when i push the image for the first time. What's more, when I execute the push the image command, the filer will immediately print the following log:

I1128 03:07:16     1 filer_grpc_server_sub_meta.go:166] => client mount@10.10.10.215:7134: rpc error: code = Unavailable desc = transport is closing
E1128 03:07:16     1 filer_grpc_server_sub_meta.go:49] processed to 2020-11-28 03:07:16.046269865 +0000 UTC: rpc error: code = Unavailable desc = transport is closing
I1128 03:07:19     1 filer_grpc_server_sub_meta.go:180] - listener mount@10.10.10.215:7134

After the first push image failed,try to push image several times, still failed, and there are no logs output.

@paochiang
Copy link
Contributor Author

paochiang commented Nov 28, 2020

In k8s, when I set the port of publicUrl and nodePort of external as 8080, the image can be pushed successfully.

# svc
seaweedfs-filer                             NodePort       10.43.43.96     <none>        8888:32601/TCP,18888:32602/TCP                                            3s
seaweedfs-master                            ClusterIP      None            <none>        9333/TCP,19333/TCP                                                        3s
seaweedfs-volume                            ClusterIP      None            <none>        8080/TCP,18080/TCP                                                        3s
seaweedfs-volume-0-external                 NodePort       10.43.171.96    <none>        8181:8080/TCP                                                             3s

The publicUrlof the volume server startup param is 10.10.10.215:8080

Then I tested the seaweedfs installed in docker with the version you fixed:

Added a fix a102157

Please try maybe 15 minutes later with chrislusf/seaweedfs:dev

And i found that once the port of volume and publicUrl are changed to non-8080, there is still a problem with pushing the image.
You can reproduce by the following process:

System Setup
version: a102157
seaweedfs-compose.yml:

version: '2'

services:
  master:
    image: chrislusf/seaweedfs:dev
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master"
  volume:
    image: chrislusf/seaweedfs:dev
    ports:
      - 7272:7272
      - 18080:18080
    command: "volume -mserver=master:9333 -port=7272 -ip=volume -publicUrl=localhost:7272 -max=0"
    depends_on:
      - master
  filer:
    image: chrislusf/seaweedfs:dev
    ports:
      - 8888:8888
      - 18888:18888
    command: 'filer -master="master:9333"'
    depends_on:
      - master
      - volume

docker-registry-config.yml:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /data
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

run:

# set up seaweed
docker-compose -f seaweedfs-compose.yml -p seaweedfs up

# use weed mount to host path
sudo mkdir -p /mnt/registry
sudo ./weed mount -outsideContainerClusterMode -cacheCapacityMB=0 -filer=127.0.0.1:8888 -filer.path=/registry -dir=/mnt/registry

# setup docker registry
docker run -d -p 5000:5000 --name localregistry -v /home/orion/lbj/localswfs/docker-registry-config.yml:/etc/docker/registry/config.yml -v /mnt/registry:/data > registry:2

Additional context
1、docker tag chrislusf/seaweedfs:dev 127.0.0.1:5000/chrislusf/seaweedfs:dev
2、docker push 127.0.0.1:5000/chrislusf/seaweedfs:dev

The push refers to repository [127.0.0.1:5000/chrislusf/seaweedfs]
b073ce872256: Pushing [==================================================>]  3.584kB
016661022a84: Pushing  2.048kB
771aacfe434b: Pushing [==================================================>]  3.072kB
1cb39e39e7a3: Pushing  2.048kB
7e5a61287dd6: Pushing [===========================>                       ]  40.08MB/73.74MB
ace0eda3e3be: Waiting

3、Pushing image is stuck

So there may be something wrong with port 8080 or pubilcUrl

@chrislusf
Copy link
Collaborator

volume:
image: chrislusf/seaweedfs:dev
ports:
- 7272:7272
- 18080:18080
command: "volume -mserver=master:9333 -port=7272 -ip=volume -publicUrl=localhost:7272 -max=0"
depends_on:
- master

use corresponding grpc port.

     ports:
       - 7272:7272
       - 17272:17272

@paochiang
Copy link
Contributor Author

paochiang commented Nov 30, 2020

It works.

And I tested in K8s, I found that port in volume server's publicUrl must be equal to the volume server's port, nodePort of svc:seaweedfs-volume-0-external must expose volume server's port and grpcPort. (
I guess the reason is that outsideContainerClusterMode needs to use grpcPort, and the value of grpcPort is obtained by adding 10000 to the value of port in publicUrl

If I setup multiple volume servers, it means I need to start the volume servers with different ports.

Now k8s does not support obtaining pod id in statefulset: kubernetes/kubernetes#40651, so it is difficult to start multiple volume servers with different ports.

So I want to know if the system can be optimized to close the association between the port of publicUrl and the port 、grpcPort of volume server , to use separate port in publicUrl (May be only need to use a separate public grpcPort)?

@chrislusf
Copy link
Collaborator

Can you try https://github.com/seaweedfs/seaweedfs-operator ? It should do everything for you. If not, please help to fix it.

@paochiang
Copy link
Contributor Author

ok, I will try seaweedfs-operator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants