filer does not work correctly with etcd storage

[whiterm]

Describe the bug
Docker registry works fine when using the docker-compose configuration from https://github.com/chrislusf/seaweedfs/blob/master/docker/compose/local-registry-compose.yml
But when I include etcd as a filer storage, it doesn't work.

System Setup
I have prepared a new docker-compose. Place this file in the docker/compose directory in the project.

version: '2'

services:
  etcd:
    image: quay.io/coreos/etcd:v3.5.4
    command: "etcd --advertise-client-urls http://etcd:2379 --listen-client-urls http://0.0.0.0:2379"
    ports:
      - 2379:2379
  master:
    image: chrislusf/seaweedfs:local
    ports:
      - 9333:9333
      - 19333:19333
    command: "master -ip=master -volumeSizeLimitMB=1024"
  volume:
    image: chrislusf/seaweedfs:local
    ports:
      - 8080:8080
      - 18080:18080
    command: "volume -mserver=master:9333 -port=8080 -ip=volume -max=0 -preStopSeconds=1"
    depends_on:
      - master
  s3:
    image: chrislusf/seaweedfs:local
    ports:
      - 8888:8888
      - 18888:18888
      - 8333:8333
    command: '-v 9 filer -master="master:9333" -s3 -s3.config=/etc/seaweedfs/s3.json -s3.port=8333'
    environment:
      WEED_LEVELDB2_ENABLED: 'false'
      WEED_ETCD_ENABLED: 'true'
      WEED_ETCD_SERVERS: "http://etcd:2379"
    volumes:
      - ./s3.json:/etc/seaweedfs/s3.json
    depends_on:
      - etcd
      - master
      - volume
  registry:
    image: registry:2
    environment:
      REGISTRY_HTTP_ADDR: "0.0.0.0:5001" # seaweedfs s3
      REGISTRY_LOG_LEVEL: "debug"
      REGISTRY_STORAGE: "s3"
      REGISTRY_STORAGE_S3_REGION: "us-east-1"
      REGISTRY_STORAGE_S3_REGIONENDPOINT: "http://s3:8333"
      REGISTRY_STORAGE_S3_BUCKET: "registry"
      REGISTRY_STORAGE_S3_ACCESSKEY: "some_access_key1"
      REGISTRY_STORAGE_S3_SECRETKEY: "some_secret_key1"
      REGISTRY_STORAGE_S3_V4AUTH: "true"
      REGISTRY_STORAGE_S3_SECURE: "false"
      REGISTRY_STORAGE_S3_SKIPVERIFY: "true"
      REGISTRY_STORAGE_S3_ROOTDIRECTORY: "/"
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
      REGISTRY_STORAGE_REDIRECT_DISABLE: "true"
      REGISTRY_VALIDATION_DISABLED: "true"
    ports:
      - 5001:5001
    depends_on:
      - s3

When I push an image to the registry docker push 127.0.0.1:5001/test:local. The command ends with an error. And I see the strange errors in the logs. It looks like s3 storage doesn't work correctly.

Expected behavior
It was expected that docker push and docker pull worked

Logs

s3_1        | I0505 22:30:40     1 filer_util.go:65] delete entry /buckets/registry/.uploads/196942c2c251e68445373035f71649f9c0bb1ea4: directory:"/buckets/registry/.uploads" name:"196942c2c251e68445373035f71649f9c0bb1ea4" is_recursive:true ignore_recursive_error:true
s3_1        | I0505 22:30:40     1 auth_credentials.go:219] v4 auth type
s3_1        | I0505 22:30:40     1 filer_grpc_server.go:280] DeleteEntry directory:"/buckets/registry/.uploads" name:"196942c2c251e68445373035f71649f9c0bb1ea4" is_recursive:true ignore_recursive_error:true
s3_1        | I0505 22:30:40     1 auth_credentials.go:248] user name: some_admin_user actions: [Admin Read List Tagging Write], action: List
s3_1        | I0505 22:30:40     1 s3api_objects_list_handlers.go:99] ListObjectsV1Handler registry
s3_1        | I0505 22:30:40     1 filer_grpc_server.go:40] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/test4/_uploads/f766278e-3c90-41bf-b849-4c554f5854c2" prefix:"data" limit:2
s3_1        | I0505 22:30:40     1 filer_delete_entry.go:111] deleting directory /buckets/registry/.uploads/196942c2c251e68445373035f71649f9c0bb1ea4 delete chunks: false
s3_1        | I0505 22:30:40     1 filer_client.go:167] exists entry /buckets/registry: directory:"/buckets" name:"registry"
s3_1        | I0505 22:30:40     1 filer_delete_entry.go:124] deleting entry /buckets/registry/.uploads/196942c2c251e68445373035f71649f9c0bb1ea4, delete chunks: false
s3_1        | I0505 22:30:40     1 filer_grpc_server.go:22] LookupDirectoryEntry /buckets/registry
s3_1        | I0505 22:30:40     1 error_handler.go:85] status 200 application/xml: <?xml version="1.0" encoding="UTF-8"?>
s3_1        | <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>registry</Name><Prefix>docker/registry/v2/repositories/test4/_uploads/f766278e-3c90-41bf-b849-4c554f5854c2/data</Prefix><Marker></Marker><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
s3_1        | I0505 22:30:40     1 s3api_object_multipart_handlers.go:86] CompleteMultipartUploadHandler<?xml version="1.0" encoding="UTF-8"?>
s3_1        | <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>registry</Bucket><ETag>&#34;b21ea8664644eb131499729bd8f9fb12&#34;</ETag><Key>docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data</Key><Location>http://172.27.0.5:8888/buckets/registry/docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data</Location></CompleteMultipartUploadResult>0
s3_1        | I0505 22:30:40     1 error_handler.go:85] status 200 application/xml: <?xml version="1.0" encoding="UTF-8"?>
s3_1        | <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Bucket>registry</Bucket><ETag>&#34;b21ea8664644eb131499729bd8f9fb12&#34;</ETag><Key>docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data</Key><Location>http://172.27.0.5:8888/buckets/registry/docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data</Location></CompleteMultipartUploadResult>
s3_1        | <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>registry</Name><Prefix>docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data</Prefix><Marker></Marker><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
registry_1  | time="2022-05-05T22:30:40.835652197Z" level=debug msg="s3aws.Move("/docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data", "/docker/registry/v2/blobs/sha256/3e/3e963259b22d111f4082349fa0787f33e4c9ca772ec7c466a52e000accd7157a/data"" go.version=go1.16.15 http.request.contenttype="application/octet-stream" http.request.host="127.0.0.1:5001" http.request.id=87617464-36e9-42f2-a533-f8a82d8dbcb3 http.request.method=PUT http.request.remoteaddr="172.27.0.1:60992" http.request.uri="/v2/test4/blobs/uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3?_state=yk9i2DUssPvZ_G6dKYzVSDJLFaV9VX-ZBe9swekN7kl7Ik5hbWUiOiJ0ZXN0NCIsIlVVSUQiOiJiODcyMzYzOC0xOGY3LTRlMzUtYWFhMC1jYzRiNTlhY2QxYTMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDUtMDVUMjI6MzA6NDAuNjc2MTgwMzg4WiJ9&digest=sha256%3A3e963259b22d111f4082349fa0787f33e4c9ca772ec7c466a52e000accd7157a" http.request.useragent="buildkit/v0.10-dev" trace.duration=20.024184ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Move" trace.id=d780acfd-6e7d-42d3-a675-57d8e752612a trace.line=185 vars.name=test4 vars.uuid=b8723638-18f7-4e35-aaa0-cc4b59acd1a3 
registry_1  | time="2022-05-05T22:30:40.835935371Z" level=error msg="unknown error completing upload: s3aws: Path not found: /docker/registry/v2/repositories/test4/_uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3/data" go.version=go1.16.15 http.request.contenttype="application/octet-stream" http.request.host="127.0.0.1:5001" http.request.id=87617464-36e9-42f2-a533-f8a82d8dbcb3 http.request.method=PUT http.request.remoteaddr="172.27.0.1:60992" http.request.uri="/v2/test4/blobs/uploads/b8723638-18f7-4e35-aaa0-cc4b59acd1a3?_state=yk9i2DUssPvZ_G6dKYzVSDJLFaV9VX-ZBe9swekN7kl7Ik5hbWUiOiJ0ZXN0NCIsIlVVSUQiOiJiODcyMzYzOC0xOGY3LTRlMzUtYWFhMC1jYzRiNTlhY2QxYTMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDUtMDVUMjI6MzA6NDAuNjc2MTgwMzg4WiJ9&digest=sha256%3A3e963259b22d111f4082349fa0787f33e4c9ca772ec7c466a52e000accd7157a" http.request.useragent="buildkit/v0.10-dev" vars.name=test4 vars.uuid=b8723638-18f7-4e35-aaa0-cc4b59acd1a3 

[chrislusf]

thanks for the detailed report!

[whiterm]

So fast fixing! Awesome!