Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[filer] Internal looping ListDirectoryPrefixedEntries (Critical) #1701

Closed
kmlebedev opened this issue Dec 23, 2020 · 14 comments
Closed

[filer] Internal looping ListDirectoryPrefixedEntries (Critical) #1701

kmlebedev opened this issue Dec 23, 2020 · 14 comments

Comments

@kmlebedev
Copy link
Contributor

Describe the bug
CPU load 100%
and internal looping ListDirectoryPrefixedEntries

System Setup
2.16 from master branch

Additional context

I1223 16:53:28     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256 from  prefix  limit 8
I1223 16:53:28     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1223 16:53:28     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1223 16:53:28     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1223 16:53:28     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256 from  prefix  limit 8
I1223 16:53:28     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1223 16:53:28     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1223 16:53:28     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256 from  prefix  limit 8
I1223 16:53:28     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
chrislusf added a commit that referenced this issue Dec 23, 2020
@chrislusf
s3: break loop if error
db62090 
fix #1701
@kmlebedev
Copy link
Contributor Author

kmlebedev commented Dec 24, 2020 •
edited

@chrislusf
the cycle starts some time after the start
And star after disk_location check

I1224 06:12:53     1 disk_location.go:367] dir /data freePercent 99.99% < min 11.00%, isLowDiskSpace: false
I1224 06:12:53     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256 from  prefix  limit 8
I1224 06:12:53     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256 from  prefix  limit 8
I1224 06:12:53     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256 from  prefix  limit 8
I1224 06:12:53     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8
I1224 06:12:53     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/22c98200-169f-4535-b013-e87a63c598f8/hashstates/sha256" limit:8

@chrislusf
Copy link
Collaborator

The error should be started from s3. Do you have logs there?

@chrislusf
Copy link
Collaborator

It is somewhere around this loop. Maybe you can put some println to see what is going on?

0ca9d89

@kmlebedev
Copy link
Contributor Author

The error should be started from s3. Do you have logs there?

The error comes from the inside without S3 requests
Try

  1. make dev_registry
  2. docker push to localhost
  3. And wait 15 minutes

@kmlebedev
Copy link
Contributor Author

It is somewhere around this loop. Maybe you can put some println to see what is going on?

0ca9d89

ok, Done

@chrislusf
Copy link
Collaborator

  1. make dev_registry

the make rule does not exist yet.

@kmlebedev
Copy link
Contributor Author

  1. make dev_registry

the make rule does not exist yet.

See https://github.com/chrislusf/seaweedfs/blob/master/docker/Makefile#L18

@kmlebedev
Copy link
Contributor Author

kmlebedev commented Dec 24, 2020 •
edited

Catch, but without my debug message for loop 0ca9d89
It looks like it's different loops

Index: weed/s3api/s3api_objects_list_handlers.go
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- weed/s3api/s3api_objects_list_handlers.go	(revision 3be363579971470fff1acc8ebf668a3f3851fa33)
+++ weed/s3api/s3api_objects_list_handlers.go	(date 1608792648752)
@@ -310,6 +310,7 @@
 
 func (s3a *S3ApiServer) isDirectoryAllEmpty(filerClient filer_pb.SeaweedFilerClient, parentDir, name string) (isEmpty bool, err error) {
 	// println("+ isDirectoryAllEmpty", dir, name)
+	glog.V(8).Infof("isDirectoryAllEmpty() start parentDir: %s", parentDir)
 	var fileCounter int
 	var subDirs []string
 	currentDir := parentDir + "/" + name
@@ -324,6 +325,7 @@
 			}
 			startFrom = entry.Name
 			isExhausted = isExhausted || isLast
+			glog.V(8).Infof("filer_pb.SeaweedList() startFrom: %s isExhausted: %t", startFrom, isExhausted, )
 			return nil
 		}, startFrom, false, 8)
 	}
@@ -338,6 +340,8 @@
 
 	for _, subDir := range subDirs {
 		isSubEmpty, subErr := s3a.isDirectoryAllEmpty(filerClient, currentDir, subDir)
+		glog.V(8).Infof("s3a.isDirectoryAllEmpty() subDir: %s, isSubEmpty: %t", subDir, isSubEmpty)
+
 		if subErr != nil {
 			return false, subErr
 		}

These are logs from all over the cluster

I1224 07:50:04     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256 from  prefix  limit 8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256 from  prefix  limit 8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filerstore_wrapper.go:224] ListDirectoryPrefixedEntries /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256 from  prefix  limit 8

  | Dec 24, 2020 @ 12:50:06.092 | I1224 07:50:04     1 filer_client.go:113] read directory: directory:"/buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates/sha256"  limit:8

@chrislusf
Copy link
Collaborator

can you put a debug message here: 0ca9d89#diff-957019626356063c1e721cb807679d634905f6ca0cd3ff3276ee4e7b08e3ef3aL319

@kmlebedev
Copy link
Contributor Author

kmlebedev commented Dec 24, 2020 •
edited

can you put a debug message here: 0ca9d89#diff-957019626356063c1e721cb807679d634905f6ca0cd3ff3276ee4e7b08e3ef3aL319

Found


Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library
-- | --

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: _manifests isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: _layers isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: _uploads isExhausted: true

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: _layers, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:50:02.661 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: _manifests isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: seaweedfs isExhausted: true

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: _layers isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: _uploads isExhausted: true

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: _layers, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.067 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: seaweedfs, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:50:03.068 | I1224 07:50:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:51:02.670 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 12:51:02.671 | I1224 07:51:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 12:53:20.757 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 12:53:20.758 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 12:53:20.758 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 12:53:20.758 | I1224 07:53:03     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:53:20.758 | I1224 07:53:03     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

  | Dec 24, 2020 @ 12:53:42.093 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:53:42.093 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:53:42.093 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 12:53:42.094 | I1224 07:52:02     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:57:36.257 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:57:36.258 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:57:36.258 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.171 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 12:57:37.174 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 12:57:37.174 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:57:37.174 | I1224 07:56:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

  | Dec 24, 2020 @ 12:57:37.174 | I1224 07:56:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 12:59:19.218 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:59:19.218 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:59:19.218 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:59:19.222 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 12:59:19.223 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 12:59:19.223 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.223 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 12:59:19.226 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 12:59:19.226 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 12:59:19.226 | I1224 07:58:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 12:59:19.226 | I1224 07:58:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 12:59:19.226 | I1224 07:58:05     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 12:59:19.227 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 12:59:20.159 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 12:59:20.159 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 12:59:20.159 | I1224 07:58:05     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 12:59:20.159 | I1224 07:58:05     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

  | Dec 24, 2020 @ 13:01:19.145 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 13:01:19.145 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 13:01:19.145 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: a46437b0f3cbdbc29cb80a5890047e790ff8adc433bd6707136bc83763c1295f isExhausted: true

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 7491ee03c411253067992734734a9ffcaf42d314b7ad5db7ffe06ecb5694b218 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 0bac2e4b89091570e07bd531a271c8fed47ada44b4ab0e38af34be6e2fa30317 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 55373631bb3a66d0f3913e626efb9385879af24ae77a62047d5f1f79825df15f isExhausted: false

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 65ee1ed4006c92fa0b21b346476a7eb1179d8d73d6e6a48983480d57d1d0b067 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 66df2ee0176c18db0ba858710725e466367876bb373e3a5007421d127c672664 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_layers/sha256

  | Dec 24, 2020 @ 13:01:19.146 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: revisions isExhausted: false

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions/sha256

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: tags isExhausted: true

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 32c9f9f34687be0b90d7bd292927380be9d1d40b1c743749b58d95688f873655 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 05fe61834e280f6c1d5e764a69cba9ddc0693d09adfe9dfdd6479a5ab03a9f96, isSubEmpty: false

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_manifests/revisions

  | Dec 24, 2020 @ 13:01:19.148 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: ff0cae7403618e02a89e4d7b55da4be58eff1245872f3790cfccdc537e7c331e isExhausted: true

  | Dec 24, 2020 @ 13:01:19.149 | I1224 08:00:09     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: sha256, isSubEmpty: false

  | Dec 24, 2020 @ 13:01:19.149 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs

  | Dec 24, 2020 @ 13:01:19.149 | I1224 08:00:09     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: 1f0e6e3283b149aeec60a90bcadd0907b0696ff892c9d47d284ae1c463484278, isSubEmpty: false

  | Dec 24, 2020 @ 13:01:19.149 | I1224 08:00:09     1 s3api_objects_list_handlers.go:343] s3a.isDirectoryAllEmpty() subDir: revisions, isSubEmpty: false

  | Dec 24, 2020 @ 13:01:19.149 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: link isExhausted: true

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 2ef85625-7f78-4a00-8ab7-84ca3c512833 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 3dc16aaf-7504-4aac-b36f-7d4290a8a6dd isExhausted: false

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 651347be-7a5f-4d34-99be-9656c039a430 isExhausted: true

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 22c98200-169f-4535-b013-e87a63c598f8 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 37bef79f-f9c2-4247-91fd-f27139fa60cb isExhausted: false

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: hashstates isExhausted: true

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 5f068edd-7533-4bba-855c-5557e18b2439 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 1737907a-41d1-4a07-b1d1-4730b44f99c8 isExhausted: false

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads

  | Dec 24, 2020 @ 13:01:19.151 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: 4ec55a06-7af7-4dc7-84d5-a1c4107e1c7e isExhausted: false

  | Dec 24, 2020 @ 13:01:21.165 | I1224 08:00:09     1 s3api_objects_list_handlers.go:328] filer_pb.SeaweedList() startFrom: sha256 isExhausted: true

  | Dec 24, 2020 @ 13:01:21.165 | I1224 08:00:09     1 s3api_objects_list_handlers.go:313] isDirectoryAllEmpty() start parentDir: /buckets/registry/docker/registry/v2/repositories/library/seaweedfs/_uploads/1737907a-41d1-4a07-b1d1-4730b44f99c8/hashstates

@chrislusf
Copy link
Collaborator

can you also share how you added the logging?

@kmlebedev kmlebedev mentioned this issue Dec 24, 2020
Closed
@kmlebedev
Copy link
Contributor Author

can you also share how you added the logging?

https://github.com/chrislusf/seaweedfs/pull/1702/files

@chrislusf
Copy link
Collaborator

I added more logs. 1620de0

Seems I reproduced once, but not yet able to reproduce it again.

chrislusf added a commit that referenced this issue Dec 24, 2020
@chrislusf
s3: avoid looping if the directory is empty
8e48a23 
fix #1701
@chrislusf
Copy link
Collaborator

I think I found the problem. Added a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chrislusf @kmlebedev