Critical - S3 Delete Multiple Objects, delete the bucket

[LazyDBA247-Anyvision]

Hi @chrislusf

SW version 2.24+

no issue on 2.23 (except empty folders are been left alone)

609daaf

created a bug, that deletes the "links" to a bucket and then removes the volumes, when we delete 2+ objects from any path inside the bucket... (using DeleteMultipleObjectsHandler)

s3+filer level 4 log:

I0215 12:02:07     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/subjectsBucket"  prefix:"82e48cb8-68d3-4690-a4d3-b4e41cdcff00"  limit:10001
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket from  prefix 82e48cb8-68d3-4690-a4d3-b4e41cdcff00 limit 1024
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket from 82e48cb8-68d3-4690-a4d3-b4e41cdcff00 prefix 82e48cb8-68d3-4690-a4d3-b4e41cdcff00 limit 1024
I0215 12:02:07     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00"  limit:10001
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 from  prefix  limit 1024
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 from backup prefix  limit 1024
I0215 12:02:07     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup"  limit:10000
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup from  prefix  limit 1024
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup from d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b prefix  limit 1024
I0215 12:02:07     1 filer_grpc_server.go:49] ListEntries directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"  limit:10000
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b from  prefix  limit 1024
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b from metadata.json prefix  limit 1024
I0215 12:02:07     1 s3api_handlers.go:89] status 200 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>subjectsBucket</Name><Prefix>82e48cb8-68d3-4690-a4d3-b4e41cdcff00</Prefix><MaxKeys>10000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg</Key><ETag>&#34;f132d0b02b96e17a1b1fae01ecf97d49&#34;</ETag><Size>7355</Size><Owner><ID>0</ID></Owner><StorageClass>STANDARD</StorageClass><LastModified>2021-02-15T11:57:37Z</LastModified></Contents><Contents><Key>82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg</Key><ETag>&#34;850a611de167e0978c8305f7956ac61e&#34;</ETag><Size>7156</Size><Owner><ID>0</ID></Owner><StorageClass>STANDARD</StorageClass><LastModified>2021-02-15T11:57:38Z</LastModified></Contents><Contents><Key>82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json</Key><ETag>&#34;2c4f4c72d2ae9b2a6b6d16557b0ed6a7&#34;</ETag><Size>74</Size><Owner><ID>0</ID></Owner><StorageClass>STANDARD</StorageClass><LastModified>2021-02-15T11:57:38Z</LastModified></Contents><KeyCount>3</KeyCount></ListBucketResult>
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg: directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00"  name:"705c8eec-00000-FaceImg-00001.jpg"  is_delete_data:true
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00"  name:"705c8eec-00000-FaceImg-00001.jpg"  is_delete_data:true
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg, delete chunks: true
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg: directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"  name:"image.jpeg"  is_delete_data:true
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"  name:"image.jpeg"  is_delete_data:true
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg, delete chunks: true
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json: directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"  name:"metadata.json"  is_delete_data:true
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"  name:"metadata.json"  is_delete_data:true
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json, delete chunks: true
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b: directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup"  name:"d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup"  name:"d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b"
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b from  prefix  limit 1025
I0215 12:02:07     1 filer_delete_entry.go:112] deleting directory /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b delete 0 chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:193] DeleteFolderChildren /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b, delete chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00: directory:"/buckets/subjectsBucket"  name:"82e48cb8-68d3-4690-a4d3-b4e41cdcff00"
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket"  name:"82e48cb8-68d3-4690-a4d3-b4e41cdcff00"
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 from  prefix  limit 1025
E0215 12:02:07     1 filer_delete_entry.go:79] deleting a folder /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 has children: backup ...
I0215 12:02:07     1 filer_delete_entry.go:35] delete directory /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00: fail to delete non-empty folder: /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00
I0215 12:02:07     1 s3api_object_handlers.go:238] directory /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 has 1 deletion but still not empty: delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00: delete directory /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00: fail to delete non-empty folder: /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup: directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00"  name:"backup"
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00"  name:"backup"
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup from  prefix  limit 1025
I0215 12:02:07     1 filer_delete_entry.go:112] deleting directory /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup delete 0 chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:193] DeleteFolderChildren /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup, delete chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00: directory:"/buckets/subjectsBucket"  name:"82e48cb8-68d3-4690-a4d3-b4e41cdcff00"
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets/subjectsBucket"  name:"82e48cb8-68d3-4690-a4d3-b4e41cdcff00"
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 from  prefix  limit 1025
I0215 12:02:07     1 filer_delete_entry.go:112] deleting directory /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00 delete 0 chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:193] DeleteFolderChildren /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00, delete chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket/82e48cb8-68d3-4690-a4d3-b4e41cdcff00
I0215 12:02:07     1 filer_util.go:60] delete entry /buckets/subjectsBucket: directory:"/buckets"  name:"subjectsBucket"
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/buckets"  name:"subjectsBucket"
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets/subjectsBucket
I0215 12:02:07     1 filer_delete_entry.go:112] deleting directory /buckets/subjectsBucket delete 0 chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:193] DeleteFolderChildren /buckets/subjectsBucket
I0215 12:02:07     1 filer_delete_entry.go:125] deleting entry /buckets/subjectsBucket, delete chunks: false
I0215 12:02:07     1 filerstore_wrapper.go:181] DeleteOneEntry /buckets/subjectsBucket
I0215 12:02:07     1 filer_util.go:60] delete entry //buckets: directory:"/"  name:"buckets"
I0215 12:02:07     1 filer_grpc_server.go:323] DeleteEntry directory:"/"  name:"buckets"
I0215 12:02:07     1 filerstore_wrapper.go:129] FindEntry /buckets
I0215 12:02:07     1 filerstore_wrapper.go:220] ListDirectoryPrefixedEntries /buckets from  prefix  limit 1025
E0215 12:02:07     1 filer_delete_entry.go:79] deleting a folder /buckets has children: infra ...
I0215 12:02:07     1 filer_delete_entry.go:35] delete directory /buckets: fail to delete non-empty folder: /buckets
I0215 12:02:07     1 s3api_object_handlers.go:238] directory /buckets has 1 deletion but still not empty: delete entry //buckets: delete directory /buckets: fail to delete non-empty folder: /buckets
I0215 12:02:07     1 s3api_handlers.go:89] status 200 application/xml: <?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Deleted><Key>82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg</Key></Deleted><Deleted><Key>82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg</Key></Deleted><Deleted><Key>82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json</Key></Deleted></DeleteResult>
I0215 12:02:09     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 73
I0215 12:02:09     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 74
I0215 12:02:09     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 76
I0215 12:02:09     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 75

regards.

[LazyDBA247-Anyvision]

that are the paths we sent:

Bucket subjectsBucket
Objects:
[
  {
    Key: '82e48cb8-68d3-4690-a4d3-b4e41cdcff00/705c8eec-00000-FaceImg-00001.jpg'
  },
  {
    Key: '82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/image.jpeg'
  },
  {
    Key: '82e48cb8-68d3-4690-a4d3-b4e41cdcff00/backup/d7dcb021-ae60-4fbe-88c6-3fbe906d3e7b/metadata.json'
  }
]

and it deleted the subjectsBucket with many other Non-Empty Folders

[LazyDBA247-Anyvision]

when deleting multiple in a single folder, and not root folder (in the bucket) all ok...
when deleting multiple in a single folder, and root folder, deletes the buckets and tries to delete the /buckets also :)

command:
aws --endpoint-url http://192.168.50.9:8333 s3api delete-objects --bucket deleteBug --delete '{"Objects":[{"Key":"rootDir/1.txt"},{"Key":"rootDir/2.txt"}]}'

I0215 18:52:40     1 filer_util.go:60] delete entry /buckets/deleteBug/rootDir/1.txt: directory:"/buckets/deleteBug/rootDir"  name:"1.txt"  is_delete_data:true
I0215 18:52:40     1 filer_util.go:60] delete entry /buckets/deleteBug/rootDir/2.txt: directory:"/buckets/deleteBug/rootDir"  name:"2.txt"  is_delete_data:true
I0215 18:52:40     1 filer_util.go:60] delete entry /buckets/deleteBug/rootDir: directory:"/buckets/deleteBug"  name:"rootDir"
I0215 18:52:40     1 filer_util.go:60] delete entry /buckets/deleteBug: directory:"/buckets"  name:"deleteBug"
I0215 18:52:40     1 filer_util.go:60] delete entry //buckets: directory:"/"  name:"buckets"
E0215 18:52:40     1 filer_delete_entry.go:79] deleting a folder /buckets has children: 14-02-2021 ...
I0215 18:52:40     1 filer_delete_entry.go:35] delete directory /buckets: fail to delete non-empty folder: /buckets
I0215 18:52:42     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 42
I0215 18:52:42     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 44
I0215 18:52:42     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 41
I0215 18:52:42     1 masterclient.go:142] filer: seaweedfs-volume-0.seaweedfs-volume:8080 masterClient removes volume 43

[chrislusf]

Thanks for the detailed report! Added a fix and updated the release.

[LazyDBA247-Anyvision]

Thanks for the quick fix, verified working.