[working as intended] Cannot remove folders in bucket when versioning is enabled or suspended.

[tomsozolins]

Hello!

mc version RELEASE.2020-11-17T00-39-14Z
minio version RELEASE.2020-11-13T20-10-18Z

I have a problem that i can't delete folder from MinIO. Below is sample directory structure:

When i click remove in MinIO browser, the directory dissapears, but when i click refresh it appears back.
I tried to do recursive delete with MC client and it shows error:
mc: <ERROR> Failed to remove node1/rpptest/. Object does not exist

Tried to restart minio service on all 3 nodes, but still the problem persists.

Now i have these two dangling directories which i can't remove:

[harshavardhana]

@tomsozolins mc admin heal -r --remove

[tomsozolins]

mc admin heal -r --remove

Ran it 3 times, but still when i try to delete both of the folders they appear back after refresh.

[harshavardhana]

Most probably some stale content left over from list cache.

[tomsozolins]

Most probably some stale content left over from list cache.

Is this normal behavior? Will the cache clean automatically? It seems like, the recursive function deleted all the object files but all the directories remained untouched. I can still browse these directories with TAB in mc client.

Tried to put object inside the directory. Then deleted the directory. The directory remained intact, but the object was deleted.

[tomsozolins]

Tried another test, i deleted all files from the problematic bucket from file system level and then the directories disappeared from the bucket.
Then i tried to upload the same directory again and the same problem happens. When delete button is clicked, all objects in directory are deleted, but directories stay untouched.
I'm uploading from Windows 10 OS using Chrome browser.

I can confirm, that this problem wasn't happening on previous versions of MinIO with the exact same directory.

[harshavardhana]

Then i tried to upload the same directory again and the same problem happens. When delete button is clicked, all objects in directory are deleted, but directories stay untouched.

if the directories were created using PUT with / at the end then they will be preserved since those we are different than the objects.

[tomsozolins]

Then i tried to upload the same directory again and the same problem happens. When delete button is clicked, all objects in directory are deleted, but directories stay untouched.

if the directories were created using PUT with / at the end then they will be preserved since those we are different than the objects.

Directories were created automatically by MinIO Browser drag and drop upload.

[tomsozolins]

Yesterday i tested ILM policy and it behaves the same. All object files were deleted from directories, but all directories remained undeleted. This is not expected behavior.
I am planning to deploy ~300TB cluster and there will thousands if not millions of these directories uploaded from Windows Clients using MinIO browser.

[tomsozolins]

I can confirm that this bug is happening on completely separate new single node MinIO server.
minio version RELEASE.2020-11-13T20-10-18Z
OS: CentOS-8-8.2.2004-20200611.2
Chrome: Version 86.0.4240.198 (Official Build) (64-bit)

[annkam]

Hello! I can confirm that this problem exists for a long time. We've been experiencing it with our 12 nodes cluster in RELEASE.2020-05-01T22-19-14Z. The only solution for now is to delete such directories from the file system. Use some configuration management tool to perform this action on all nodes at once in distributed setups.
Moreover, if such irremovable directories exist, you can't delete a bucket.

[tomsozolins]

Hello! I can confirm that this problem exists for a long time. We've been experiencing it with our 12 nodes cluster in RELEASE.2020-05-01T22-19-14Z. The only solution for now is to delete such directories from the file system. Use some configuration management tool to perform this action on all nodes at once in distributed setups.
Moreover, if such irremovable directories exist, you can't delete a bucket.

Obviously this is not a serious solution, because it is a bug that should be fixed in the initial root.
I think we shouldn't touch files at filesystem level to fix a problem.

[tomsozolins]

Hello! I can confirm that this problem exists for a long time. We've been experiencing it with our 12 nodes cluster in RELEASE.2020-05-01T22-19-14Z. The only solution for now is to delete such directories from the file system. Use some configuration management tool to perform this action on all nodes at once in distributed setups.
Moreover, if such irremovable directories exist, you can't delete a bucket.

Obviously this is not a serious solution, because it is a bug that should be fixed in the initial root.
I think we shouldn't touch files at filesystem level to fix a problem.

Today i found out that this bug is happening only on buckets with versioning enabled. If i do versioning suspend on bucket, the problem still persists and directories are not deleted. On buckets with no versioning, directories are deleted correctly.

[harshavardhana]

Today i found out that this bug is happening only on buckets with versioning enabled. If i do versioning suspend on bucket, the problem still persists and directories are not deleted. On buckets with no versioning, directories are deleted correctly.

This is because of not a bug it is simply the fact that Deletes on versioning enabled buckets creates delete markers i.e the object is actually not deleted. Deleting from browser doesn't delete the objects on versioning enabled buckets instead it creates a delete marker so the object is not visible but its prefixes are i.e top level directories. This is behaving in accordance with AWS S3 behavior.

if you wish to delete them mc rm --versions --force --recursive . Even when versioning is suspended according to AWS S3 the versions cannot be deleted you have to still create a delete marker with null version ID to protect the previous versions. If you are not interested in versioning never enable it.

For ILM to delete the delete markers as well as older versions you should enable ILM to delete the delete markers as well.

Hello! I can confirm that this problem exists for a long time. We've been experiencing it with our 12 nodes cluster in RELEASE.2020-05-01T22-19-14Z. The only solution for now is to delete such directories from the file system. Use some configuration management tool to perform this action on all nodes at once in distributed setups.

This is a separate problem @annkam not related to what @tomsozolins is talking about.