Support for "directory" objects #10499

krishnasrinivas · 2020-09-16T20:03:21Z

Description

This is a requirement from a potential customer:

Without the fix:

krishna@escape:~$ mc mb min/test/
Bucket created successfully `min/test/`.
krishna@escape:~$ mc mb min/test/prefix/
Bucket created successfully `min/test/prefix/`.
krishna@escape:~$ mc cp /etc/passwd min/test/prefix/
/etc/passwd:                          2.42 KiB / 2.42 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 50.28 KiB/s 0skrishna@escape:~$ mc rm min/test/prefix/passwd
Removing `min/test/prefix/passwd`.
krishna@escape:~$ mc ls min/test/
krishna@escape:~$

i.e "prefix/" is gone.

With fix, "prefix/" is not deleted:

krishna@escape:~$ mc mb min/test/
Bucket created successfully `min/test/`.
krishna@escape:~$ mc mb min/test/prefix/
Bucket created successfully `min/test/prefix/`.
krishna@escape:~$ mc cp /etc/passwd min/test/prefix/
/etc/passwd:                          2.42 KiB / 2.42 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 43.41 KiB/s 0skrishna@escape:~$ mc rm min/test/prefix/passwd
Removing `min/test/prefix/passwd`.
krishna@escape:~$ mc ls min/test/
[2020-09-16 12:59:44 PDT]      0B prefix/
krishna@escape:~$ mc ls -r min/test/
[2020-09-16 12:59:30 PDT]      0B prefix/
krishna@escape:~$

Request to reviewers: Please see if this causes any side effects to healing of the "prefix/' object.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation needed
Unit tests needed

cmd/globals.go

cmd/tree-walk.go

harshavardhana · 2020-09-17T01:11:04Z

cmd/tree-walk.go

@@ -57,6 +57,9 @@ func filterMatchingPrefix(entries []string, prefixEntry string) []string {
 // isLeaf should be done in listDir()
 func delayIsLeafCheck(entries []string) bool {
 	for i, entry := range entries {
+		if HasSuffix(entry, globalDirSuffixWithSlash) {


Add a comment on why its skipped?

Please add comments in this portion of the code @krishnasrinivas

harshavardhana · 2020-09-17T01:11:30Z

cmd/tree-walk.go

@@ -96,7 +99,20 @@ func filterListEntries(bucket, prefixDir string, entries []string, prefixEntry s
 	entries = filterMatchingPrefix(entries, prefixEntry)

 	// Listing needs to be sorted.
-	sort.Strings(entries)
+	sort.Slice(entries, func(i, j int) bool {
+		if !HasSuffix(entries[i], globalDirSuffixWithSlash) && !HasSuffix(entries[j], globalDirSuffixWithSlash) {


Add a comment on how this handles the order for __MINIO__DIR_

cmd/tree-walk.go

harshavardhana · 2020-09-17T01:26:53Z

Found that on versioning enabled buckets we end up creating versions for empty directories we need to fix that @krishnasrinivas

~ mc rm -r --force myminio/testbucket1/
mc: <ERROR> Failed to remove `myminio/testbucket1/`. Object does not exist

~ mc ls myminio/testbucket1/ --versions
[2020-09-16 18:26:33 PDT]      0B prefix/
[2020-09-16 18:26:33 PDT]      0B prefix/
[2020-09-16 18:26:33 PDT]      0B prefix__MINIO_DIR__/
[2020-09-16 18:26:33 PDT]      0B prefix__MINIO_DIR__/

~ mc ls myminio/testbucket1/

harshavardhana · 2020-09-17T01:27:20Z

~ ls /tmp/fs-new{1,2,3,4}/testbucket1/prefix__MINIO_DIR__/
[2020-09-16 18:20:56 PDT]  4.0KiB b2b4d7f9-83cc-4873-a567-7b8ba9f67197/
[2020-09-16 18:23:39 PDT]    356B xl.meta
[2020-09-16 18:20:56 PDT]  4.0KiB b2b4d7f9-83cc-4873-a567-7b8ba9f67197/
[2020-09-16 18:23:39 PDT]    356B xl.meta
[2020-09-16 18:20:56 PDT]  4.0KiB b2b4d7f9-83cc-4873-a567-7b8ba9f67197/
[2020-09-16 18:23:39 PDT]    356B xl.meta
[2020-09-16 18:20:56 PDT]  4.0KiB b2b4d7f9-83cc-4873-a567-7b8ba9f67197/
[2020-09-16 18:23:39 PDT]    356B xl.meta

harshavardhana · 2020-09-17T01:27:53Z

I am not sure if this was intentional @krishnasrinivas

harshavardhana

...edited...

cmd/tree-walk.go

cmd/server_test.go

harshavardhana · 2020-09-19T03:45:33Z

cmd/tree-walk.go

@@ -57,6 +57,9 @@ func filterMatchingPrefix(entries []string, prefixEntry string) []string {
 // isLeaf should be done in listDir()
 func delayIsLeafCheck(entries []string) bool {
 	for i, entry := range entries {
+		if HasSuffix(entry, globalDirSuffixWithSlash) {


Please add comments in this portion of the code @krishnasrinivas

cmd/tree-walk.go

harshavardhana · 2020-09-19T04:08:11Z

@krishnasrinivas the tests don't pass yet can you fix them?

Also, I did my Hadoop tests the migration jobs seem to run fine from hadoop to MinIO without much trouble.

minio-trusted · 2020-09-19T09:09:36Z

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-gateway-azure.sh	❌ more...

10499-6c421c2/mint-gateway-azure.sh.log:

Running with
SERVER_ENDPOINT:      minio-c2.minio.io:31429
ACCESS_KEY:           minioazure
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 4167803e4cd8:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 8 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 1 minutes and 15 seconds
(4/15) Running aws-sdk-ruby tests ... done in 19 seconds
(5/15) Running awscli tests ... done in 2 minutes and 49 seconds
(6/15) Running healthcheck tests ... done in 0 seconds
(7/15) Running mc tests ... done in 3 minutes and 53 seconds
(8/15) Running minio-dotnet tests ... done in 1 minutes and 44 seconds
(9/15) Running minio-go tests ... FAILED in 8 minutes and 7 seconds
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x83fd53]

goroutine 1 [running]:
main.testGetObjectModified()
	/mint/run/core/minio-go/main.go:5688 +0xc63
main.main()
	/mint/run/core/minio-go/main.go:11269 +0x1e0
(9/15) Running minio-java tests ... done in 10 minutes and 49 seconds
(10/15) Running minio-js tests ... FAILED in 45 seconds
{
  "name": "minio-js",
  "function": "\"after all\" hook in \"functional tests\"",
  "duration": 79,
  "status": "FAIL",
  "error": "S3Error: The bucket you tried to delete is not empty at Object.parseError (node_modules/minio/dist/main/xml-parsers.js:86:11) at /mint/run/core/minio-js/node_modules/minio/dist/main/transformers.js:156:22 at DestroyableTransform._flush (node_modules/minio/dist/main/transformers.js:80:10) at DestroyableTransform.prefinish (node_modules/readable-stream/lib/_stream_transform.js:129:10) at prefinish (node_modules/readable-stream/lib/_stream_writable.js:611:14) at finishMaybe (node_modules/readable-stream/lib/_stream_writable.js:620:5) at endWritable (node_modules/readable-stream/lib/_stream_writable.js:643:3) at DestroyableTransform.Writable.end (node_modules/readable-stream/lib/_stream_writable.js:571:22) at IncomingMessage.onend (_stream_readable.js:682:10) at endReadableNT (_stream_readable.js:1252:12) at processTicksAndRejections (internal/process/task_queues.js:80:21)"
}
(10/15) Running minio-py tests ... done in 18 minutes and 15 seconds
(11/15) Running s3cmd tests ... done in 2 minutes and 12 seconds
(12/15) Running s3select tests ... done in 41 seconds
(13/15) Running security tests ... done in 0 seconds

Executed 13 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally
Error: No such image: minio/minio:10499-6c421c2

harshavardhana requested changes Sep 16, 2020

View reviewed changes

krishnasrinivas force-pushed the support-prefix-dir branch from 5405304 to 30a6388 Compare September 16, 2020 21:17

harshavardhana requested changes Sep 17, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

harshavardhana requested changes Sep 17, 2020

View reviewed changes

cmd/tree-walk.go Outdated Show resolved Hide resolved

krishnasrinivas force-pushed the support-prefix-dir branch 2 times, most recently from 152e5cb to 9778539 Compare September 18, 2020 19:06

harshavardhana requested changes Sep 18, 2020

View reviewed changes

cmd/server_test.go Outdated Show resolved Hide resolved

harshavardhana requested changes Sep 19, 2020

View reviewed changes

cmd/server_test.go Outdated Show resolved Hide resolved

cmd/server_test.go Outdated Show resolved Hide resolved

harshavardhana requested changes Sep 19, 2020

View reviewed changes

krishnasrinivas added 2 commits September 19, 2020 00:41

Support for "directory" objects

7687195

Review comments

6c421c2

krishnasrinivas force-pushed the support-prefix-dir branch from 3be248e to 6c421c2 Compare September 19, 2020 08:08

harshavardhana approved these changes Sep 19, 2020

View reviewed changes

harshavardhana merged commit 230fc0d into minio:master Sep 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for "directory" objects #10499

Support for "directory" objects #10499

krishnasrinivas commented Sep 16, 2020

harshavardhana Sep 17, 2020

harshavardhana Sep 19, 2020

harshavardhana Sep 17, 2020

harshavardhana commented Sep 17, 2020 •

edited

harshavardhana commented Sep 17, 2020

harshavardhana commented Sep 17, 2020

harshavardhana left a comment •

edited

This comment has been minimized.

This comment has been minimized.

harshavardhana Sep 19, 2020

harshavardhana commented Sep 19, 2020

minio-trusted commented Sep 19, 2020

Support for "directory" objects #10499

Support for "directory" objects #10499

Conversation

krishnasrinivas commented Sep 16, 2020

Description

Types of changes

Checklist:

harshavardhana Sep 17, 2020

Choose a reason for hiding this comment

harshavardhana Sep 19, 2020

Choose a reason for hiding this comment

harshavardhana Sep 17, 2020

Choose a reason for hiding this comment

harshavardhana commented Sep 17, 2020 • edited

harshavardhana commented Sep 17, 2020

harshavardhana commented Sep 17, 2020

harshavardhana left a comment • edited

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

harshavardhana Sep 19, 2020

Choose a reason for hiding this comment

harshavardhana commented Sep 19, 2020

minio-trusted commented Sep 19, 2020

Mint Automation

10499-6c421c2/mint-gateway-azure.sh.log:

harshavardhana commented Sep 17, 2020 •

edited

harshavardhana left a comment •

edited