feat: Implement listing version 3.0 #12605

klauspost · 2021-06-30T14:34:55Z

Description

design document

Motivation and Context

See design doc above

How to test this PR?

Start server, perform listing operations.

minio/mint#327 required for mint.

Types of changes

New feature (non-breaking change which adds functionality)
Optimization (provides speedup with no functional changes)

Checklist:

Unit tests added/updated

# Conflicts: # cmd/metacache-set.go

Fixes brought forward from minio#12605 Fixes resolution when an object is in prefix of another and one zone returns the directory and another the object. Fixes resolution on single entries that arrive first, so resolution doesn't depend on order.

harshavardhana · 2021-07-05T00:31:34Z

I ended up squashing all the commits and pushing to this branch @klauspost - since this is a simple enough change and also fixes the local erasure-coded setup behavior.

harshavardhana

This works great in initial tests, ready to take it in.

harshavardhana · 2021-07-05T06:44:45Z

Although this PR is good there are perhaps a lot more tuning that we need to do, there are subtle bugs like this

warp list --warp-client 192.168.103.{11...16} --host 192.168.103.{5...10}:9000 --access-key minio --secret-key minio123 --storage-class REDUCED_REDUNDANCY --noprefix

Throughput, split into 103 x 1s:
 * Fastest: 607.7KiB/s, 622.38 obj/s (1s, starting 00:30:11 MDT)
 * 50% Median: 570.2KiB/s, 584.04 obj/s (1s, starting 00:31:08 MDT)
 * Slowest: 420.8KiB/s, 430.95 obj/s (1s, starting 00:31:53 MDT)

----------------------------------------
Operation: LIST (409). Concurrency: 0.
Errors: 409
First Errors:
 * http://192.168.103.6:9000, 2021-07-05 00:34:08 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.8:9000, 2021-07-05 00:34:11 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:33:40 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.5:9000, 2021-07-05 00:33:34 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.7:9000, 2021-07-05 00:33:34 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.9:9000, 2021-07-05 00:33:53 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.6:9000, 2021-07-05 00:35:19 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.8:9000, 2021-07-05 00:34:16 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.5:9000, 2021-07-05 00:33:55 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:33:38 -0600 MDT: Unexpected object count, want 10000, got 60000

Skipping LIST too few samples. Longer benchmark run required for reliable results.

Without --noprefix things run fine of course because each prefix only has limited entries.

harshavardhana · 2021-07-05T07:05:43Z

warp list --warp-client 192.168.103.{11...16} --host 192.168.103.{5...10}:9000 --access-key minio --secret-key minio123 --storage-class REDUCED_REDUNDANCY --noprefix --concurrent 1

----------------------------------------
Operation: LIST (154). Concurrency: 0.
Errors: 154
First Errors:
 * http://192.168.103.10:9000, 2021-07-05 00:53:41 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:53:40 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:53:40 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:53:41 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:53:40 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.10:9000, 2021-07-05 00:53:40 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.5:9000, 2021-07-05 00:53:47 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.5:9000, 2021-07-05 00:53:47 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.5:9000, 2021-07-05 00:55:46 -0600 MDT: Unexpected object count, want 10000, got 60000
 * http://192.168.103.5:9000, 2021-07-05 00:55:46 -0600 MDT: Unexpected object count, want 10000, got 60000

Skipping LIST too few samples. Longer benchmark run required for reliable results.

Also reproduces this situation @klauspost

harshavardhana

👎🏽 until we investigate the warp --noprefix issue

klauspost · 2021-07-05T09:27:34Z

Sure 👌

klauspost · 2021-07-05T13:15:23Z

Interesting. Getting fine results, with set: SET WARP_HOST=127.0.0.1:{9001...9002}. Using 1 ss, 2 servers, 4 disks each or 4ss, 1 set each, 4 disks each.

Using warp list -noprefix --concurrent 1 -duration=30s. @harshavardhana What server setup are you using?

harshavardhana · 2021-07-05T14:35:52Z

Interesting. Getting fine results, with set: SET WARP_HOST=127.0.0.1:{9001...9002}. Using 1 ss, 2 servers, 4 disks each or 4ss, 1 set each, 4 disks each.

Using warp list -noprefix --concurrent 1 -duration=30s. @harshavardhana What server setup are you using?

It's a 6 node setup with 20 drives each @klauspost and 6 clients..

… listing-v3-refactor # Conflicts: # cmd/metacache-server-pool.go # cmd/metacache-set.go

klauspost · 2021-07-05T15:00:34Z

@harshavardhana Interesting. Are you cleaning between runs? Without prefixes all objects will be in the root, so without cleanup there will be too many objects.

I got a want 50000, got 50001 when it failed to clean up a single file from the last run. Testing on 6 servers in one pool, 4 disks each ATM, but I guess I can add more disks.

harshavardhana · 2021-07-05T16:52:47Z

@harshavardhana Interesting. Are you cleaning between runs? Without prefixes all objects will be in the root, so without cleanup there will be too many objects.

I got a want 50000, got 50001 when it failed to clean up a single file from the last run. Testing on 6 servers in one pool, 4 disks each ATM, but I guess I can add more disks.

Yes I just let warp clean itself, I don't specify --noclear

harshavardhana · 2021-07-05T17:19:12Z

I suspect this issue to be some kind of a timeout problem, where the listing timesout and then ends up listing all the entries without honoring the max-keys @klauspost

harshavardhana · 2021-07-05T18:20:21Z

@klauspost another thing I found is the master branch

warp list --warp-client 192.168.103.{11...16} --host 192.168.103.{5...10}:9000 --access-key minio --secret-key minio123 --storage-class REDUCED_REDUNDANCY --noprefix

Works fine with this where as this PR does not, in-fact it seems to increase the load on the system significantly higher is that expected?

harshavardhana · 2021-07-05T18:26:12Z

@klauspost another thing I found is the master branch
warp list --warp-client 192.168.103.{11...16} --host 192.168.103.{5...10}:9000 --access-key minio --secret-key minio123 --storage-class REDUCED_REDUNDANCY --noprefix
Works fine with this where as this PR does not, in-fact it seems to increase the load on the system significantly higher is that expected?

Actually this is not correct - looks like even master branch has this issue

----------------------------------------
Operation: LIST (2276). Concurrency: 0.
Errors: 2276
First Errors:
 * http://192.168.103.5:9000, 2021-07-05 12:16:36 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.6:9000, 2021-07-05 12:16:38 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.8:9000, 2021-07-05 12:16:39 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.9:9000, 2021-07-05 12:16:39 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.7:9000, 2021-07-05 12:16:38 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.10:9000, 2021-07-05 12:16:36 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.5:9000, 2021-07-05 12:16:36 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.6:9000, 2021-07-05 12:16:38 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.8:9000, 2021-07-05 12:16:40 -0600 MDT: Unexpected object count, want 10000, got 59999
 * http://192.168.103.9:9000, 2021-07-05 12:16:38 -0600 MDT: Unexpected object count, want 10000, got 59999

Skipping LIST too few samples. Longer benchmark run required for reliable results.

klauspost · 2021-07-05T18:32:41Z

@harshavardhana I can only repro if it is not cleaned up. I did find a bug where disks would be duplicated. Just sent a fix for that. That is this PR only, though.

I added a check for results in warp:

				// Wait for errCh to close.
				last := ""
				for oi := range listCh {
					if oi.Err != nil {
						d.Error(oi.Err)
						op.Err = oi.Err.Error()
					}
					if last >= oi.Key {
						console.Errorln("duplicate entry", oi.Key, "last was", last)
					}
					if true {
						found := false
						for _, o := range d.objects {
							for _, obj := range o {
								if obj.Name == oi.Key {
									found = true
									break
								}
							}
						}
						if !found {
							console.Errorln("Unknown key:", oi.Key)
						}
					}
					last = oi.Key

This should detect unknown or duplicated entries returned. Insert after // Wait for errCh to close.

Oh you are running multiple clients. Probably a bug with that.

klauspost · 2021-07-05T18:38:10Z

@harshavardhana Yes, multiple clients will do that, since objects are dumped in the same folder, they cannot separate them when listing.

cmd/metacache-server-pool.go

minio-trusted · 2021-07-05T22:08:32Z

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-compress-encrypt-dist-erasure.sh	✔️
Deleting image on docker hub
Deleting image locally

harshavardhana · 2021-07-05T22:34:24Z

The PR looks good and also fixes actually some existing problems in the server implementation. Taking this in to further tweak the master branch.

klauspost added 10 commits June 16, 2021 09:42

Listing v3

503b852

Merge branch 'master' into listing-v3-refactor

6da9998

Merge branch 'master' into listing-v3-refactor

e3b6805

Simplify

cfe7f26

Add separate saver...

4357339

First test pass.

93e535d

Remove transient code.

15ada4e

Tweak settings.

6225012

Merge branch 'master' into listing-v3-refactor

23b8f6a

Clean up more unused code.

f9444c0

klauspost force-pushed the listing-v3-refactor branch from 4a2b0b2 to f9444c0 Compare June 30, 2021 15:37

Merge branch 'master' into listing-v3-refactor

5c57098

# Conflicts: # cmd/metacache-set.go

harshavardhana mentioned this pull request Jul 1, 2021

volume not found error while calling ListObjectsV2 for a bucket with too many files #12587

Closed

klauspost added 2 commits July 1, 2021 16:29

Fix various bugs, including existing resolution.

94787cf

Fix object/dir conflicts.

66d2aaf

klauspost force-pushed the listing-v3-refactor branch from 656e253 to 66d2aaf Compare July 1, 2021 16:28

klauspost added 7 commits July 2, 2021 10:34

Fix forwarding. Don't leak results.

f160e05

Remove outdated test.

ba7affb

Add cancellation. Remove unused stuff.

91d8750

Merge branch 'master' into listing-v3-refactor

194dbf7

Fix two scanner races

eded5a2

Close on errror

2501334

Delete deprecated key

6c423f3

klauspost marked this pull request as ready for review July 2, 2021 11:18

klauspost requested review from harshavardhana and vadmeste July 2, 2021 11:18

klauspost mentioned this pull request Jul 2, 2021

Fix multiway list entry resolution #12617

Merged

1 task

klauspost added 2 commits July 2, 2021 14:51

Two way resolve

cbe8d5e

Return a if b fails.

70f7f45

harshavardhana force-pushed the listing-v3-refactor branch from 0c30700 to 19f893e Compare July 5, 2021 00:12

harshavardhana changed the title ~~Listing v3 refactor~~ feat: Implement listing version 3.0 Jul 5, 2021

harshavardhana self-requested a review July 5, 2021 06:21

harshavardhana approved these changes Jul 5, 2021

View reviewed changes

harshavardhana requested changes Jul 5, 2021

View reviewed changes

Cancel all upstream listing when mergechannels returns.

2744ef2

Merge branch 'listing-v3-refactor' of github.com:klauspost/minio into…

5611e04

… listing-v3-refactor # Conflicts: # cmd/metacache-server-pool.go # cmd/metacache-set.go

Add some sanity checks.

896e134

Fix shuffle causing duplicated disks.

8372474

harshavardhana requested changes Jul 5, 2021

View reviewed changes

cmd/metacache-server-pool.go Outdated Show resolved Hide resolved

work in local erasure coded mode

e0314b8

harshavardhana approved these changes Jul 5, 2021

View reviewed changes

harshavardhana merged commit 05aebc5 into minio:master Jul 5, 2021

harshavardhana mentioned this pull request Jul 6, 2021

Improve memory usage on highly concurrent listing #12548

Closed

klauspost deleted the listing-v3-refactor branch July 6, 2021 08:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Implement listing version 3.0 #12605

feat: Implement listing version 3.0 #12605

klauspost commented Jun 30, 2021 •

edited

Loading

harshavardhana commented Jul 5, 2021

harshavardhana left a comment

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana left a comment

klauspost commented Jul 5, 2021

klauspost commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

klauspost commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

klauspost commented Jul 5, 2021 •

edited

Loading

klauspost commented Jul 5, 2021

minio-trusted commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

feat: Implement listing version 3.0 #12605

feat: Implement listing version 3.0 #12605

Conversation

klauspost commented Jun 30, 2021 • edited Loading

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

harshavardhana commented Jul 5, 2021

harshavardhana left a comment

Choose a reason for hiding this comment

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana left a comment

Choose a reason for hiding this comment

klauspost commented Jul 5, 2021

klauspost commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

klauspost commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

harshavardhana commented Jul 5, 2021

klauspost commented Jul 5, 2021 • edited Loading

klauspost commented Jul 5, 2021

minio-trusted commented Jul 5, 2021

Mint Automation

harshavardhana commented Jul 5, 2021

klauspost commented Jun 30, 2021 •

edited

Loading

klauspost commented Jul 5, 2021 •

edited

Loading