Add support for multi destination replication

[poornas]

Description

Motivation and Context

AWS supports replicating to multiple destination buckets - this PR adds equivalent functionality for MinIO
Needs companion mc PR minio/mc#3766

How to test this PR?

With the mc PR referenced above, and the replication docs.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Documentation updated
• Unit tests added/updated

[klauspost]

@poornas Is there a madmin-go PR as well, or is it already merged?

[poornas]

@poornas Is there a madmin-go PR as well, or is it already merged?

minio/madmin-go@e6d765f - PR was merged but I can't remove the local dependency reference because the minio speedtest autotune PR needs to be merged first - madmin.Speedtestresult structure has changed and those need to be applied before this PR will compile. madmin v1.1.1 with the commit above is needed for this PR to compile for now.

[klauspost]

Some minor stuff/questions.

[klauspost]

Let's make this build. See if you can push for the upstream changes you need.

[vadmeste]

@poornas I tested today and I found two issues:

• Modtime of delete marker in the target site is not correct
• When uploading to three sites at the same time, the three sites won't sync and there is these logs:

vadmeste@fastdebian:~/work/go/gospace/code/replication$                                            
API: SYSTEM()
Time: 20:47:08 CET 09/09/2021
DeploymentID: 785dcbbf-5ef6-40a2-99b6-01c7c71a86e1
Error: Unable to replicate for object testbucket/dir/file(1df00c0d-af44-44bb-9d2e-25d9ce89caca): Put "http://localhost:9002/testbucket/dir/file?versionId=1df00c0d-af44-44bb-9d2e-25d9ce89caca": context canceled (*errors.errorString)                                                                                                                                                                       
       2: /home/vadmeste/work/go/gospace/code/minio/cmd/bucket-replication.go:1094:cmd.replicateObjectToTarget()     
       1: /home/vadmeste/work/go/gospace/code/minio/cmd/bucket-replication.go:859:cmd.replicateObject.func2()                                                                                          
                                                 
API: SYSTEM()                
Time: 20:47:08 CET 09/09/2021                                                                      
DeploymentID: 785dcbbf-5ef6-40a2-99b6-01c7c71a86e1                                         
Error: failed to get lock for object: dir/file bucket:testbucket arn: (*errors.errorString)                                                                                                            
       2: /home/vadmeste/work/go/gospace/code/minio/cmd/bucket-replication.go:835:cmd.replicateObject()              
       1: /home/vadmeste/work/go/gospace/code/minio/cmd/bucket-replication.go:1281:cmd.(*ReplicationPool).AddWorker()                                                                                  
               

[poornas]

@poornas I tested today and I found two issues:

• Modtime of delete marker in the target site is not correct
• When uploading to three sites at the same time, the three sites won't sync and there is these logs:

I haven't been able to repro this yet - will ping you on the test scenario

[vadmeste]

@poornas to reproduce modtime issue, you need to take a look directly at xl.meta using xl-meta tool. You don't see it because the modtime of the delete marker in the source & target site are very close, at the same second most of the time.

Also I checked, it works fine with master

[poornas]

@poornas to reproduce modtime issue, you need to take a look directly at xl.meta using xl-meta tool. You don't see it because the modtime of the delete marker in the source & target site are very close, at the same second most of the time.

Also I checked, it works fine with master

fixed this

[harshavardhana]

Can you resolve the conflicts on the PR ? @poornas

[harshavardhana]

@poornas

cmd/data-usage_test.go:589:5: unknown field `ReplicaSize` in struct literal (typecheck)
				ReplicaSize:    4,

[minio-trusted]

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-compress-encrypt-dist-erasure.sh	✔️
Deleting image on docker hub
Deleting image locally