Option to enable disable ILM expiry for site replication #4698

shtripat · 2023-09-28T10:07:11Z

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the [Apache 2 license] (https://www.apache.org/licenses/LICENSE-2.0).
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Added options replicate-ilm-expiry and disable-ilm-expiry-replication for mc admin replicate add and mc admin replicate update commands respectively.

Motivation and Context

How to test this PR?

Follow below detailed steps for verifying the different scenarios

Step-1: Create three MinIO sites sat m1, m2 and m3

Step-2: Setup site replication between the first two sites now
mc admin replicate add m1 m2 --replicate-ilm-expiry

Step-3: Create a bucket for first site (test-bucket)
mc mb m1/test-bucket

Step-4: Setup warm tier to m3 for site m1

$ ./mc mb m3/test-bucket
$ ./mc ilm tier add minio m1 WARM-TIER --endpoint http://localhost:9020 --access-key minioadmin --secret-key minioadmin --bucket test-bucket

Step-5: Check details of the replication

$ mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true     

$ mc admin replicate info m2 
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true

Step-6: Add ILM rules to the first site's bucket test-bucket

$ ./mc ilm add m1/test-bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "shu" --tags "tag1=val1&tag2=val2&tag3=val3"

Step-7: Check ILM rules list for sites

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Step-8: Check replication status details as below

$ ./mc admin replicate status m1
Bucket replication status:
●  1/1 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present

ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

Object replication status:
Replication status since 7 minutes 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

$ ./mc admin replicate status m2
Bucket replication status:
●  1/1 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present

ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

Object replication status:
Replication status since 8 minutes 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

$ ./mc admin replicate status m1 --ilm-expiry-rules
ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

$ ./mc admin replicate status m1 --ilm-expiry-rule cksbdd5b4dtt2fpuur80 
●  ILM Expiry Rule replication summary for: cksbdd5b4dtt2fpuur80

ILMExpiryRule   | M1              | M2             
ILM Expiry Rule | ✔               | ✔

Step-9: Check the replication of removal of expiry rules across sites

$ ./mc ilm rule remove --id cl1l3n5b4dti3t4apagg m1/test-bucket 
Rule ID `cl1l3n5b4dti3t4apagg` from target m1/test-bucket removed.

$./mc ilm rule list m1/test-bucket
mc: <ERROR> Unable to ls lifecycle configuration: lifecycle configuration not set.

$./mc ilm rule list m2/test-bucket 
mc: <ERROR> Unable to get lifecycle. The lifecycle configuration does not exist.

The replication of removal of expiry rule would take few seconds to reflect in site2 as it happens as part of a sync process
Repeat this process keeping second site up and down both. The replication of removed expiry rules should happen as part of sync when second site comes up.

Step-10: Check removal of all ILM rules from first site and all expiry rules should get replicated (removed) from second site as well

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule remove --all --force m1/test-bucket 
Rules for `m1/test-bucket` removed.

$ ./mc ilm rule list m1/test-bucket 
mc: <ERROR> Unable to ls lifecycle configuration: lifecycle configuration not set.

$ ./mc ilm rule list m2/test-bucket 
mc: <ERROR> Unable to get lifecycle. The lifecycle configuration does not exist.

again this replication of expiry rules removal would take few secs for site2 (as runs as part of sync)

Step-11: Test editing of expiry rules details and their replication

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘


$ ./mc ilm edit --id "cl69m4lb4dtn1aio3rp0" --expire-days "10" m1/test-bucket
Lifecycle configuration rule with ID `cl69m4lb4dtn1aio3rp0` modified  to m1/test-bucket.

$ ./mc ilm rule list m1/test-bucket
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │             10 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │             10 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Step-12: Check disabling and re-enabling of replication of ilm expiry across site replicated sites

$ ./mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true         

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true                


$ ./mc admin replicate update m1 --disable-ilm-expiry-replication
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: false

$ ./mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | false               
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | false                

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | false               
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | false                

$ ./mc admin replicate update m1 --enable-ilm-expiry-replication 
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: true

$ ./mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true                

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true

Step-13: Verify healing of enabled/disabled ILM expiry to sites which were offline while changes done
Bring down one of the remote sites and update the ILM expiry flag for the up site

$ ./mc admin replicate update m1 --disable-ilm-expiry-replication
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: false
- partially failed for few remote sites as they could be down/unreachable at the moment
$ ./mc admin replicate info m1 
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
8ec1747f-a8c7-4061-8fa1-e3bb23982690 | m1              | http://localhost:9000                          |      | N/A        | false               
a6b7e015-ff9b-49db-ae0c-958d2d68de5a | m2              | http://localhost:9010                          |      | N/A        | false                

$ ./mc admin replicate info m2
mc: <ERROR> Unable to get cluster replication information. Get "http://localhost:9010/minio/admin/v3/site-replication/info?api-version=1": dial tcp [::1]:9010: connect: connection refused.

Bring back the remote site and wait for few secs and then check the replication info for the site

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
8ec1747f-a8c7-4061-8fa1-e3bb23982690 | m1              | http://localhost:9000                          |      | N/A        | false               
a6b7e015-ff9b-49db-ae0c-958d2d68de5a | m2              | http://localhost:9010                          |      | N/A        | false

Update same rule on both site separately with different values. The values should not be healed even after few mins (as it needs ILM expiry to be enabled for all sites for heal to happen)

$ ./mc ilm edit --id "cl69m4lb4dtn1aio3rp0" --expire-days "333" m1/test-bucket 
Lifecycle configuration rule with ID `cl69m4lb4dtn1aio3rp0` modified  to m1/test-bucket.
$ ./mc ilm edit --id "cl69m4lb4dtn1aio3rp0" --expire-days "444" m2/test-bucket
Lifecycle configuration rule with ID `cl69m4lb4dtn1aio3rp0` modified  to m2/test-bucket.

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            333 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            444 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Re-enable ILM expiry for the site we disabled earlier and wait for few secs. Rules should be healed and latest one should win to update to all sites

$ ./mc admin replicate update m1 --enable-ilm-expiry-replicatione-ilm-expiry 
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: true

$ ./mc ilm rule list m1/test-bucket
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            444 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            444 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Needs corresponding changes from minio, console, minio-go and madmin-go

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Optimization (provides speedup with no functional changes)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Unit tests added/updated
Internal documentation updated
Create a documentation update request here

poornas · 2023-10-16T22:04:20Z

you should provide the option to enable ilm expiry replication later as well?

shtripat · 2023-10-17T04:53:32Z

you should provide the option to enable ilm expiry replication later as well?

Ah ok, makes sense. Will add option for the same.

shtripat · 2023-10-17T13:07:56Z

you should provide the option to enable ilm expiry replication later as well?

Ah ok, makes sense. Will add option for the same.

Done @poornas

poornas · 2023-10-19T00:30:12Z

validations are broken - on update of enable|disable of the new flag, it expects endpoint also to be passed

mc admin replicate info should display the ilm-expiry replication setting
mc admin replicate status needs to display mismatches in ilm expiry repl. as well

shtripat · 2023-10-19T04:54:52Z

validations are broken - on update of enable|disable of the new flag, it expects endpoint also to be passed

mc admin replicate info should display the ilm-expiry replication setting mc admin replicate status needs to display mismatches in ilm expiry repl. as well

I am able to enable|disable ILM expiry replication below

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --enable-ilm-expiry-replication
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: true

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --enable-ilm-expiry-replicationcate-ilm-expiry 
Some sites could not be configured for replication.
ILM expiry already set to true

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --disable-ilm-expiry-replicatione-ilm-expiry 
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: false

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --disable-ilm-expiry-replicationate-ilm-expiry 
Some sites could not be configured for replication.
ILM expiry already set to false

It does not mandate to pass --endpoint argument. Am I missing something?

Regarding mc admin replicate info to show ilm expiry settings, I will update the PR. --json output already shows this.
And mc admin replicate status to show mismatches, I will check this and update accordingly.

shtripat · 2023-10-20T07:28:29Z

Updated and now ILM expiry replication details are shown as below

$ ./mc admin replicate info m1                                                                ST 1   support-replicate-ilm-expiry 
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync
7141750e-6f48-4960-8669-200dc9619552 | m1              | http://localhost:9000                          |     
0ecce7e4-ea6a-41be-a066-b873605c68f8 | m2              | http://localhost:9010                          |     

ILM Expiry Replication: true

and added option --ilm-expiry-rules amd --ilm-expiry-rule <name> to mc admin replicate status command as below

$ ./mc admin replicate status m1                                                              ST 1   support-replicate-ilm-expiry 
Bucket replication status:
●  1/1 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present

ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

Object replication status:
Replication status since 1 minute 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

$ ./mc admin replicate status m1 --ilm-expiry-rules                                           ST 1   support-replicate-ilm-expiry 
ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

$ ./mc admin replicate status m1 --ilm-expiry-rule ckok9v5b4dtgofkbi6tg                       ST 1   support-replicate-ilm-expiry 
●  ILM Expiry Rule replication summary for: ckok9v5b4dtgofkbi6tg

ILMExpiryRule   | M1              | M2             
ILM Expiry Rule | ✔               | ✔

cmd/admin-replicate-info.go

cmd/ilm/options.go

poornas

LGTM

go.mod

Added options `replicate-ilm-expiry` and `disable-ilm-expiry-replication` for `mc admin replicate add` and `mc admin replicate update` commands respectively. Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Don't pass `--deployment-id` while using `--disable-ilm-expiry-replication` or `--enable-ilm-expiry-replication` with `mc admin replicate update` Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

It would be calculated at server side Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

shtripat force-pushed the support-replicate-ilm-expiry branch 3 times, most recently from 36d2ba1 to 0df318e Compare September 29, 2023 12:47

shtripat changed the title ~~Added flag replicate-ilm-expiry to mc admin replication add~~ Option to enable diable ILM expiry for site replication Sep 29, 2023

shtripat force-pushed the support-replicate-ilm-expiry branch 2 times, most recently from a36701f to 95b5435 Compare October 9, 2023 06:04

shtripat changed the title ~~Option to enable diable ILM expiry for site replication~~ Option to enable disable ILM expiry for site replication Oct 9, 2023

shtripat marked this pull request as ready for review October 9, 2023 11:11

shtripat requested a review from poornas October 9, 2023 11:12

shtripat force-pushed the support-replicate-ilm-expiry branch 2 times, most recently from cdeb151 to 5ee893f Compare October 12, 2023 10:05

shtripat force-pushed the support-replicate-ilm-expiry branch 2 times, most recently from 8c92d6f to be3553c Compare October 17, 2023 12:58

shtripat force-pushed the support-replicate-ilm-expiry branch from be3553c to 62aa11c Compare October 20, 2023 07:25

shtripat force-pushed the support-replicate-ilm-expiry branch from 62aa11c to 3bdc1d6 Compare October 23, 2023 11:13

shtripat mentioned this pull request Oct 23, 2023

Replicate Expiry ILM configs while site replication minio/minio#18130

Merged

8 tasks

shtripat force-pushed the support-replicate-ilm-expiry branch 7 times, most recently from 94a8e9d to bc4efea Compare October 30, 2023 11:39

shtripat force-pushed the support-replicate-ilm-expiry branch from bc4efea to 0348585 Compare November 2, 2023 10:29

poornas reviewed Nov 6, 2023

View reviewed changes

cmd/admin-replicate-info.go Show resolved Hide resolved

shtripat force-pushed the support-replicate-ilm-expiry branch from 0348585 to eafbf36 Compare November 7, 2023 12:59

shtripat requested a review from poornas November 7, 2023 13:46

poornas approved these changes Nov 7, 2023

View reviewed changes

poornas requested changes Nov 8, 2023

View reviewed changes

cmd/ilm/options.go Outdated Show resolved Hide resolved

shtripat force-pushed the support-replicate-ilm-expiry branch from eb128bf to 6c94adc Compare November 9, 2023 08:33

shtripat requested a review from poornas November 9, 2023 08:34

poornas approved these changes Nov 10, 2023

View reviewed changes

poornas requested a review from vadmeste November 21, 2023 17:37

vadmeste approved these changes Nov 22, 2023

View reviewed changes

harshavardhana requested changes Nov 22, 2023

View reviewed changes

go.mod Outdated Show resolved Hide resolved

shtripat added 12 commits November 22, 2023 09:51

Option to enable diable ILM expiry for site replication

6f017f6

Added options `replicate-ilm-expiry` and `disable-ilm-expiry-replication` for `mc admin replicate add` and `mc admin replicate update` commands respectively. Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Added option to enable back ILM expiry for replication

f89a922

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Show ILM expiry replication flag in mc admin replicate info

14c6c13

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

ILM expiry replication details for mc admin replicatte status

93bf324

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Support replication of deleted ILM expiry rules

2387387

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Updated to latest madmin-go changes from show-ilm-expiry-details

5d7d4a0

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Use updated minio-go

980df46

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Report ILM Expiry Replication flag sitewise

a90b29d

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Apply ilm expiry replication changes to all sites

d766137

Don't pass `--deployment-id` while using `--disable-ilm-expiry-replication` or `--enable-ilm-expiry-replication` with `mc admin replicate update` Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Correct suppressed header

10b2b90

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Dont send expiryRuleRemoved from client

1cb960e

It would be calculated at server side Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

Use latest madmin-go v3.0.32

f550126

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>

shtripat force-pushed the support-replicate-ilm-expiry branch from f4921bd to f550126 Compare November 22, 2023 04:25

shtripat requested a review from harshavardhana November 22, 2023 04:25

harshavardhana approved these changes Nov 22, 2023

View reviewed changes

harshavardhana merged commit 56d84e6 into minio:master Nov 22, 2023
5 checks passed

djwfyi mentioned this pull request Dec 4, 2023

[RELEASE] mc RELEASE.2023-12-02T02-03-28Z minio/docs#1082

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Option to enable disable ILM expiry for site replication #4698

Option to enable disable ILM expiry for site replication #4698

shtripat commented Sep 28, 2023 •

edited

poornas commented Oct 16, 2023

shtripat commented Oct 17, 2023

shtripat commented Oct 17, 2023

poornas commented Oct 19, 2023

shtripat commented Oct 19, 2023 •

edited

shtripat commented Oct 20, 2023

poornas left a comment

Option to enable disable ILM expiry for site replication #4698

Option to enable disable ILM expiry for site replication #4698

Conversation

shtripat commented Sep 28, 2023 • edited

Community Contribution License

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

poornas commented Oct 16, 2023

shtripat commented Oct 17, 2023

shtripat commented Oct 17, 2023

poornas commented Oct 19, 2023

shtripat commented Oct 19, 2023 • edited

shtripat commented Oct 20, 2023

poornas left a comment

Choose a reason for hiding this comment

shtripat commented Sep 28, 2023 •

edited

shtripat commented Oct 19, 2023 •

edited