-
Notifications
You must be signed in to change notification settings - Fork 0
decommission
Cesar Celis Hernandez edited this page Apr 11, 2024
·
38 revisions
A manual way to test decommission when fails are observed in our CI/CD Pipeline
- Create cluster
createcluster
- install operator
installoperator
- install tenant lite
installtenant
- increase from 1 pool to 2 pools
![Screenshot 2024-01-29 at 3 37 06 PM](https://private-user-images.githubusercontent.com/6667358/300597016-db63baee-0389-4ef3-b552-9ca00fd08f6e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxMTE1MjEsIm5iZiI6MTcyMjExMTIyMSwicGF0aCI6Ii82NjY3MzU4LzMwMDU5NzAxNi1kYjYzYmFlZS0wMzg5LTRlZjMtYjU1Mi05Y2EwMGZkMDhmNmUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MjdUMjAxMzQxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NWExZTBiYjIwNGRkODExOTY1MDY5MGZjZTNhY2JiN2VlNGNkYjNkYWFlZDVlNzYyYzE3ZWE3OGY3MjMzZWI0MyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.OB8c6cqgO55a-VXl0tkfHLJCpv8ns7OaYtS38uVGFAs)
- Wait couple of minutes until is ready:
![Screenshot 2024-01-29 at 3 44 25 PM](https://private-user-images.githubusercontent.com/6667358/300598791-5b574c9b-f177-4f07-82bd-0c4369772dad.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxMTE1MjEsIm5iZiI6MTcyMjExMTIyMSwicGF0aCI6Ii82NjY3MzU4LzMwMDU5ODc5MS01YjU3NGM5Yi1mMTc3LTRmMDctODJiZC0wYzQzNjk3NzJkYWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDcyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MjdUMjAxMzQxWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTE2Y2VhYzczZDUxNDY1M2ViMzYzOTljODI0OWFmZTU4N2ZlMmU3ZWFkZDkxNmE3NGUyMjU3OGFjMmZkMDdjZiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.q0SLIiT7CkXFXGL6abA-3ROFW9azLEkTrgsa87nIBpE)
- Put some data
mc alias set myminio https://127.0.0.1:30082 minio minio123 --insecure
mc mb myminio/celis --insecure
echo "a" > a.txt
mc cp a.txt myminio/celis --insecure
mc ls myminio/celis --insecure
- Get status:
mc admin decommission status myminio --insecure
Expected:
$ mc admin decommission status myminio --insecure
┌─────┬───────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┬────────┐
│ ID │ Pools │ Raw Drives Usage │ Status │
│ 1st │ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Active │
│ 2nd │ https://myminio-pool-1-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Active │
└─────┴───────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────┘
- Start Decommission:
mc admin decommission start myminio/ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} --insecure
Expected:
$ mc admin decommission start myminio/ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} --insecure
Decommission started successfully for `https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1}`.
- Monitor until complete:
$ mc admin decommission status myminio/ --insecure
┌─────┬───────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┬──────────┐
│ ID │ Pools │ Raw Drives Usage │ Status │
│ 1st │ https://myminio-pool-0-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Complete │
│ 2nd │ https://myminio-pool-1-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 16.2% (total: 332 GiB) │ Active │
└─────┴───────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴──────────┘
- Remove decommissioned pool from tenant spec
k edit tenant myminio -n tenant-lite
![Screenshot 2023-02-03 at 5 51 15 PM](https://user-images.githubusercontent.com/6667358/216725889-d05d2acb-96a5-4cf3-928a-2c32f14183fd.png)
![Screenshot 2023-02-03 at 5 51 28 PM](https://user-images.githubusercontent.com/6667358/216725908-0a338c6b-afeb-40d3-b24c-6c769f09b32c.png)
Cesars-MacBook-Pro:~ cniackz$ k edit tenant storage-lite -n tenant-lite
tenant.minio.min.io/storage-lite edited
![Screenshot 2023-02-03 at 5 51 46 PM](https://user-images.githubusercontent.com/6667358/216725942-45680b53-2ca7-4b63-87e0-392737ceb97f.png)
- Operator log:
I0130 15:18:40.193678 1 event.go:298] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"tenant-lite", Name:"myminio", UID:"58a44125-aff4-4b93-8b7f-95367bc4fceb", APIVersion:"minio.min.io/v2", ResourceVersion:"2509", FieldPath:""}): type: 'Normal' reason: 'PoolRemoved' Tenant pool myminio-pool-0 removed
- Status:
$ mc admin decommission status myminio --insecure
┌─────┬───────────────────────────────────────────────────────────────────────────────────────┬────────────────────────┬────────┐
│ ID │ Pools │ Raw Drives Usage │ Status │
│ 2nd │ https://myminio-pool-1-{0...3}.myminio-hl.tenant-lite.svc.cluster.local/export{0...1} │ 14.9% (total: 332 GiB) │ Active │
└─────┴───────────────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────┘
- Pods from pool 0 are gone, pods from pool 1 are present:
$ k get pods -n tenant-lite
NAME READY STATUS RESTARTS AGE
myminio-pool-1-0 2/2 Running 0 9m40s
myminio-pool-1-1 2/2 Running 0 9m43s
myminio-pool-1-2 2/2 Running 0 9m46s
myminio-pool-1-3 2/2 Running 0 10m
- Check data is still present after decommission:
$ mc ls myminio/celis --insecure
[2024-04-11 16:23:22 EDT] 2B STANDARD a.txt
- Decommission worked in 4.5.8
- Decommission worked in 5.0.11