Missing Object Parts After Node Maintenance - Auto-Healing Not Working?

[WMP]

Description:

Hi MinIO Team,

I have a MinIO cluster running with the following setup:

minio server --address :9091 --console-address :9092 --certs-dir /etc/minio/ssl \
http://node{01...03}:9091/mnt/port1i/box1/bay{1...4}/minio \
http://node{01...03}:9091/mnt/port2i/box2/bay{1...4}/minio \
http://node{01...03}:9091/mnt/port3i/box3/bay{1...4}/minio

Additionally, I have Keepalived and Sidekick running.

Issue:

I planned maintenance for node node03. Before shutting it down, I stop Keepalived, Sidekick, and MinIO. The node was offline for about an hour. During this time, new objects were uploaded to the cluster. After bringing the node back up, I expected MinIO to automatically reconstruct the missing parts of these objects on node03.

However, checking the data structure, I noticed:

• The uploaded objects exist on node01 and node02
• No parts of these objects were placed on node03
• Running the same du command after one hour still shows that the objects are not reconstructed on node03

Commands used to check the object parts:

On node01 and node02, the objects exist:

root@node01:/mnt# du -s port*i/box*/bay*/minio//backup-2025-03/documents/report-12345.SENT/*
176     port1i/box1/bay1/minio/backup-2025-03/documents/report-12345.SENT/part-01
4       port1i/box1/bay1/minio/backup-2025-03/documents/report-12345.SENT/xl.meta
176     port1i/box1/bay2/minio/backup-2025-03/documents/report-12345.SENT/part-02
4       port1i/box1/bay2/minio/backup-2025-03/documents/report-12345.SENT/xl.meta
...

root@node02:/mnt# du -s port*i/box*/bay*/minio//backup-2025-03/documents/report-12345.SENT/*
176     port1i/box1/bay1/minio/backup-2025-03/documents/report-12345.SENT/part-01
4       port1i/box1/bay1/minio/backup-2025-03/documents/report-12345.SENT/xl.meta
176     port1i/box1/bay2/minio/backup-2025-03/documents/report-12345.SENT/part-02
4       port1i/box1/bay2/minio/backup-2025-03/documents/report-12345.SENT/xl.meta
...

On node03, no parts exist:

root@node03:/mnt# du -s port*i/box*/bay*/minio//backup-2025-03/documents/report-12345.SENT/*
du: cannot access 'port*i/box*/bay*/minio/backup-2025-03/documents/report-12345.SENT': No such file or directory

MinIO Status Check:

Despite this, mc admin info reports that all nodes and drives are healthy, and no issues are detected:

$ mc admin info minio-cluster/
●  node01:9091
   Uptime: 1 month
   Version: 2024-12-18T13:15:44Z
   Network: 3/3 OK
   Drives: 12/12 OK
   Pool: 1, 2, 3

●  node02:9091
   Uptime: 1 month
   Version: 2024-12-18T13:15:44Z
   Network: 3/3 OK
   Drives: 12/12 OK
   Pool: 1, 2, 3

●  node03:9091
   Uptime: 1 hour
   Version: 2024-12-18T13:15:44Z
   Network: 3/3 OK
   Drives: 12/12 OK
   Pool: 1, 2, 3

221 TiB Used, 66 Buckets, 290,823,474 Objects
36 drives online, 0 drives offline, EC:4

I also checked the storage class settings:

$ mc admin config get minio-cluster storage_class
storage_class standard=EC:4 rrs=EC:4 optimize=availability

Concerns:

1. What settings should I change to make MinIO automatically reconstruct missing objects after a node comes back online?
2. Why does mc admin info show that everything is fine when some objects are actually missing parts?
3. How can I check via MinIO API if an object has all its parts stored across the expected disks?
4. How can I trigger MinIO to rebuild the missing parts for objects that were uploaded while a node was offline?

Thanks for your help!

[harshavardhana]

It won't just start healing after the node is backup, it will be scheduled via scanner.

If you want it faster run mc admin heal -r