Heal Operation Timeouts

[eyalrif]

Expected behavior

I have a setup of 8 servers with 96 disks (12 on each server).The object size i am using is 1MB and the amount of objects is around 5M. when running heal operation on a single object - i get a timeout

/usr/local/bin/mc admin heal --json --recursive Minio/wekabucket1/358adf34/d/d/95/20/3696 .... {"status":"success","type":"system","name":"bucket-metadata:.minio.sys/config/config.json","before":{"color":"green","offline":0,"online":8,"missing":0,"corrupted":0,"drives":[{"uuid":"","endpoint":"/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.2:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.3:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.4:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.5:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.6:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.7:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.8:9000/minio/minio_drive10","state":"ok"}]},"after":{"color":"green","offline":0,"online":8,"missing":0,"corrupted":0,"drives":[{"uuid":"","endpoint":"/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.2:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.3:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.4:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.5:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.6:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.7:9000/minio/minio_drive10","state":"ok"},{"uuid":"","endpoint":"http://172.21.8.8:9000/minio/minio_drive10","state":"ok"}]},"size":0} Heal had an error - Operation timed out: {"status":"success","type":"summary","objects_scanned":0,"objects_healed":0,"items_scanned":2,"items_healed":0,"size":0,"duration":1861}

mc ls operation returns in reasonable time:

[root@xxxxx-wekabe-1 minio]# time mc ls Minio/wekabucket1/358adf34/d/d/95/20/3696
[2018-10-24 13:30:47 IDT] 296KiB 50ef-4da8-ade2-02eae1f8ba36

real 0m3.420s
user 0m0.015s
sys 0m0.008s

mc version

Version: 2018-10-18T00:40:05Z
Release-tag: RELEASE.2018-10-18T00-40-05Z
Commit-id: 0dd32c1

[root@xxxx-wekabe-1 minio]# minio version
Version: 2018-10-18T00:28:58Z
Release-Tag: RELEASE.2018-10-18T00-28-58Z
Commit-ID: 3b55357

System information

8 servers - 12 disks per server (10TB each disk)

[harshavardhana]

thanks @eyalrif we will take a look

[harshavardhana]

I am not able to reproduce this issue @eyalrif - but we bringing a few features to be able to stop healing on a prefix. #2584

[kannappanr]

PR #2584 has been merged. Closing the issue with #2854 being a way to handle this issue. Please let us know if you have any concerns.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.