A Option to disable self-healing

[bratao]

Hi, after the daily self-healing feature, my minio cluster is most of time using 100% of my HD IO.

This cluster contains billions of small objects, and even the shallow healing takes ages.

Describe the solution you'd like
A way to disable the daily self-healing.

[nitisht]

@bratao request you to please follow the issue template to ensure we understand the overall scenario here.

[deekoder]

Yes @bratao please give us more details on your env. It will help us out a lot. We will discuss internally and let you know whether we can support this request.

[poornas]

@bratao, we can definitely add an option to turn healing on/off thru a mc admin command option. While it may appear that healing is aggressive here, it is actually designed to yield to any incoming requests to the server, so it will not impact server performance.

We are working actively on making improvements to healing - and would like your feedback on whether you would use 1) an option to choose healing frequency 2) an option to set priority of healing thread (low, medium, high) in your setup. Between the two options, which would be most useful to you and why? It would also be helpful to know the underlying disk(s) in use here, and usage stats on your cluster to see how healing can be improved for your use case.

[bratao]

Hello @poornas , I would prefer to have a option to define the frequency/disable.

Looking at "iotop" I saw hundreds of threads consuming all the available disk IO. I would guess that the priority would not solve this problem.

For this cluster(4 machines with erasure, approximately 5 Millions files , in a regular HD) it took 18 hours to finish the healing in the version RELEASE.2019-07-05T21-20-21Z. But it ran daily without the option to disable.

After reversing to a version without self-healing, everything went good again.

Thank you!

[poornas]

@bratao, can you upgrade to the latest release RELEASE.2019-07-10T00-34-56Z and see if you still see high I/O - recent improvements to healing should mitigate the high I/O issue you were seeing.

[harshavardhana]

Healing has moved to once in a month @bratao

[bratao]

Thanks everyone. This fix it for me

[harshavardhana]

Fixed by #7868 #7934

[adiospeds]

Just to add to the conversation. I loved it when daily auto healing feature was added. However, even I have a cluster with loads of small files. Even I had recently upgraded to the daily auto healing version and noticed that my cpu was always high and kept receiving lot of socket timeouts on my monitoring apps. Hopefully upgrading to this version will fix this issue for me. But I'd like to have a way to control the auto healing frequency. Some kind of setting in the config file, bcas daily healing worked great for small cluster on vm's with lesser medium sized files. These vm's were oftenly brought down for patching purposes and daily healing fixed the missing files on them. I guess the healing frequency is moreover dependent on size of files on cluster and frequency at which the cluster nodes go down. Providing a knob to tune the frequency of healing would be greatly helpful.

[poornas]

@adiospeds, auto heal once a month was released 2 days back in RELEASE.2019-07-17T22-54-12Z. If this still does not work for you, please feel free to open a new issue.Will bring up your feedback with the team on a frequency knob for healing.