HDDS-3234. Fix retry interval default in Ozone client.

[bharatviswa504]

What changes were proposed in this pull request?

change retry interval value from 1s -> 15s.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3234

How was this patch tested?

Tested with this value on a cluster, where we are doing billion object test.

[bshashikant]

@bharatviswa504 , i would prefer to not change it right away, because it might not hold good for all cases. Let's implement a smarter retry policy instead.

[bharatviswa504]

@bshashikant I agree with you. This is not a permanent solution, till exponential back off with exception based retry policy is implemented it is a temporary fix. As right now default 1s, we see that the system is doing a lot of retries, and the queue limit is reaching its max size very quickly. By changing it to 15s, we have observed that the queue limit is under control and at max reached around 200.

Do you see any issues with changing to 15s?

[dineshchitlangia]

@bshashikant I agree with you. This is not a permanent solution, till then it is a temporary fix. As right now default 1s, we see that the system is doing a lot of retries, and the queue limit is reaching its max size very quickly. By changing it to 15s, we have observed that the queue limit is under control and at max reached around 200.

Do you see any issues with changing to 15s?

In various other components outside of Ozone, I have seen a retry policy of 60s. Considering that, 15s is still reasonable for now.

[bshashikant]

@bshashikant I agree with you. This is not a permanent solution, till then it is a temporary fix. As right now default 1s, we see that the system is doing a lot of retries, and the queue limit is reaching its max size very quickly. By changing it to 15s, we have observed that the queue limit is under control and at max reached around 200.
Do you see any issues with changing to 15s?

In various other components outside of Ozone, I have seen a retry policy of 60s. Considering that, 15s is still reasonable for now.

@bharatviswa504 , the default retry policy would make it sleep for 15 sec even when a request fails with NotLeader or LeaderNotReady or in general any intermittent IO Exception from network as well. What instead we can do for now is., enforce ExceptionBasedRetryPolicy for ratis in Ozone and make it 15s for ResourceUnavailable which can be changed to exponential backoff retry policy in ozone later and for other exceptions make it 3s or so. What do you think?

[bharatviswa504]

@bshashikant I agree with you. This is not a permanent solution, till then it is a temporary fix. As right now default 1s, we see that the system is doing a lot of retries, and the queue limit is reaching its max size very quickly. By changing it to 15s, we have observed that the queue limit is under control and at max reached around 200.
Do you see any issues with changing to 15s?

In various other components outside of Ozone, I have seen a retry policy of 60s. Considering that, 15s is still reasonable for now.

@bharatviswa504 , the default retry policy would make it sleep for 15 sec even when a request fails with NotLeader or LeaderNotReady or in general any intermittent IO Exception from network as well. What instead we can do for now is., enforce ExceptionBasedRetryPolicy for ratis in Ozone and make it 15s for ResourceUnavailable which can be changed to exponential backoff retry policy in ozone later and for other exceptions make it 3s or so. What do you think?

Thank You @bshashikant for offline discussion, I will change this using request based retry policy combined with exception-based retry policy.

[bharatviswa504]

/pending "to address comments from @bshashikant"

[github-actions]

Marking this issue as un-mergeable as requested.

Please use /ready comment when it's resolved.

"to address comments from @bshashikant"

[arp7]

Let's get in the simple fix to increase the retry interval and keep working on the more sophisticated retry policy. The retry policy may need more extensive testing and may take more time to stabilize.