HDDS-3336. Metrics for Recon OzoneManager DB sync.

[avijayanhwx]

What changes were proposed in this pull request?

Added some useful metrics for tracking how Recon's OM DB requests are going on.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3336

How was this patch tested?

Manually tested on top of #715.
Added unit tests.

[avijayanhwx]

cc @vivekratnavel / @swagle

[swagle]

Minor comment, overall LGTM.

[avijayanhwx]

Thanks for the review @swagle.

[vivekratnavel]

Overall LGTM

[elek]

Thanks the patch @avijayanhwx

FYI: HDDS-3239 introduced message level metrics for all the RPC endpoints. I think some of the metrics can be removed from this patch (eg. generic counter / latency)

[avijayanhwx]

Thanks the patch @avijayanhwx

FYI: HDDS-3239 introduced message level metrics for all the RPC endpoints. I think some of the metrics can be removed from this patch (eg. generic counter / latency)

Thank you for the information Marton! I am wondering if I should still keep the counter and time to keep it specific per recon client instance. In the future, more clients may use the same request with different input arguments, and the metrics will be averaged out across all clients.

[elek]

Thank you for the information Marton! I am wondering if I should still keep the counter and time to keep it specific per recon client instance. In the future, more clients may use the same request with different input arguments, and the metrics will be averaged out across all clients.

I am not sure if it will ever happen related to the OM db download RPC, but how knows....

But if you need a client side metrics: what about adding a ProtocolMessageMetrics to the OzoneManagerProtocolClientSideTranslatorPB (with some client specific tag, if needed). In that case you can have well-defined, unified metrics for all the OM calls on the recon side (without increasing the complexity anywhere else in the Recon code).

[avijayanhwx]

Thank you for the information Marton! I am wondering if I should still keep the counter and time to keep it specific per recon client instance. In the future, more clients may use the same request with different input arguments, and the metrics will be averaged out across all clients.

I am not sure if it will ever happen related to the OM db download RPC, but how knows....

But if you need a client side metrics: what about adding a ProtocolMessageMetrics to the OzoneManagerProtocolClientSideTranslatorPB (with some client specific tag, if needed). In that case you can have well-defined, unified metrics for all the OM calls on the recon side (without increasing the complexity anywhere else in the Recon code).

@elek I think that is a good idea. I have removed the generic counter and timer metric for the RPC call, and I will do the client specific protocol message metrics in a future JIRA (since there is no other client who is going to use the getDBUpdates API in the near future).

[vivekratnavel]

@elek Please review

[elek]

@elek I think that is a good idea. I have removed the generic counter...

Thanks the answer. My feeling is that we can remove even more counters from these in the future, when we will have some generic metrics on the client side (especially with flexible tag support).

But I am fine even with the original patch. I am more interested about having a consensus about the middle-term plans and have a plan how it can be achieved.

I assume you need the numbers right now, and the proposed generic solution is not yet there...

I am merging it now, thanks the contribution.