feat(metrics): add sampled (de)serialization duration metrics for RPC

[Sunt-ing]

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

PLEASE DO NOT LEAVE THIS EMPTY !!!
Add sampled (de)serialization duration metrics for RPC with the given frequency

Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

#3284

[Sunt-ing]

Bench result in 3-compute-node mode:

[codecov]

Codecov Report

Merging #3618 (903a142) into main (70a281a) will decrease coverage by 0.01%.
The diff coverage is 59.61%.

@@            Coverage Diff             @@
##             main    #3618      +/-   ##
==========================================
- Coverage   74.38%   74.37%   -0.02%     
==========================================
  Files         776      776              
  Lines      110110   110155      +45     
==========================================
+ Hits        81905    81926      +21     
- Misses      28205    28229      +24     

Flag	Coverage Δ
rust	74.37% <59.61%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/compute/src/rpc/service/exchange_metrics.rs	0.00% <0.00%> (ø)
src/compute/src/rpc/service/exchange_service.rs	0.00% <0.00%> (ø)
src/stream/src/executor/merge.rs	92.33% <100.00%> (+0.41%)	⬆️
src/stream/src/executor/monitor/streaming_stats.rs	100.00% <100.00%> (ø)
src/meta/src/hummock/mock_hummock_meta_client.rs	40.56% <0.00%> (-0.95%)	⬇️
src/meta/src/manager/id.rs	94.94% <0.00%> (-0.57%)	⬇️
src/connector/src/filesystem/file_common.rs	80.35% <0.00%> (-0.45%)	⬇️
src/frontend/src/expr/utils.rs	98.99% <0.00%> (-0.26%)	⬇️
src/storage/src/hummock/local_version_manager.rs	81.38% <0.00%> (-0.12%)	⬇️
src/storage/src/hummock/iterator/merge_inner.rs	90.00% <0.00%> (+0.62%)	⬆️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[skyzh]

I believe this metrics is only used within scale service, and should not be put in dashboard.

[Sunt-ing]

No metric is used in all services. And since the panels of these two metrics locate in the "streaming actors" section, where most metrics that the scaling service needs locate, developers who do not care for scaling service can easily fold and unsee this section.

[skyzh]

The metrics you added do not convey any actual meaning. If developers observe "30ms" from the metrics, what can they know about the system?

[skyzh]

... and thus it's better not to add it to dashboard.

[Sunt-ing]

IMO, since sampling is used, the important information is not the accurate number but the existence and trend. If the graph of this metric exists, developers can know that this metric is not been disrupted. If this metric is growing, developers can know how the (de)serialization durations change.
Also, IMO this Grafana panel is temporal and will be removed after the Orchestrator is fully built. And that, we may use other methods to observe it.

[skyzh]

IMO, since sampling is used, the important information is not the accurate number but the existence and trend.

If we want to explore trend, we should use rate function.

If this metric is growing, developers can know how the (de)serialization durations change.

It will change as long as there's an exchange. There's no need to have yet another metrics to prove a fact.

Also, IMO this Grafana panel is temporal and will be removed after the Orchestrator is fully built. And that, we may use other methods to observe it.

You can just leave it on your local branch, instead of merging into main.

[Sunt-ing]

With the rapid development, rebase can be miserable. Also, others who also develop the scaling service can benefit from it. I don't think this brings more overhead than its benefits.

[skyzh]

I'd only approve for the changes of metrics and not the dashboard. You can find another reviewer who agree on you.

[Sunt-ing]

Thanks for your review of the changes of metrics.