[core] add grpc opencensus plugin #39082

rynewang · 2023-08-29T22:00:43Z

gRPC has a built in plugin to export metrics via opencensus, as detailed in https://github.com/census-instrumentation/opencensus-specs/blob/master/stats/gRPC.md . We already use opencensus to collect Ray metrics, and here we add the gRPC-sourced metrics to the list.

In consideration of potential performance impacts, we disable it by default and can be enabled by component via config enable_grpc_metrics_collection_for. Now we only support gcs, in following patches we plan to support raylet and core_worker. The reason we don't support it right away is, we need to configure before any gRPC traffic, but in raylet and in core worker we init our stats after a grpc call to gcs/raylet to get the configs updated.

jjyao · 2023-08-30T15:26:47Z

Nice, it would be very valuable.

rynewang · 2023-08-30T20:32:14Z

it works now!

dashboard/modules/reporter/reporter_agent.py

src/ray/stats/metric.cc

rynewang · 2023-08-30T23:03:32Z

wait, it compiles in my laptop but not in CI

jjyao

Thanks! LGTM but I'll defer to @rkooo567 since he is more familiar with this code.

rkooo567 · 2023-08-30T23:16:29Z

can we make the grpc stats off by default (add a system config to enable it)?

Also can you give me a list of grpc stats? I wonder if we should filter some of them since it will be a lot.

Lastly, can you pick which stats we should track from the multi cloud setup?

rynewang · 2023-08-31T00:20:46Z

can we make the grpc stats off by default (add a system config to enable it)?

config added.

Also can you give me a list of grpc stats? I wonder if we should filter some of them since it will be a lot.

https://github.com/census-instrumentation/opencensus-specs/blob/master/stats/gRPC.md

Lastly, can you pick which stats we should track from the multi cloud setup?

grpc.io/client/roundtrip_latency and grpc.io/server/server_latency is my special interest

rkooo567 · 2023-08-31T05:37:47Z

SGTM. I guess we can also track the completed rpcs with status to understand the network failures

Why no error counts?
Error counts can be computed on your metrics backend by totalling the different per-status values.

rkooo567

LGTM! Excited to have grpc metrics :)! Nit comments on adding unit tests

python/ray/_private/metrics_agent.py

src/ray/stats/metric_exporter.cc

rkooo567

TODO

Allow to choose components (gcs, worker, raylet) to export grpc metrics due to cardinality concern.

I think we can do it by creating a separate method & call it in the beg of the entrypoint script

rynewang · 2023-09-05T21:17:38Z

Blocked on #39210 because we need grpc to be upgraded to >= 1.49.0 to have grpc/grpc#30567. Once our #39210 is merged I will merge to this PR and CI should work.

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

… fix. Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

src/ray/common/ray_config_def.h

python/ray/_private/metrics_agent.py

rkooo567 · 2023-09-09T11:21:06Z

Can you merge the latest master? Also lint failure + left of comments

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

rynewang · 2023-09-10T23:04:27Z

ready to merge

rkooo567 · 2023-09-11T10:26:25Z

Can you update the PR description? Also, test_prometheus_physical_stats_record failure seesm related

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

rynewang · 2023-09-11T21:07:15Z

updated

gRPC has a built in plugin to export metrics via opencensus, as detailed in https://github.com/census-instrumentation/opencensus-specs/blob/master/stats/gRPC.md . We already use opencensus to collect Ray metrics, and here we add the gRPC-sourced metrics to the list. In consideration of potential performance impacts, we disable it by default and can be enabled by component via config enable_grpc_metrics_collection_for. Now we only support gcs, in following patches we plan to support raylet and core_worker. The reason we don't support it right away is, we need to configure before any gRPC traffic, but in raylet and in core worker we init our stats after a grpc call to gcs/raylet to get the configs updated. Signed-off-by: Simon Zehnder <simon.zehnder@gmail.com>

gRPC has a built in plugin to export metrics via opencensus, as detailed in https://github.com/census-instrumentation/opencensus-specs/blob/master/stats/gRPC.md . We already use opencensus to collect Ray metrics, and here we add the gRPC-sourced metrics to the list. In consideration of potential performance impacts, we disable it by default and can be enabled by component via config enable_grpc_metrics_collection_for. Now we only support gcs, in following patches we plan to support raylet and core_worker. The reason we don't support it right away is, we need to configure before any gRPC traffic, but in raylet and in core worker we init our stats after a grpc call to gcs/raylet to get the configs updated. Signed-off-by: Victor <vctr.y.m@example.com>

rynewang assigned rkooo567 Aug 29, 2023

rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 30, 2023

rynewang marked this pull request as ready for review August 30, 2023 20:40

rynewang assigned jjyao Aug 30, 2023

rynewang removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 30, 2023

jjyao reviewed Aug 30, 2023

View reviewed changes

dashboard/modules/reporter/reporter_agent.py Outdated Show resolved Hide resolved

src/ray/stats/metric.cc Outdated Show resolved Hide resolved

jjyao reviewed Aug 30, 2023

View reviewed changes

rkooo567 approved these changes Aug 31, 2023

View reviewed changes

python/ray/_private/metrics_agent.py Outdated Show resolved Hide resolved

python/ray/_private/metrics_agent.py Outdated Show resolved Hide resolved

python/ray/_private/metrics_agent.py Outdated Show resolved Hide resolved

src/ray/stats/metric_exporter.cc Show resolved Hide resolved

rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 31, 2023

rkooo567 requested changes Sep 5, 2023

View reviewed changes

rynewang requested a review from a team as a code owner September 6, 2023 00:08

rynewang added 11 commits September 7, 2023 16:31

add grpc opencensus plugin

676a254

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

directly register grpc

f5713cf

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

TODO: merge dists

6d4e79e

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

fix the metrics incompatibilities

1b4d3ac

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

move the spaces

59a4be4

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

removed bad log, added global tags to grpc things

a0c8713

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

add flags to diable it

1e30a97

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

add flag to test

2ecad8f

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

fix issue that grpc needs to be configured early. added tests for the…

af9a56b

… fix. Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

remove bad header

36d333f

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

move grpc enablement to gcs only

74ed3d3

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

rynewang force-pushed the grpc-stats branch from c58aab2 to 74ed3d3 Compare September 7, 2023 23:33

add new cpp test, fix existing test

4d8e4ab

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

rkooo567 approved these changes Sep 8, 2023

View reviewed changes

src/ray/common/ray_config_def.h Outdated Show resolved Hide resolved

python/ray/_private/metrics_agent.py Show resolved Hide resolved

rynewang added 3 commits September 9, 2023 12:16

update flag to string, optimise python side metric fix func

2a44aa4

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

fix bazel

dcc3b90

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

Merge remote-tracking branch 'origin/master' into grpc-stats

51cad14

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

rynewang removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 10, 2023

rkooo567 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 11, 2023

fix tests

c2eb1f0

Signed-off-by: Ruiyang Wang <rywang014@gmail.com>

rynewang removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Sep 11, 2023

rkooo567 merged commit 77b4cb9 into ray-project:master Sep 13, 2023
107 of 112 checks passed

rickyyx mentioned this pull request Oct 24, 2023

[core] microbenchmark regression #40606

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] add grpc opencensus plugin #39082

[core] add grpc opencensus plugin #39082

rynewang commented Aug 29, 2023 •

edited

jjyao commented Aug 30, 2023

rynewang commented Aug 30, 2023

rynewang commented Aug 30, 2023

jjyao left a comment

rkooo567 commented Aug 30, 2023

rynewang commented Aug 31, 2023

rkooo567 commented Aug 31, 2023

rkooo567 left a comment •

edited

rkooo567 left a comment

rynewang commented Sep 5, 2023

rkooo567 commented Sep 9, 2023

rynewang commented Sep 10, 2023

rkooo567 commented Sep 11, 2023

rynewang commented Sep 11, 2023

[core] add grpc opencensus plugin #39082

[core] add grpc opencensus plugin #39082

Conversation

rynewang commented Aug 29, 2023 • edited

jjyao commented Aug 30, 2023

rynewang commented Aug 30, 2023

rynewang commented Aug 30, 2023

jjyao left a comment

Choose a reason for hiding this comment

rkooo567 commented Aug 30, 2023

rynewang commented Aug 31, 2023

rkooo567 commented Aug 31, 2023

rkooo567 left a comment • edited

Choose a reason for hiding this comment

rkooo567 left a comment

Choose a reason for hiding this comment

rynewang commented Sep 5, 2023

rkooo567 commented Sep 9, 2023

rynewang commented Sep 10, 2023

rkooo567 commented Sep 11, 2023

rynewang commented Sep 11, 2023

rynewang commented Aug 29, 2023 •

edited

rkooo567 left a comment •

edited