[core] add perf metrics for 2.54.0 by khluu · Pull Request #60994 · ray-project/ray

khluu · 2026-02-12T00:53:18Z

REGRESSION 47.55%: single_client_put_gigabytes (THROUGHPUT) regresses from 18.177676237873847 to 9.534215154530786 in microbenchmark.json
REGRESSION 29.79%: 1_1_async_actor_calls_sync (THROUGHPUT) regresses from 1433.5390152119278 to 1006.5372823091388 in microbenchmark.json
REGRESSION 28.18%: 1_1_actor_calls_sync (THROUGHPUT) regresses from 1989.7334090798931 to 1429.043792654831 in microbenchmark.json
REGRESSION 23.35%: single_client_tasks_sync (THROUGHPUT) regresses from 844.7209532677355 to 647.4580540478063 in microbenchmark.json
REGRESSION 21.10%: 1_1_actor_calls_async (THROUGHPUT) regresses from 8591.539773771437 to 6778.77263118084 in microbenchmark.json
REGRESSION 20.12%: tasks_per_second (THROUGHPUT) regresses from 379.30168512953065 to 302.9857172164898 in benchmarks/many_nodes.json
REGRESSION 18.48%: single_client_tasks_async (THROUGHPUT) regresses from 6769.634231009387 to 5518.2930278813765 in microbenchmark.json
REGRESSION 18.31%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 12.521640061929554 to 10.22955093525802 in microbenchmark.json
REGRESSION 15.82%: n_n_actor_calls_async (THROUGHPUT) regresses from 22593.67022851302 to 19019.07842122874 in microbenchmark.json
REGRESSION 15.75%: 1_n_actor_calls_async (THROUGHPUT) regresses from 6838.2845805526595 to 5761.245883325508 in microbenchmark.json
REGRESSION 15.08%: single_client_wait_1k_refs (THROUGHPUT) regresses from 4.716418799247922 to 4.0053068721409355 in microbenchmark.json
REGRESSION 14.83%: n_n_actor_calls_with_arg_async (THROUGHPUT) regresses from 3263.220674257469 to 2779.4109021768286 in microbenchmark.json
REGRESSION 12.05%: 1_n_async_actor_calls_async (THROUGHPUT) regresses from 6280.583274671035 to 5523.506827202886 in microbenchmark.json
REGRESSION 11.46%: multi_client_tasks_async (THROUGHPUT) regresses from 20114.199533908533 to 17809.169218546187 in microbenchmark.json
REGRESSION 10.56%: n_n_async_actor_calls_async (THROUGHPUT) regresses from 19945.253372184772 to 17839.72259466486 in microbenchmark.json
REGRESSION 9.86%: 1_1_actor_calls_concurrent (THROUGHPUT) regresses from 4965.99522007048 to 4476.469992509627 in microbenchmark.json
REGRESSION 8.55%: placement_group_create/removal (THROUGHPUT) regresses from 678.9244842339416 to 620.8842587027023 in microbenchmark.json
REGRESSION 7.43%: pgs_per_second (THROUGHPUT) regresses from 18.933742626442143 to 17.52771503829712 in benchmarks/many_pgs.json
REGRESSION 7.42%: 1_1_async_actor_calls_async (THROUGHPUT) regresses from 3853.261228971964 to 3567.5020424578674 in microbenchmark.json
REGRESSION 4.62%: 1_1_async_actor_calls_with_args_async (THROUGHPUT) regresses from 2490.991223668351 to 2375.9373482961805 in microbenchmark.json
REGRESSION 3.54%: client__put_gigabytes (THROUGHPUT) regresses from 0.10217222369611438 to 0.09855542381003202 in microbenchmark.json
REGRESSION 353.19%: dashboard_p95_latency_ms (LATENCY) regresses from 25.72 to 116.561 in benchmarks/many_nodes.json
REGRESSION 220.39%: dashboard_p99_latency_ms (LATENCY) regresses from 55.245 to 176.997 in benchmarks/many_nodes.json
REGRESSION 126.61%: dashboard_p50_latency_ms (LATENCY) regresses from 30.152 to 68.326 in benchmarks/many_actors.json
REGRESSION 60.21%: stage_3_creation_time (LATENCY) regresses from 1.3985865116119385 to 2.240680456161499 in stress_tests/stress_test_many_tasks.json
REGRESSION 59.36%: avg_iteration_time (LATENCY) regresses from 0.8959456539154053 to 1.4278212523460387 in stress_tests/stress_test_dead_actors.json
REGRESSION 28.35%: dashboard_p50_latency_ms (LATENCY) regresses from 3.729 to 4.786 in benchmarks/many_pgs.json
REGRESSION 25.98%: stage_0_time (LATENCY) regresses from 5.5611889362335205 to 7.005899667739868 in stress_tests/stress_test_many_tasks.json
REGRESSION 25.95%: stage_1_avg_iteration_time (LATENCY) regresses from 14.662270617485046 to 18.467847657203674 in stress_tests/stress_test_many_tasks.json
REGRESSION 18.96%: dashboard_p50_latency_ms (LATENCY) regresses from 5.21 to 6.198 in benchmarks/many_tasks.json
REGRESSION 10.69%: stage_2_avg_iteration_time (LATENCY) regresses from 41.472771883010864 to 45.90533618927002 in stress_tests/stress_test_many_tasks.json
REGRESSION 6.22%: dashboard_p95_latency_ms (LATENCY) regresses from 3504.953 to 3722.916 in benchmarks/many_actors.json

Signed-off-by: Lonnie Liu <lonnie@anyscale.com>

gemini-code-assist

Code Review

This pull request updates the performance metrics for release 2.54.0. While the changes correctly reflect the new benchmark results, they reveal several critical performance regressions. The pull request description lists numerous regressions, with some latency metrics showing degradation of over 300% and throughput dropping by nearly 50% in key microbenchmarks. I have added comments highlighting some of the most severe regressions. These issues are critical and should be thoroughly investigated and addressed before these metrics are accepted as the new baseline.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "dashboard_p50_latency_ms",
            "perf_metric_type": "LATENCY",
-            "perf_metric_value": 30.152
+            "perf_metric_value": 68.326


A 126.61% regression in dashboard_p50_latency_ms is a critical performance degradation. This significant increase in median latency for the many_actors benchmark needs investigation.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "dashboard_p95_latency_ms",
            "perf_metric_type": "LATENCY",
-            "perf_metric_value": 25.72
+            "perf_metric_value": 116.561


A 353.19% regression in dashboard_p95_latency_ms is a critical performance degradation. This indicates a severe issue that needs to be investigated and addressed.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "dashboard_p99_latency_ms",
            "perf_metric_type": "LATENCY",
-            "perf_metric_value": 55.245
+            "perf_metric_value": 176.997


A 220.39% regression in dashboard_p99_latency_ms is a critical performance degradation. This is likely related to the p95 latency regression and points to a severe issue with request tail latency.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "single_client_put_gigabytes",
            "perf_metric_type": "THROUGHPUT",
-            "perf_metric_value": 18.177676237873847
+            "perf_metric_value": 9.534215154530786


A 47.55% regression in single_client_put_gigabytes throughput is a critical performance degradation. This could have a major impact on workloads involving large object puts.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "dashboard_p50_latency_ms",
            "perf_metric_type": "LATENCY",
-            "perf_metric_value": 3.729
+            "perf_metric_value": 4.786


A 28.35% regression in dashboard_p50_latency_ms is a high-severity performance degradation. This increase in median latency for the many_pgs benchmark should be investigated.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "1_1_actor_calls_sync",
            "perf_metric_type": "THROUGHPUT",
-            "perf_metric_value": 1989.7334090798931
+            "perf_metric_value": 1429.043792654831


A 28.18% regression in 1_1_actor_calls_sync throughput is a high-severity performance degradation that should be investigated.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "1_1_async_actor_calls_sync",
            "perf_metric_type": "THROUGHPUT",
-            "perf_metric_value": 1433.5390152119278
+            "perf_metric_value": 1006.5372823091388


A 29.79% regression in 1_1_async_actor_calls_sync throughput is a high-severity performance degradation that warrants investigation.

gemini-code-assist · 2026-02-12T00:54:31Z

            "perf_metric_name": "avg_iteration_time",
            "perf_metric_type": "LATENCY",
-            "perf_metric_value": 0.8959456539154053
+            "perf_metric_value": 1.4278212523460387


A 59.36% regression in avg_iteration_time for the dead actors stress test is a high-severity performance degradation. This could indicate issues with actor management or fault tolerance mechanisms.

gemini-code-assist · 2026-02-12T00:54:32Z

            "perf_metric_name": "stage_3_creation_time",
            "perf_metric_type": "LATENCY",
-            "perf_metric_value": 1.3985865116119385
+            "perf_metric_value": 2.240680456161499


A 60.21% regression in stage_3_creation_time for the many tasks stress test is a high-severity performance degradation. This suggests a potential bottleneck in the task creation or scheduling path.

github-actions · 2026-02-26T12:30:05Z

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

Update performance metrics for 2.54.0

afa5b94

Signed-off-by: Lonnie Liu <lonnie@anyscale.com>

gemini-code-assist Bot reviewed Feb 12, 2026

View reviewed changes

aslonnie requested review from a team, MengjinYan, dayshah and edoakes February 12, 2026 00:57

aslonnie changed the title ~~Add perf metrics for 2.54.0~~ [core] add perf metrics for 2.54.0 Feb 12, 2026

github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Feb 26, 2026

ray-gardener Bot added the community-contribution Contributed by the community label Feb 27, 2026

github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Feb 27, 2026

Navigation Menu

Uh oh!

[core] add perf metrics for 2.54.0 - #60994

[core] add perf metrics for 2.54.0#60994
khluu wants to merge 1 commit into
masterfrom
2.54.0_perf_metrics

khluu commented Feb 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Uh oh!

github-actions Bot commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

khluu commented Feb 12, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants