[Serve] Add objref resolution latency metric

[vaishdho1]

Description

The PR adds a new Prometheus histogram metric serve_objref_resolution_latency_ms that tracks how long the Serve router spends resolving upstream DeploymentResponse arguments before a request enters the routing queue.
This gives visibility into resolution wait time that was previously hidden as part of fulfillment_time_ms .

Related issues

Fixes #62286

Additional information

Used the following reproduction script that compares two cases: one passing an unresolved DeploymentResponse as an argument and another passing a plain dict, to isolate the resolution overhead.

@serve.deployment(num_replicas=1)
class SlowDependency:
    async def __call__(self, request=None):
        await asyncio.sleep(2)
        return {"source": "upstream", "ts": time.time()}

@serve.deployment(num_replicas=1)
class DownstreamUnresolved:
    async def __call__(self, upstream_result):
        return {
            "downstream_received": upstream_result,
            "processed_at": time.time(),
        }


@serve.deployment(num_replicas=1)
class PipelineUnresolved:
    def __init__(self, upstream_handle, downstream_handle):
        self.upstream = upstream_handle
        self.downstream = downstream_handle

    async def __call__(self, request):
        upstream_resp = self.upstream.remote()
        return await self.downstream.remote(upstream_resp)

@serve.deployment(num_replicas=1)
class DownstreamPreresolved:
    async def __call__(self, upstream_result):
        return {
            "downstream_received": upstream_result,
            "processed_at": time.time(),
        }


@serve.deployment(num_replicas=1)
class PipelinePreresolved:
    def __init__(self, upstream_handle, downstream_handle):
        self.upstream = upstream_handle
        self.downstream = downstream_handle

    async def __call__(self, request):
        upstream_result = await self.upstream.remote()
        return await self.downstream.remote(upstream_result)

up_a = SlowDependency.bind()
down_a = DownstreamUnresolved.bind()
pipe_a = PipelineUnresolved.bind(up_a, down_a)
serve.run(pipe_a, name="pipeline_unresolved", route_prefix="/pipeline-unresolved")

up_b = SlowDependency.bind()
down_b = DownstreamPreresolved.bind()
pipe_b = PipelinePreresolved.bind(up_b, down_b)
serve.run(pipe_b, name="pipeline_preresolved", route_prefix="/pipeline-preresolved")

[gemini-code-assist]

Code Review

This pull request introduces a new metric, serve_objref_resolution_latency_ms, to track the time spent resolving upstream ObjectRef or DeploymentResponse arguments in the Ray Serve router. The changes include the metric initialization, measurement logic within the routing process, and a corresponding test case. Feedback suggests using time.monotonic() instead of time.time() for more reliable duration measurements, as the latter is susceptible to system clock adjustments.

[harshit-anyscale]

cc: @abrarsheikh for further review and merge

[cursor]

Metric emitted for all requests, not just objref resolutions

Low Severity

The serve_objref_resolution_latency_ms metric is observed for every request where pr.resolved is False, which is the default for all new PendingRequest instances. This includes requests with plain arguments (no DeploymentResponse or ObjectRef). For these, _resolve_request_arguments iterates args, finds nothing to resolve, and returns almost instantly — but the metric still records a near-zero value. This dilutes the histogram with noise, making percentile calculations misleading when a deployment receives a mix of request types. The metric name and description claim it tracks "resolving upstream ObjectRef or DeploymentResponse arguments," so it would be more accurate to only observe it when actual resolution work occurred.

 

^{Reviewed by Cursor Bugbot for commit ad96b6c. Configure here.}

[abrarsheikh]

update monitoring.md file

[abrarsheikh]

Suggested change

	"serve_objref_resolution_latency_ms",
	"serve_router_args_resolution_latency_ms",

[vaishdho1]

Fixed in a2a7a06

[abrarsheikh]

self_actor_id cannot be None right?

[vaishdho1]

Thats right. It is always a string. Fixed in a2a7a06

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Reviewed by Cursor Bugbot for commit ab8b5e6. Configure here.}

[cursor]

Variable name inconsistent with renamed metric name

Low Severity

The attribute _objref_resolution_latency_ms was not renamed to match the metric name serve_router_args_resolution_latency_ms after the metric was renamed per reviewer feedback. The reviewer suggested changing from serve_objref_resolution_latency_ms to serve_router_args_resolution_latency_ms, and the metric name was updated, but the Python attribute name still uses the old "objref" terminology. This inconsistency between the internal variable name and the exported metric name can confuse future maintainers trying to grep for or understand the metric.

Additional Locations (1)

• python/ray/serve/_private/router.py#L948-L949

 

^{Reviewed by Cursor Bugbot for commit ab8b5e6. Configure here.}