[Serve] Metric ray_serve_controller_num_starts doesn't show up

[architkulkarni]

What happened + What you expected to happen

1. Start Ray with ray start --head
2. Run the Ray Serve quickstart in ipython (leaving the script open)

import requests
from starlette.requests import Request
from typing import Dict

from ray import serve


# 1: Define a Ray Serve application.
@serve.deployment
class MyModelDeployment:
    def __init__(self, msg: str):
        # Initialize model state: could be very large neural net weights.
        self._msg = msg

    def __call__(self, request: Request) -> Dict:
        return {"result": self._msg}


app = MyModelDeployment.bind(msg="Hello world!")

# 2: Deploy the application locally.
serve.run(app, route_prefix="/")

# 3: Query the application and print the result.
print(requests.get("http://localhost:8000/").json())
# {'result': 'Hello world!'}

3. Inspect the metrics at http://localhost:57482/, or whatever port is specified in /tmp/ray/prom_metrics_service_discovery.json. If you search for num_starts there are no results, despite the fact that other Serve metrics appear to be there such as ray_serve_controller_num_control_loops.
4. The problem persists after killing the Serve controller with ray.kill(ray.get_actor("SERVE_CONTROLLER_ACTOR")) and rerunning the quickstart code in the same ipython session.

Versions / Dependencies

master

Reproduction script

above

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[edoakes]

@shrekris-anyscale PTAL