Health metrics (Part 2)

[Nadine-H]

Part of #2736

Adding two custom metrics:

• dstack_submit_to_provision_duration_seconds: Time from when a run has been submitted and first job provisioning
• dstack_pending_runs_total: Total number of pending runs

We can add metrics later too, but I think for now these two are helpful to see if there are runs stuck in SUBMITTED or PENDING states, which could be due to an issue with dstack or the underlying infrastructure.

[un-def]

I don't think the background package is the right place for this module. Not sure what is the best place, maybe server.services.prometheus? We can convert it to a package with submodules to separate the existing functions from the new RunMetrics class/instance.

cc @r4victor

[Nadine-H]

Makes sense, I'm not sure of the best naming that would distinguish the existing prometheus.py module and the new metrics.py. The major difference between them is how the metrics are collected; the earlier is more of a pull approach where the metrics are fetched when the /metrics endpoint is invoked, and the latter is more a push approach where the metrics are written to the prometheus client right when the runs are being processed. So maybe something like pull_metrics and push_metrics?