Add per-route request metrics to the API server

[jayachandrakasarla]

Closes #66792

The API server emits process-level metrics but nothing per-route, so operators can't see request rate, latency, or error rate for individual endpoints.

This adds an ASGI middleware that records a counter and a latency histogram per completed request, tagged with the templated route, HTTP method, and status. The templated route (/api/v2/dags/{dag_id}/dagRuns) is used as the tag so per-dag_id requests don't explode metric cardinality.

Metrics go through the existing Stats backend, so they work with StatsD, Datadog, and OpenTelemetry. The middleware is only wired in when a metrics backend is enabled, so deployments without one doesn't export these.

Metrics from /metrics endpoint of otel collector running on port 28889

# HELP airflow_api_server_request_count_total 
# TYPE airflow_api_server_request_count_total counter
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/assets/events",status="200"} 2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/auth/login",status="307"} 1
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/dags/{dag_id}/dagRuns",status="200"} 2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/dags/{dag_id}/dagRuns",status="403"} 1
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/dags/{dag_id}/dagRuns/{dag_run_id}/hitlDetails",status="200"} 2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/importErrors",status="200"} 2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/monitor/health",status="200"} 2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/plugins",status="200"} 2
airflow_api_server_request_count_total{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/plugins/importErrors",status="200"} 2
# HELP airflow_api_server_request_duration_milliseconds 
# TYPE airflow_api_server_request_duration_milliseconds histogram
airflow_api_server_request_duration_milliseconds_bucket{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/assets/events",status="200",le="+Inf"} 2
airflow_api_server_request_duration_milliseconds_sum{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/assets/events",status="200"} 47.551493
airflow_api_server_request_duration_milliseconds_count{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/assets/events",status="200"} 2
airflow_api_server_request_duration_milliseconds_bucket{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/auth/login",status="307",le="+Inf"} 1
airflow_api_server_request_duration_milliseconds_sum{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/auth/login",status="307"} 2.166875
airflow_api_server_request_duration_milliseconds_count{instance="a682a7ed-927f-438d-bf78-e0693f498dab",job="test",method="GET",otel_scope_name="airflow._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",route="/api/v2/auth/login",status="307"} 1

Screenshots of PromQL queries:

And here's the screenshot of custom metrics dashboard on Grafana:

---

Was generative AI tooling used to co-author this PR?

• Yes (Claude Opus 4.8)

[pierrejeambrun]

Thanks for the PR.

Unfortunately there is already #64523 doing the same, which has already gone by a number of code review, enhancements, and dev list discussion.

This one is much more likely to get merged at this point.

Closing in favor of #64523.