Add dedicated /metrics endpoints for Gateway and LLM services

[Copilot]

• Understand issue 8.3.2 — Dedicated Metrics Endpoints for Gateway and LLM Services #39 requirements for dedicated metrics endpoints
• Explore repository structure and existing implementations
• Add /metrics endpoint to gateway service
• Add /metrics endpoint to LLM service
• Update K8s deployments - prometheus.io/path: "/metrics"
• Update ServiceMonitor - gateway and LLM paths to /metrics
• Update documentation with health vs metrics distinction
• Add tests for metrics functionality
• Refactored to use prometheus_client for Prometheus-compatible format (review feedback)
  • Gateway metrics: Counter, Gauge, Histogram for requests, connections, latencies
  • LLM metrics: Counter, Histogram for requests, latencies, tokens
  • Both /metrics endpoints now return OpenMetrics text format
  • Made endpoints async for thread safety
• All tests pass (90.26% coverage)
• Linting passes
• CodeQL security scan passed (0 alerts)

Original prompt

"isseu 39 - 8.3.2 — Dedicated Metrics Endpoints for Gateway and LLM Services" to gamedev agent

Created from Copilot CLI via the copilot delegate command.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[SorraTheOrc]

@copilot there have been many linting changes in main. Can you merge them into this branch

[Copilot]

@copilot there have been many linting changes in main. Can you merge them into this branch

I'm unable to merge main into this branch due to sandbox limitations - I can't pull branches from GitHub. You can merge main using either:

1. GitHub UI: Use the "Update branch" button on the PR page
2. Locally:

git fetch origin main
git merge origin/main --no-edit
git push origin copilot/add-dedicated-metrics-endpoints

Let me know once you've merged main and I can address any conflicts or linting issues that arise.

[SorraTheOrc]

@copilot Review feedback:\n\n1. The new /metrics endpoints return JSON, but Prometheus expects OpenMetrics/text format. The current implementation will not work with Prometheus ServiceMonitor or annotations as described. Please refactor to use the prometheus_client library to expose metrics in the correct format.\n2. The Gateway service's get_metrics endpoint is synchronous (def, not async def) but is accessed from both the event loop and thread pool, which can cause concurrency issues. Please make it async and ensure safe access to shared state.\n3. For performance, consider using collections.deque(maxlen=N) for latency samples instead of list+pop(0).\n\n@copilot Please address these issues to ensure Prometheus compatibility and robust metrics handling.

[SorraTheOrc]

@copilot see the review feedback inthe previous comment and fix it please.

[Copilot]

@copilot Review feedback:\n\n1. The new /metrics endpoints return JSON, but Prometheus expects OpenMetrics/text format. The current implementation will not work with Prometheus ServiceMonitor or a...

Addressed all review feedback in commit 3a9a364:

1. Prometheus format: Refactored Gateway and LLM metrics to use prometheus_client library. Both /metrics endpoints now return OpenMetrics text format instead of JSON.

2. Async endpoint: Made get_metrics async in both services for thread-safe access.

3. Performance: Replaced list+pop(0) with Prometheus's built-in Histogram metrics which handle latency bucketing efficiently.

Example metrics output now looks like:

# HELP gateway_requests_total Total number of requests processed
# TYPE gateway_requests_total counter
gateway_requests_total 150.0
# HELP gateway_request_latency_seconds Request latency in seconds
# TYPE gateway_request_latency_seconds histogram
gateway_request_latency_seconds_bucket{request_type="command",le="0.1"} 80.0

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com//advisories
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)