fix: API deprecation headers, OTel sampling, email queue drain, and pool metrics (#680 #681 #682 #683)

[Grace-CODE-D]

Summary

Resolves four independent backend issues across observability, reliability, and API contracts.

• versioning.rs API versioning does not deprecate old versions with sunset headers #683 — versioning.rs: Deprecation header was set to false; deprecated API versions produced no server-side warning.
• tracing_config.rs does not set a sampling rate for production traces #681 — tracing_config.rs: OTel standard env vars (OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG) were not read; production defaulted to 100 % sampling.
• shutdown.rs graceful shutdown does not drain the email queue before exiting #682 — shutdown.rs: Email queue worker shared the global shutdown timeout with blockchain workers, risking in-flight email loss on slow graceful exits.
• metrics.rs Prometheus metrics do not include database pool utilization #680 — metrics.rs: Prometheus did not expose SQLx connection pool utilisation metrics needed to diagnose DB bottlenecks.

---

Changes

fix(#683) — RFC 8594 Deprecation / Sunset headers (versioning.rs, API_SPEC.md)

• Fixed Deprecation: false → Deprecation: true (RFC 8594 boolean signals deprecation).
• Added DEPRECATED_VERSIONS constant (["v1"]) to track which versions are deprecated.
• Added tracing::warn! in versioning_middleware when a deprecated version is detected in the request header.
• Added tracing::warn! in v1_deprecation_middleware on every response so server-side monitoring can track client migration progress.
• Created services/api/API_SPEC.md documenting the deprecation policy, sunset date (2026-04-25), version selection, and migration guide.

fix(#681) — OTel sampling env vars with 10 % production default (tracing_config.rs, TRACING.md)

• Added pub fn sample_rate_from_env(default_rate: f64) -> f64 that reads OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG per the OpenTelemetry SDK environment variable specification.
• init_tracing now calls sample_rate_from_env at the top; OTel standard env vars take precedence over the legacy TRACE_SAMPLE_RATE config value.
• Default production sampling rate is 10 % (traceidratio at 0.1) when neither env var is set.
• Supports always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio.
• Updated TRACING.md to document all sampler options, the 10 % default, and updated all examples to use the OTel standard vars.

fix(#682) — Email queue drain with dedicated configurable timeout (shutdown.rs, main.rs)

• Added pub fn email_queue_drain_timeout() -> Duration to shutdown.rs reading EMAIL_QUEUE_DRAIN_TIMEOUT_SECS (default 60 s — more generous than the 30 s global timeout because losing in-flight emails is more costly than delaying exit).
• Gave the email queue worker its own ShutdownCoordinator::new(1) in main.rs, separate from the blockchain workers coordinator.
• Shutdown sequence now: (1) drain email queue with EMAIL_QUEUE_DRAIN_TIMEOUT_SECS, then (2) drain blockchain workers with SHUTDOWN_TIMEOUT_SECS. Both timeouts are independently configurable via env var.

fix(#680) — SQLx pool utilisation metrics (metrics.rs, grafana-dashboard.json)

• Added db_pool_connections_active (IntGaugeVec) — connections currently checked out.
• Added db_pool_connections_idle (IntGaugeVec) — connections sitting idle in the pool.
• Added db_pool_acquire_duration_seconds (HistogramVec) — time spent waiting to acquire a connection; buckets from 0.1 ms to 1 s.
• Added observe_pool_connections(pool, active, idle) and observe_pool_acquire(pool, duration) methods.
• Updated Grafana dashboard panel 7 to reference the real metric names (db_pool_connections_active / db_pool_connections_idle).
• Added Grafana panel 10 showing p50 / p95 pool acquire duration with a 100 ms alert threshold.

---

Closes

Closes #683
Closes #681
Closes #682
Closes #680

[drips-wave]

@Grace-CODE-D Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits