Fix all metrics

[aditya1702]

Closes #548

What

This PR overhauls the observability layer across all subsystems of the wallet-backend — DB queries, connection pools, ingestion pipeline, RPC client, and GraphQL API. The changes fall into six areas:

1. DB Query Metrics — Correctness & Error Classification

• Fixed metric emission ordering: QueriesTotal is now incremented before the error check, so every query (success or failure) is counted.
  Previously, early-return error paths skipped the counter, under-reporting total query volume.
• Error classification: Error labels now use utils.GetDBErrorType(err) instead of a generic "query_error" string, enabling breakdown by error
  category (timeout, connection, constraint violation, etc.).
• Removed unused TransactionsTotal / TransactionDuration metrics from DBMetrics — these were never populated by any call site.

2. Ingestion Metrics — Retry, Error & Lag Observability

• Added RetriesTotal, RetryExhaustionsTotal, and ErrorsTotal counters with a phase label (ledger_fetch, db_persist, batch_flush,
  ingest_live) to track retry behavior and failure modes.
• Added LedgerFetchDuration histogram to measure the full retry-inclusive latency of fetching a ledger from the backend.
• Added LagLedgers gauge that captures how far behind live ingestion is from the RPC tip at startup.
• Changed Duration from a HistogramVec (with no labels) to a plain Histogram — removing the unnecessary label dimension.
• Tuned histogram buckets to better match real-world ingestion latencies.

3. RPC Metrics — Histograms, Gauges & Structure

• Added ServiceHealth gauge (1 = healthy, 0 = unhealthy) set during the heartbeat loop and on GetHealth failures.
• Added LatestLedger gauge updated on each successful health check.
• Replaced the transport-level RequestDuration summary with a histogram and added explicit RequestsTotal and RequestErrors counters at
  the transport layer.
• Added tailored histogram buckets (rpcDurationBuckets) covering 10ms–10s to match Stellar RPC latency profiles.
• Removed the unused GetHeartbeatChannel() method from the RPCServiceInterface.
• Added PromQL-ready doc comments to every metric field.

4. Pool Metrics — Worker Pool + DB Connection Pool

• Worker pool: renamed the channel label to pool_name (using ConstLabels) and added wallet_pool_tasks_waiting and
  wallet_pool_tasks_submitted_total gauges for queue-depth and throughput visibility.
• DB connection pool: expanded from 4 to 12 metrics — added constructing_conns, acquire_total, empty_acquire_total, canceled_acquire_total,
  new_conns_total, max_lifetime_destroy_total, max_idle_destroy_total to cover the full pgxpool stat surface.
• Renamed the misleading acquire_duration_seconds (a monotonic counter of cumulative wait time) to proper counters.

5. GraphQL Metrics — Operation-Level Observability

• Added a new operation-level metrics middleware (graphql_operation_metrics.go) that tracks end-to-end operation latency, in-flight operations,
  response size, and error counts — all labeled by operation name and type (query/mutation).
• Refactored the existing field-level middleware to be simpler: removed error tracking and duration from field interceptors (now handled at the operation
  level) and kept only field-level resolution counts.
• Added ResponseSize histogram and InFlightOperations gauge.
• Added comprehensive test coverage for both middleware layers.

6. DB Pool Config

• Added QueryExecMode to PoolConfig, allowing callers to override pgx's default CacheStatement mode when needed.

Why

The existing metrics had several gaps that made production debugging and alerting difficult:

• Silent failures: Query error counters were skipped on early-return paths, making dashboards under-report errors.
• No retry visibility: Ingestion retries and exhaustions were invisible — an operator couldn't tell if the system was recovering or silently failing.
• Coarse RPC observability: Only summaries (no histograms) were available, making it impossible to compute proper percentile alerts. There was no
  health gauge, so degraded RPC state required log-grepping.
• Pool blind spots: Worker pool queue depth and DB connection pool lifecycle stats (new connections, destroys, acquire contention) weren't tracked —
  making it hard to diagnose connection exhaustion or pool saturation.
• GraphQL metrics were field-only: There was no way to see operation-level latency, throughput, or in-flight concurrency — critical for API SLO
  tracking.

[aristidesstaffieri]

Code review

Found 3 issues:

1. ErrorsTotal comment claims a closed set of 7 error types, but classifyGraphQLError has an unbounded default branch. The default case at line 90 passes the raw code string through as a label value. Any unrecognized extension code creates a new time series, risking unbounded Prometheus cardinality. Fix: map unknown codes to "unknown" in the default branch.

wallet-backend/internal/metrics/graphql.go

Lines 37 to 40 in 6599e2f

	// ErrorsTotal counts GraphQL errors classified by type at the operation level.
	// Types: validation_error, parse_error, bad_input, auth_error, forbidden, internal_error, unknown.
	// Labels: operation_name, error_type.
	ErrorsTotal *prometheus.CounterVec

wallet-backend/internal/serve/middleware/graphql_operation_metrics.go

Lines 88 to 91 in 6599e2f

	return "internal_error"
	default:
	return code
	}

2. heartbeatChannel is dead code -- allocated but unreachable after GetHeartbeatChannel() was removed from the interface. The channel is still created in NewRPCService (line 67) and stored in the struct (line 42), but no production code reads or writes to it. Additionally, RPCServiceMock.GetHeartbeatChannel() in mocks.go:61 is an orphaned mock method for a method that no longer exists on the interface.

wallet-backend/internal/services/rpc_service.go

Lines 41 to 44 in 6599e2f

	httpClient utils.HTTPClient
	heartbeatChannel chan entities.RPCGetHealthResult
	metrics *metrics.RPCMetrics
	healthCheckWarningInterval time.Duration

wallet-backend/internal/services/rpc_service.go

Lines 66 to 72 in 6599e2f

	heartbeatChannel := make(chan entities.RPCGetHealthResult, 1)
	return &rpcService{
	rpcURL: rpcURL,
	httpClient: httpClient,
	heartbeatChannel: heartbeatChannel,
	metrics: rpcMetrics,

3. InFlightOperations gauge can permanently inflate on panic. Inc() fires at line 30 before next(ctx) is called. Dec() only fires inside the returned response handler at line 64. If next(ctx) or the response handler panics, gqlgen's RecoverHandler catches it at a higher level but Dec() never runs. The codebase has explicit panic() calls in generated GraphQL code and resolvers, making this realistic in practice. Fix: use defer m.metrics.InFlightOperations.Dec() immediately after Inc(), or guard with a defer-recover.

wallet-backend/internal/serve/middleware/graphql_operation_metrics.go

Lines 29 to 35 in 6599e2f

	func (m *GraphQLOperationMetrics) Middleware(ctx context.Context, next graphql.OperationHandler) graphql.ResponseHandler {
	m.metrics.InFlightOperations.Inc()
	startTime := time.Now()
	responseHandler := next(ctx)
	return func(ctx context.Context) *graphql.Response {

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}