Instrumenting the outbound sync channel (flagd → flag-server)

[raflFaisal]

Context:
Flagd is deployed across multiple servers, and we have encountered issues related to stale flags. During the investigation, we found that there are currently no metrics available to monitor the flagd sync interface or detect issues with the sync channel.

Problem Statement
Today, /metrics exposes inbound evaluation traffic (http_server_) and evaluation outcomes (feature_flag_flagd_), but there are no metrics for the flag-sync-server <-> flagd sync channel. As a result, we currently have no way to reliably alert on conditions such as:

• The sync stream being down
• Flags configuration becoming stale from flagd pov
• The upstream sync connection flapping
• Changes in sync payloads or inbound sync traffic

I'm working on a prototype to add the following metrics:

Proposed Metrics:

1. otelgrpc client stats handler on the sync gRPC client -> gives the standard rpc.client.* metrics.

   • rpc_client_duration_milliseconds_ -> Histogram -> Stream lifetime + _count{rpc_grpc_status_code} = "why streams ended"
   • rpc_client_response_size_bytes_ -> Histogram -> Payload size distribution + rate(_sum) = inbound bandwidth

2. flagd-specific sync-lifecycle metrics (otelgrpc records histograms only on RPC end, so for a long-lived SyncFlags stream those don't help real-time):

   • flagd_sync_stream_active -> UpDownCounter -> Number of currently open flagd → flag-server sync streams. 1 while the SyncFlags stream is open, 0 when disconnected.
   • flagd_sync_stream_reconnects_total -> Counter -> Total number of successful sync stream reconnections after an initial failure
   • flagd_sync_flag_config_received_total -> Counter -> Total number of flag-configuration payloads received from the flag-server over SyncFlags streams.
   • flagd_sync_flag_config_last_received_timestamp_seconds -> Gauge (seconds) -> Timestamp of the most recent flag-configuration push received from the flag-server.

Scope:
The current prototype is limited to the gRPC sync channel.

I'd be interested in feedback on the proposed metrics, particularly whether there are other sync-specific signals that would be useful for monitoring or alerting.

Let me know your thoughts, @toddbaert @beeme1mr.

Thanks !

[toddbaert]

Just to be perfectly clear, because flagd can be both a sync server and a sync client ... I'm guessing you are looking at adding metrics for the latter? If so, I would add some specific/standard gRPC metrics, but also some more general sync metrics that apply to all sync sources (HTTP, file, gRPC (sync), etc). The stream/connection stuff is gRPC-specific, but staleness (e.g. time since last successful config apply) applies to every source, so I'd rather not make it gRPC-only.

We already have feature_flag.flagd.sync.active_streams and sync.stream.duration for the server side sync metrics. We should keep the OTel semconv dot-notation under a feature_flag.flagd. prefix here too, and figure out how we want to represent the client/server distinction consistently; OTel splits client/server in the metric name I think (rpc.client./rpc.server.), so mirroring that is probably the cleanest, but we'd want to weigh renaming the existing server-side metrics against just adding client-side ones.

[raflFaisal]

Thanks @toddbaert for the nice suggestions, let me extend the scope and propose a concrete direction.

• This is the client side (flagd as sync client, connecting outbound to a flag-server / flagd-proxy / any flagd.sync.v1.FlagSyncService producer).

• Fully agree, staleness metrics and "did I get a config?" should apply to every sync provider (grpc, http, file etc). I'll refactor so the general metrics live one level up (core/pkg/sync/ with a small sync-metrics package) and each provider records into them; only the stream-lifecycle counters stay in the gRPC providers. Proposed split:

Universal (all sync providers):
feature_flag.flagd.sync.client.flag_config.applied -> Counter, +1 per successful config apply from any source
feature_flag.flagd.sync.client.flag_config.last_applied_timestamp -> Int64Gauge (unit s), wall-clock of last apply

Stream-based only (gRPC):
feature_flag.flagd.sync.client.stream.active -> UpDownCounter, 1 while the streaming RPC is open
feature_flag.flagd.sync.client.stream.reconnects -> Counter, +1 per successful reconnect after a drop

• Common attribute set: feature_flag.flagd.sync.type (grpc / http / file / kubernetes / blob), feature_flag.flagd.sync.uri, feature_flag.flagd.sync.selector (when non-empty). During the transition I'll keep the pre-existing flagd.sync.provider / flagd.sync.uri / flagd.sync.selector attribute keys emitted alongside the new ones so any current dashboards keep working; drop-old is a coordinated breaking release later.

• Rename the existing server-side metrics for symmetry:
  feature_flag.flagd.sync.active_streams → feature_flag.flagd.sync.server.active_streams
  feature_flag.flagd.sync.stream.duration → feature_flag.flagd.sync.server.stream.duration

Let me know if you have any further thoughts !

[raflFaisal]

Hey @toddbaert

I’ve implemented the proposed changes in PR #2051, including the client-side sync metrics and gRPC stream lifecycle metrics.

I also added a Grafana dashboard snapshot to the PR #2051 to demonstrate how the metrics can be used to detect stale configuration and sync issues.

Please let me know if you have any feedback or concerns about the metric naming or the client/server distinction. I’d be happy to incorporate any suggestions.

[toddbaert]

Sounds good. I will review in the next day or two!

[toddbaert]

Available in the next flagd release.