Record feature flag evaluation metrics

[thompson-tomo]

Is there an existing issue for this?

I have searched the existing issues but can't find one

Is your feature request related to a problem? Please describe the problem.

I would like that the evaluation of feature flags results in metrics being recorded so that usage statistics can be visualised using our existing tool chain ie ELK.

Describe the solution you'd like

Each Feature flag has it's result stored in a dedicated instrument (histogram) with the name = Featuremanagement.<> so that both true & false results can be captured and the aggregation can than occur in the visualisation tools. At the same time additional metrics about usage of filters etc should be collected.

Additional context

No response

[zhiyuanliang-ms]

Hi, @thompson-tomo
The telemetry will be an important piece for the Feature Management in the feature. Actually, you can find we have supported telemetry in the preview branch. We are going to publish the preview release(in the following weeks) with the new features of variants and telemetry.
The telemetry about the feature flag state will be sent whenever the feature flag is evaluated.
You can refer to the README of preview branch for more details about the telemetry in FeatureManagement. Also #3 would be helpful.

I am digesting your request. Do you think telemetry will be a solution for your case? If not, what's the limitation?

Each Feature flag has it's result stored in a dedicated meter (histogram) with the name = Featuremanagement.

Are you expecting that the FeatureManagement library can generate something like log or live summary for you? If the answer is yes, which kind of form do you want it be, a file or a telemetry event?

You called out the usage of filters should be collected. That's a good point. If you go to see the ITelemetryPublisher, you will find that, currently, we pass EvaluationEvent to the telemetry publisher. The feature filters information of the feature flag will not be collected.

[thompson-tomo]

Hi @zhiyuanliang-ms

It is good to some investment is being made in the telemetry Space for feature flag management however just like the comment in #3, I question why a custom solution has been implemented rather than simply leveraging System.Diagnostics.Metrics. The reason I prefer this approach is that an OTEL collector can collect all metrics from a variety of sources within an application and transfer them to an analysis/visualisation tool such as ELK. At the same time a feature flag collector might only process FeatureManagement metrics and transfer just those to the corresponding tool. This approach allows for an OTEL collector to log metrics to file etc.

The measurements captured by the instruments would be:

• Feature Flag Evaluation stored in a Histogram. The name, type are stored as attributes. If variants are used the name of the variant is also stored as an attribute value, otherwise enabled/disabled is stored as an attribute.

I would also like to have the ability at the top level enable/disable telemetry rather than having to do it for each feature flag.

[zhiyuanliang-ms]

Hi @thompson-tomo
Thank you for the detailed explanation. You make the issue and potential user case so clear now.

I believe that your user case can somehow be accomplished if you implement a custom ITelemetry which uses System.Diagnostics.Metrics to set metrics. The EvaluationEvent will be passed as a parameter. The EvaluationEvent contains the FeatureDefinition where feature filter information is included. Also, the feature flag state and variant will also be included in the EvaluationEvent. The EvaluationEvent will be published whenever the feature flag evaluation happens

public interface ITelemetryPublisher
{
    public ValueTask PublishEvent(EvaluationEvent evaluationEvent, CancellationToken cancellationToken);
}

So, your ultimate question is WHY we choose a custom solution (ITelemetryPublisher) instead of leveraging System.Diagnostics (the same question was called out in #3).

One important reason is that we want the whole library focuses on DI usage (as mentioned in #3). However, for your case, actually System.Diagnostics.Metrics supports the DI usage. It makes it feasible to include it in the library. This definitly will be an enhancement for the library in the future. Maybe we can provide another built-in telemetry publisher built on System.Diagnostics.Metrics

From my side (maybe I am wrong), I think providing the interface ITelemetryPublisher is the appropriate way for us now. Because, it makes telemetry publisher work as plugins and users can decide what kind of telemetry or metric they want to have.

Any thoughts you have on the topic will be appreciated :)

[thompson-tomo]

Hi @zhiyuanliang-ms

You are correct, it could be implemented as a custom provider however I think it would be worthwhile to have the library natively instrumented with Metrics to lower the barrier to entry and further encourage adoption of OpenTelemtry which Microsoft is supporting. I like a plugin approach with my preference being to have them as opentelemetry collectors but can see the benefit of providing a richer model for collection and plugins to leverage.

I think from here it would be worthwhile to consider also offering metrics using System.Diagnostics.Metrics natively rather than a plugin and have a high level setting control if recording occurs and that way it simply works with a single setting. If a third party tool wants to visualise the metrics they are free to implement a plugin or even do an OTEL collector based on their needs.