TSDB: Modeling of delta metrics in elasticsearch

[lalit-satapathy]

Delta metrics are emitted by GCP and needs to be stored in elastic TSDS. The definition of delta metric is "The change in a value during a time interval"

The current proposal is to model the delta metrics as gauge, in the GCP integration metrics.

Suggestion by Felix: I think we should just map GCP DELTA metrics to gauges. I wouldn’t consider this a hack or a short-term solution as I don’t expect this have any negative impact.
Maybe the reason they have different types for delta vs gauge is that the consumers can differ between the two so that they can convert delta metrics into cumulative counters if they need to. But ES has excellent support for both deltas and counters so I don’t think that’s necessary.

Keeping this issue open to track any comments/suggestions for the same and potential future requirements to support delta metrics natively as a time_series_metric directly.

CC: @constanca-m @mlunadia @felixbarny

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[ruflin]

One area where it could be helpful to have knowledge about delta metrics is when building queries. Assuming we know a value is a delta and not a gauge value, we could recommend some aggregations vs others and explain to the user in the UI why one should be used vs the other.

[felixbarny]

The thing is that a lot of protocols don't have the distinction between gauge vs delta. For example, in OTLP, there's only the Gauge data point type. See also https://opentelemetry.io/docs/specs/otel/metrics/data-model/#opentelemetry-protocol-data-model. So I don't feel like it makes sense to invest in a delta metric type in Elasticsearch if it's only beneficial for GCP metrics and if the only benefits are that we could potentially recommend different aggregations.

[martijnvg]

@felixbarny If I understand your comment correctly, then a delta gauge isn't really needed now? So we can close this issue?

[felixbarny]

I think what I said back then wasn't entirely correct. In OpenTelemetry, there's the concept of temporality that describes whether a counter is delta or cumulative.

I agree with Ruflin that you'd perform different default aggregations depending on whether something is a delta counter vs a gauge.

Therefore, I think we shouldn't close this issue and instead work on properly supporting metric temporalities. Having said that, I don't feel like this is very urgent.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)