[Feature Request] [Rust SDK] Add Exponential/Native Histograms

[mckalexee]

Is your feature request related to a problem? Please describe.

Our organization has moved over to native histograms wherever we can due to the increased granularity without blowing up cardinality in our observability platform (Chronosphere). It's also tedious to manage bucket sizes manually when we have activities that take very different amounts of time to complete.

We had a specific request from an engineer to add more buckets to temporal_activity_succeed_endtoend_latency, but it would be much more convenient if we could do this within the SDK itself.

Describe the solution you'd like

OpenTelemetry exponential histograms, which Prometheus-compatible backends store as native histograms, are already supported by the opentelemetry_sdk crate pinned in this project.

I'm not the biggest rust-head, but through some LLM help I have the interface for the minimum change that should enable this. My goal with this approach is to keep existing metric names consistent, and allow emitting both metric types to allow cutover. Since you cannot export a native and traditional histogram with the same metric name in OpenTelemetry, the native histograms carry a suffix, _native by default, in every mode. The suffix can be changed but not removed, so switching between modes never renames a series.

Here is the proposed change to crates/common/src/telemetry.rs, based off of the pattern established in LoggerFormat.

/// Selects how histogram instruments are aggregated by the OTLP exporter.
#[derive(Debug, Default, Clone)]
#[non_exhaustive]
pub enum HistogramAggregation {
    /// Fixed bucket boundaries: core's defaults per metric, or `histogram_bucket_overrides`.
    #[default]
    ExplicitBuckets,
    /// OpenTelemetry base-2 exponential histogram only, under the metric name plus
    /// `exponential_name_suffix`. Backends expose it as a native histogram.
    Base2Exponential {
        /// Maximum buckets in each of the positive and negative ranges. OTel default: 160.
        max_size: u32,
        /// Maximum scale factor, -10 to 20. OTel default: 20. Prometheus reduces any scale
        /// above 8 on ingest, so higher values buy nothing there.
        max_scale: i8,
    },
    /// Emit both: the classic histogram unchanged, plus an exponential copy under the
    /// metric name plus `exponential_name_suffix`.
    Both {
        max_size: u32,
        max_scale: i8,
    },
}

pub struct OtelCollectorOptions {
    // existing fields ...
    /// Histogram aggregation for all histogram instruments. Defaults to explicit buckets.
    #[builder(default)]
    pub histogram_aggregation: HistogramAggregation,
    /// Appended to the metric name of every exponential histogram stream, in both
    /// `Base2Exponential` and `Both`. Default: "_native". Must be non-empty: OTLP cannot
    /// carry a classic and an exponential metric under one name, and a fixed suffix means
    /// the native series has the same name in every mode, so a migration never renames it.
    #[builder(default = Cow::Borrowed("_native"))]
    pub exponential_name_suffix: Cow<'static, str>,
}

Along with a change to crates/sdk-core-c-bridge/src/runtime.rs for .NET.

#[repr(C)]
pub struct OpenTelemetryOptions {
    // existing fields ...
    pub histogram_aggregation: OpenTelemetryHistogramAggregation,
    /// 0 means the OTel default (160).
    pub exponential_histogram_max_size: u32,
    pub exponential_histogram_max_scale: i8,
    pub exponential_histogram_max_scale_set: bool,
    /// Appended to every exponential stream's name. Empty means the default, "_native".
    pub exponential_name_suffix: ByteArrayRef,
}

#[repr(C)]
pub enum OpenTelemetryHistogramAggregation {
    ExplicitBuckets = 0,
    Base2Exponential = 1,
    Both = 2,
}

This will also require updates to the various language libraries once this lands here. Happy to flesh this out on this feature request or open new issues on the various repositories.

Enabling these histograms in TypeScript could look something like this.

Runtime.install({
  telemetryOptions: {
    metrics: {
      otel: {
        url: 'http://opentelemetry-collector:4318/v1/metrics',
        http: true,
        histogramAggregation: { type: 'both' },
      },
    },
  },
});

Additional context

I am happy to drive the implementation here and create the PRs. Our team loves Temporal and this would be a big observability win for us.

Please let me know if I can answer any questions or provide any more detail.

Thanks!

Related: #6633 asks for the same thing across server and SDKs.

[karasowles]

Hi Alex, nice to meet you! Thanks for this; I'll find the right folks to pull in here 👀

[Sushisource]

@mckalexee Thanks for the quality writeup!

This is something I think we'd be happy to accept. My main concern would be ensuring we keep backwards compatibility, which you've addressed. I'm slightly dubious about the Both functionality, but I can see the reason you want it. Would simply deploying a worker with Base2Exponential into a staging environment be sufficient? Maybe we can avoid needing Both in that case?

Lastly, it would be good to consider if this is something we'd need/want to port to Go/Java as well. They use different metrics libraries, so the situation may be a bit different there, but it's worth addressing. In any case, I would be fine starting with just Core.

[Sushisource]

Yes, agreed. I would be fine with _native. When you say _exp matches the "spec" more closely... which spec are you referring to? Otel? Some quick googling seems to show OTel doesn't really have a specific opinion, in which case _native to match Prom works for me. In fact, maybe we just want that option to be inside the Base2Exponential variant so they're obviously associated.

I'm good with this as a starting point. Happy to have you open a PR and continue discussion there.

Thanks!!

[mckalexee]

By "spec" I meant this writeup. We call the option Base2Exponential and have exponential_ as variable names. So there could be a case for having the suffix match. Putting it in the Base2Exponential variant works for me.

I'll start working on a PR and let you know when it ready for review and further discussion. Thanks again!

[mckalexee]

@Sushisource I have opened the PR. It's currently in draft since I want to make sure that CI passes first. Would you be the correct person to allow CI to run on this PR?

temporalio/sdk-rust#1654

[Sushisource]

Done! Thanks!

[mckalexee]

I've marked it as ready for review. Some integration test workflows timed out but from what I can see it's not related to the changes in the PR.

[Sushisource]

Done in temporalio/sdk-rust#1654

Thanks @mckalexee

[mckalexee]

Thanks for helping land this @Sushisource

For the Typescript SDK updates, what is the preferred process? Should I start a new discussion?

I can also wait for temporalio/sdk-typescript#2535 to land and then open a PR. Let me know what's preferred.

[Sushisource]

Don't think we need a new discussion. Yes, happy for you to open a PR after the core bump merges. Any other languages you'd be keen on contributing we'd be happy to take as well. Thanks!