Skip to content

feat(go.d): chart histogram buckets as range heatmaps - #23013

Merged
ilyam8 merged 3 commits into
netdata:masterfrom
ilyam8:histogram-range-heatmaps
Jul 6, 2026
Merged

feat(go.d): chart histogram buckets as range heatmaps#23013
ilyam8 merged 3 commits into
netdata:masterfrom
ilyam8:histogram-range-heatmaps

Conversation

@ilyam8

@ilyam8 ilyam8 commented Jul 6, 2026

Copy link
Copy Markdown
Member
Summary
Test Plan
Additional Information
For users: How does this change affect me?

Summary by cubic

Histogram bucket metrics are flattened into non-overlapping range totals and rendered as heatmaps by chartengine (autogen and templates), with buckets ordered numerically and +Inf last. This replaces cumulative buckets and removes collector-local workarounds.

  • New Features

    • metrix.ReadFlatten() emits range bucket totals; +Inf is count - last finite bucket. Typed Histogram() reads remain cumulative.
    • chartengine forces _bucket series to heatmap and keeps algorithm: incremental; le remains the upper-bound label. Planner orders bucket dimensions numerically with +Inf last, applies this in inference and lifecycle, and sorts histogram dims after other dynamic dims when mixed.
    • Autogen emits heatmaps; planner enforces heatmap for bucket charts; docs and tests updated across metrix, chartengine, and go.d/prometheus.
  • Migration

    • No config changes. Histogram bucket charts display as heatmaps even if templates specify another type.
    • Remove any collector-local cumulative-to-range logic. If dashboards expected cumulative bucket values, update to per-range totals.
    • Histograms without finite bounds flatten all observations into the +Inf range bucket.

Written for commit 98d06e2. Summary will update on new commits.

Review in cubic

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR changes Go metrix histogram bucket flattening from cumulative buckets to non-overlapping per-range bucket totals, and updates chartengine (both autogen and template-driven paths) to render histogram bucket charts as heatmaps.

Changes:

  • Update metrix.ReadFlatten() histogram bucket emission to output per-range bucket totals (including +Inf = count - last_finite_bucket).
  • Force histogram _bucket charts to heatmap in chartengine (planner + autogen) even if templates specify a different chart type.
  • Refresh tests and documentation to reflect range-bucket semantics and heatmap rendering.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/go/plugin/go.d/collector/prometheus/writer_test.go Updates Prometheus writer expectations for range buckets and explicit +Inf.
src/go/plugin/go.d/collector/prometheus/testdata/golden/histogram.json Adjusts golden output for new +Inf value and heatmap type.
src/go/plugin/go.d/collector/prometheus/collector_test.go Updates histogram assertions to match range-bucket +Inf semantics.
src/go/plugin/framework/charttpl/README.md Documents forced heatmap type for histogram bucket charts and non-cumulative bucket values.
src/go/plugin/framework/chartengine/README.md Documents heatmap rendering and range-bucket semantics for histogram buckets.
src/go/plugin/framework/chartengine/planner.go Forces histogram bucket charts to heatmap during plan construction (template + autogen).
src/go/plugin/framework/chartengine/planner_test.go Extends planner tests to assert heatmap type for histogram bucket charts.
src/go/plugin/framework/chartengine/autogen.go Switches histogram bucket autogen chart type to heatmap.
src/go/plugin/framework/chartengine/autogen_test.go Adds assertion that histogram bucket autogen route uses heatmap chart type.
src/go/pkg/metrix/runtime_store_test.go Updates runtime histogram flatten expectations for range buckets.
src/go/pkg/metrix/README.md Documents range-bucket flattening and +Inf derivation; clarifies typed Histogram reads remain cumulative.
src/go/pkg/metrix/reader.go Implements range-bucket flattening by differencing cumulative buckets and deriving +Inf.
src/go/pkg/metrix/host_scope_test.go Updates scoped histogram flatten expectation for range buckets.
src/go/pkg/metrix/histogram_store_test.go Updates existing histogram tests and adds coverage for “no finite bounds” => all count in +Inf.
.agents/skills/project-writing-go-modules-framework-v2/SKILL.md Updates contributor guidance: avoid collector-local bucket workarounds; rely on heatmap + range buckets.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/go/plugin/framework/chartengine/planner.go Outdated
Comment thread src/go/plugin/framework/chartengine/planner.go

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 15 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Prom as Prometheus Collector
    participant Store as CollectorStore (metrix)
    participant Reader as ReadFlatten()
    participant Engine as chartengine Planner
    participant Template as charttpl Template
    participant Dashboard

    Note over Prom,Dashboard: Histogram Bucket Flow – Range Heatmap

    Prom->>Store: Observe histogram points (cumulative buckets + count + sum)
    Store->>Store: Store cumulative bucket cumulatives per commit cycle

    alt When ReadFlatten() is called
        Store->>Reader: ReadFlatten()
        Reader->>Reader: Compute non-overlapping range bucket totals
        Note over Reader: For each finite bound le[i]:<br/>bucketValue = cumulative[i] - cumulative[i-1]<br/>(first bucket is <= le[0])
        Note over Reader: For +Inf bound:<br/>bucketValue = count - last finite cumulative
        Reader-->>Prom: Return flattened series with range bucket values + count + sum
    end

    Note over Prom,Engine: Planner consumes flattened series

    Prom->>Engine: Feed flattened read snapshot
    Engine->>Engine: scanPlanSeries() iterates all series
    Engine->>Engine: isHistogramBucketSeries() checks meta.SourceKind==Histogram and FlattenRole==HistogramBucket
    alt Histogram bucket series detected
        Engine->>Engine: forceHistogramBucketChartType() -> sets chartType to heatmap
    end

    Engine->>Engine: accumulateRoute() applies chart meta overrides
    alt Series is histogram bucket
        Engine->>Engine: Enforce cs.meta.Type = ChartTypeHeatmap
    end
    Engine-->>Prom: Plan with heatmap charts

    Note over Prom,Template: Template-driven path (charttpl)

    Template->>Engine: Template defines type: stacked or omitted
    Engine->>Engine: Template type overridden to heatmap for _bucket series
    Engine-->>Dashboard: Emit heatmap chart with incremental-algo bucket dimensions

    Note over Dashboard: Heatmap rendering – le label is upper bound dimension key
Loading

Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 8 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/go/plugin/framework/chartengine/planner_lifecycle.go
@ilyam8
ilyam8 marked this pull request as ready for review July 6, 2026 20:38
@ilyam8
ilyam8 enabled auto-merge (squash) July 6, 2026 20:40
@sonarqubecloud

sonarqubecloud Bot commented Jul 6, 2026

Copy link
Copy Markdown

@ilyam8
ilyam8 merged commit 4e60d57 into netdata:master Jul 6, 2026
149 of 155 checks passed
@ilyam8
ilyam8 deleted the histogram-range-heatmaps branch July 6, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants