Skip to content

Commit

Permalink
[exporter/prometheusremotewrite] Hash labels using xxhash for perform…
Browse files Browse the repository at this point in the history
…ance (#31385)

**Description:**
Improve performance in pkg/translator/prometheusremotewrite by using the
same [xxhash](https://github.com/cespare/xxhash) based time series
signature algorithm as Prometheus itself
([`labels.StableHash()`](https://github.com/prometheus/prometheus/blob/c6c8f63516741fd38ece5387254afee798cfc8d7/model/labels/sharding.go#L24),
which is guaranteed to not change over time). I became aware of time
series signature calculation being a bit of a bottleneck when profiling
Grafana Mimir's OTLP endpoint.

This change involves moving from a string hash, to a `uint64` one. My
thesis is that if Prometheus uses this algorithm to identify time
series/label sets, it should be suitable for this translation logic too.

#### Hash collisions

I've attempted to handle hash collisions in the same way as Prometheus:
`PrometheusConverter` has a `unique` field, which is its main `map` from
hash to `TimeSeries`, as well as a `conflicts` field, being its
secondary `map` from hash to a slice of `TimeSeries` in case of hash
collisions. If a label set should hash to an existing entry in `unique`,
but not be equal to the existing entry's label set, the label set is
attempted matched to the `conflicts` `map` instead. If its equal is not
found among the conflicts for the hash in question either, it's added to
the conflicts slice (for the hash).

**Testing:**
I've run `make test`/`make lint` and run the `BenchmarkFromMetrics`
benchmark. Benchmark stats included below, they show an average speedup
of 3.68% and an average memory reduction of 17.13%.

**NB:** The benchmark stats reveal performance regressions in a few
cases, because of using the `prometheusConverter` API _via_ the
`FromMetrics` function.

---------

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>
  • Loading branch information
aknuds1 and Aneurysm9 committed Apr 12, 2024
1 parent d8a2c4f commit 247a9f9
Show file tree
Hide file tree
Showing 12 changed files with 887 additions and 694 deletions.
27 changes: 27 additions & 0 deletions .chloggen/arve_xxhash.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: prometheusremotewrite

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Optimize the prometheusremotewrite.FromMetrics function, based around more performant metric identifier hashing.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [31385]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [api]
1 change: 1 addition & 0 deletions pkg/translator/prometheusremotewrite/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ module github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/
go 1.21

require (
github.com/cespare/xxhash/v2 v2.3.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.98.0
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.98.0
github.com/prometheus/common v0.52.2
Expand Down
2 changes: 2 additions & 0 deletions pkg/translator/prometheusremotewrite/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 247a9f9

Please sign in to comment.