Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Derive series from pprof sample labels #2387

Merged
merged 12 commits into from
Sep 25, 2023
Merged

Conversation

kolesnikovae
Copy link
Collaborator

@kolesnikovae kolesnikovae commented Sep 11, 2023

Currently, distributor does not preserve sample labels at normalisation. For example, samples

stack trace A, {foo=bar,baz=quz}, 1
stack trace A, {foo=fred,baz=zoo}, 1

cut to:

stack trace A, {foo=bar,baz=quz}, 2

With this change, labeled pprof samples are exported as indpendent series. Thus, if a profile with external label set

{service_name="my-service", instance="x"}

contains samples labeled with span_name="A" and span_name="B", distributor will extract them and send to ingesters in accordance to the newly created series labels:

{service_name="my-service", instance="x"} // Contains no samples with labels
{service_name="my-service", instance="x", span_name="A"} // Only contains samples with the span_name="A" label
{service_name="my-service", instance="x", span_name="B"} // Only contains samples with the span_name="B" label

In practice this allows for more granular queries. For example, profile of an HTTP/gRPC endpoint:

Screen.Recording.2023-09-21.at.18.10.25.mov

@kolesnikovae kolesnikovae marked this pull request as ready for review September 11, 2023 08:40
@kolesnikovae kolesnikovae requested a review from a team as a code owner September 11, 2023 08:40
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kolesnikovae kolesnikovae marked this pull request as draft September 12, 2023 14:04
@kolesnikovae kolesnikovae changed the title Preserve sample labels at normalisation WIP: Derive series from pprof sample labels Sep 12, 2023
@kolesnikovae kolesnikovae marked this pull request as ready for review September 21, 2023 10:15
@kolesnikovae kolesnikovae changed the title WIP: Derive series from pprof sample labels Derive series from pprof sample labels Sep 21, 2023
@korniltsev
Copy link
Collaborator

nit: maybe add go.cpu.labels.pprof to pkg/test/integration/ingest_pprof_test.go

@kolesnikovae
Copy link
Collaborator Author

Should also fix #2429

@kolesnikovae kolesnikovae merged commit 15e9b77 into main Sep 25, 2023
16 checks passed
@kolesnikovae kolesnikovae deleted the chore/preserve-sample-labels branch September 25, 2023 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants