Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an ability to select Kafka partition based on resource tags for metrics #31675

Closed
SHaaD94 opened this issue Mar 11, 2024 · 5 comments
Closed

Add an ability to select Kafka partition based on resource tags for metrics #31675

SHaaD94 opened this issue Mar 11, 2024 · 5 comments
Labels
enhancement New feature or request exporter/kafka Stale

Comments

@SHaaD94
Copy link
Contributor

SHaaD94 commented Mar 11, 2024

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

Our metric aggregation backend consumes Kafka partitions and needs each metrics from each server to land into the same Kafka partition.

Describe the solution you'd like

I would like to have a flag in kafka exporter, which would indicate the result kafka payload would have a partitioning key, calculated based on sorted resource tags.

Describe alternatives you've considered

Alternative to this would be to write an additional balancing middleware.

Additional context

I have already implemented the feature in the pull request #31315
Would really appreciate any feedback on it.

Thank you!
@SHaaD94 SHaaD94 added enhancement New feature or request needs triage New item requiring triage labels Mar 11, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@SHaaD94 SHaaD94 mentioned this issue Mar 11, 2024
Merged
@SHaaD94 SHaaD94 changed the title Add an ability to select Kafka partition based on resource tags Add an ability to select Kafka partition based on resource tags for metrics Mar 11, 2024
@github-actions github-actions bot mentioned this issue Mar 12, 2024
Closed
@crobert-1
Copy link
Member

Thanks for filing this issue, I agree that the exporter should be able to configure the key on exported telemetry. I have some thoughts on implementation but I can comment on the PR.

@crobert-1 crobert-1 removed the needs triage New item requiring triage label Mar 15, 2024
@crobert-1 crobert-1 mentioned this issue Mar 15, 2024
Open
dmitryax pushed a commit that referenced this issue Apr 29, 2024
@SHaaD94 @crobert-1
[exporter/kafka] Impelement partitioning for OTLP metrics (#31315)
cec2153 
**Description:** Add resource attributes based partitioning for OTLP
metrics

In our backend we really need an ability to distribute metrics based on
resource attributes.
For this I added additional flag to the configuration.
Some code from traces partitioning by traceId reused.

Judging by issues, this feature is anticipated by several more people.

**Link to tracking Issue:**
[31675](#31675)

Additionally this feature was menioned in these issues:
[29433](#29433),
[30666](#30666)


**Testing:**
Added tests for hashing utility.
Added tests for marshalling and asserting correct keys and the number of
messages.
Tested locally with host metrics and chained OTLP metrics receiver.

**Documentation:** 
Changelog entry
Flag is added to the doc of KafkaExporter

---------

Co-authored-by: Curtis Robert <crobert@splunk.com>
rimitchell pushed a commit to rimitchell/opentelemetry-collector-contrib that referenced this issue May 8, 2024
@SHaaD94 @crobert-1 @rimitchell
[exporter/kafka] Impelement partitioning for OTLP metrics (open-telem…
763c2d7 
…etry#31315)

**Description:** Add resource attributes based partitioning for OTLP
metrics

In our backend we really need an ability to distribute metrics based on
resource attributes.
For this I added additional flag to the configuration.
Some code from traces partitioning by traceId reused.

Judging by issues, this feature is anticipated by several more people.

**Link to tracking Issue:**
[31675](open-telemetry#31675)

Additionally this feature was menioned in these issues:
[29433](open-telemetry#29433),
[30666](open-telemetry#30666)


**Testing:**
Added tests for hashing utility.
Added tests for marshalling and asserting correct keys and the number of
messages.
Tested locally with host metrics and chained OTLP metrics receiver.

**Documentation:** 
Changelog entry
Flag is added to the doc of KafkaExporter

---------

Co-authored-by: Curtis Robert <crobert@splunk.com>
jlg-io pushed a commit to jlg-io/opentelemetry-collector-contrib that referenced this issue May 14, 2024
@SHaaD94 @crobert-1 @jlg-io
[exporter/kafka] Impelement partitioning for OTLP metrics (open-telem…
6371895 
…etry#31315)

**Description:** Add resource attributes based partitioning for OTLP
metrics

In our backend we really need an ability to distribute metrics based on
resource attributes.
For this I added additional flag to the configuration.
Some code from traces partitioning by traceId reused.

Judging by issues, this feature is anticipated by several more people.

**Link to tracking Issue:**
[31675](open-telemetry#31675)

Additionally this feature was menioned in these issues:
[29433](open-telemetry#29433),
[30666](open-telemetry#30666)


**Testing:**
Added tests for hashing utility.
Added tests for marshalling and asserting correct keys and the number of
messages.
Tested locally with host metrics and chained OTLP metrics receiver.

**Documentation:** 
Changelog entry
Flag is added to the doc of KafkaExporter

---------

Co-authored-by: Curtis Robert <crobert@splunk.com>
@github-actions GitHub Actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label May 15, 2024
@SHaaD94
Copy link
Contributor Author

SHaaD94 commented May 15, 2024

Closing, since the change was merged.

@SHaaD94 SHaaD94 closed this as completed May 15, 2024
@crobert-1
Copy link
Member

Fixed by #31315 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request exporter/kafka Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SHaaD94 @crobert-1