Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds firehose http endpoint protocol integration for opentelemetry ingestion. #6037

Open
f41gh7 opened this issue Mar 28, 2024 · 4 comments
Open

Adds firehose http endpoint protocol integration for opentelemetry ingestion. #6037

f41gh7 opened this issue Mar 28, 2024 · 4 comments
Assignees
@AndrewChubatiuk
Labels
enhancement New feature or request

Comments

@f41gh7
Copy link
Contributor

f41gh7 commented Mar 28, 2024

Is your feature request related to a problem? Please describe

It's possible to forward AWS CloudWatch metrics to the VictoriaMetrics with Firehose http endpoint opentelemetry format. https://aws.amazon.com/blogs/aws/cloudwatch-metric-streams-send-aws-metrics-to-partners-and-to-your-apps-in-real-time/

But it has the following issues:

Describe the solution you'd like

  • Fix compatibility with firehose.
  • Introduce new flag for metric name conversion into prometheus compatible format.
  • properly parse nested maps for attributes.

Describe alternatives you've considered

No response

Additional information

No response
@f41gh7 f41gh7 added the enhancement New feature or request label Mar 28, 2024
@AndrewChubatiuk AndrewChubatiuk mentioned this issue Mar 28, 2024
Merged
@valyala
Copy link
Collaborator

valyala commented Apr 2, 2024

FYI, the compatibility with AWS Firehose should be fixed in the commit 509df44

valyala added a commit that referenced this issue Apr 2, 2024
@valyala
lib/protoparser/opentelemetry: follow-up after 47892b4
fb42380 
- Rename -opentelemetry.sanitizeMetrics command-line flag to more clear -opentelemetry.usePrometheusNaming
- Clarify the description of the change at docs/CHANGELOG.md
- Rename promrelabel.SanitizeLabelNameParts to more clear promrelabel.SplitMetricNameToTokens
- Properly split metric names at '_' char in promerlabel.SplitMetricNameToTokens.
- Add tests for various edge cases for Prometheus metric names' normalization
  according to the code at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/b8655058501bed61a06bb660869051491f46840b/pkg/translator/prometheus/normalize_name.go
- Extract the code responsible for Prometheus metric names' normalization into a separate file (santize.go)

Updates #6037
Updates #6035
@valyala
Copy link
Collaborator

valyala commented Apr 2, 2024

FYI, the -opentelemetry.usePrometheusNaming flag has been introduced in the commit fb42380 . If this flag is set, then VictoriaMetrics automatically converts metric names and label names into Prometheus-compatible names for samples ingested via OpenTelemetry format.

valyala added a commit that referenced this issue Apr 3, 2024
@valyala
lib/protoparser/opentelemetry: follow-up after 47892b4
bb9bb60 
- Rename -opentelemetry.sanitizeMetrics command-line flag to more clear -opentelemetry.usePrometheusNaming
- Clarify the description of the change at docs/CHANGELOG.md
- Rename promrelabel.SanitizeLabelNameParts to more clear promrelabel.SplitMetricNameToTokens
- Properly split metric names at '_' char in promerlabel.SplitMetricNameToTokens.
- Add tests for various edge cases for Prometheus metric names' normalization
  according to the code at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/b8655058501bed61a06bb660869051491f46840b/pkg/translator/prometheus/normalize_name.go
- Extract the code responsible for Prometheus metric names' normalization into a separate file (santize.go)

Updates #6037
Updates #6035
@valyala
Copy link
Collaborator

valyala commented Apr 4, 2024

FYI, vmagent and single-node VictoriaMetrics properly return responses to Amazon Firehose starting from v1.100.0. They also support the -opentelemetry.usePrometheusNaming command-line flag starting from v1.100.0 release.

@AndrewChubatiuk
Copy link
Contributor

@f41gh7
we're converting otel metrics in the same manner as it's done in collector, they are also storing array and map attributes as JSON

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AndrewChubatiuk AndrewChubatiuk

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@valyala @AndrewChubatiuk @f41gh7