Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Send logs from otel-collector to victorialogs #4839

Open
3 tasks
yantingqiu opened this issue Aug 15, 2023 · 1 comment · May be fixed by #6218
Open
3 tasks

Send logs from otel-collector to victorialogs #4839

yantingqiu opened this issue Aug 15, 2023 · 1 comment · May be fixed by #6218
Assignees
@valyala
Labels
question The question issue

Comments

@yantingqiu
Copy link

yantingqiu commented Aug 15, 2023 •
edited

Is your question request related to a specific component?

When I try to send logs to victorialogs using the loki exporter in otel-collector, I encounter the following error. What could be the cause of this?

Opentelemetry-collector logs:

error   exporterhelper/queued_retry.go:401      Exporting failed. The error is not retryable. Dropping data.   
       {"kind": "exporter", "data_type": "logs", "name": "loki", "error": 
       "Permanent error: HTTP 400 \"Bad Request\": remoteAddr: \"10.244.1.81:50446\"; 
       requestURI: /insert/loki/api/v1/push; unsupported path requested: \"/insert/loki/api/v1/push\"", "dropped_items": 80}

victorialogs logs:

 {"ts":"2023-08-15T06:48:45.376Z","level":"warn","caller":"VictoriaMetrics/lib/httpserver/httpserver.go:348","msg":"remoteAddr: 
       \"10.244.1.81:51858\"; requestURI: /insert/loki/api/v1/push; unsupported path requested: \"/insert/loki/api/v1/push\""}

Describe the question in detail

Loki JSON API example test

curl -H "Content-Type: application/json" -XPOST "http://10.1.133.153:9428/insert/loki/api/v1/push?_stream_fields=instance,job" --data-raw \
  '{"streams": [{ "stream": { "instance": "host123", "job": "app42" }, "values": [ [ "0", "foo fizzbuzz bar" ] ] }]}'

error:  
unsupported path requested: "/insert/loki/api/v1/push"

otel-collector configmap

apiVersion: v1
data:
  relay: |
    exporters:
      prometheus:
        endpoint: 0.0.0.0:9464
      clickhouse:
        endpoint: tcp://10.1.242.16:9000
        username: admin
        password: 27ff0399-0d3a-4bd8-919d-17c2181e6fb9
        database: otel
        ttl_days: 3
        logs_table_name: otel_logs
        traces_table_name: otel_traces
        metrics_table_name: otel_metrics
        timeout: 5s
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
          max_elapsed_time: 300s
      prometheusremotewrite:
        endpoint: http://10.1.190.177:8428/api/v1/write
      loki:
        endpoint: "http://10.1.133.153:9428/insert/loki/api/v1/push"
    extensions:
      health_check: {}
      memory_ballast:
        size_in_percentage: 40
    processors:
      batch:
        timeout: 5s
        send_batch_size: 100000
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: ${MY_POD_IP}:4317
          http:
            cors:
              allowed_origins:
              - http://*
              - https://*
            endpoint: ${MY_POD_IP}:4318
      prometheus:
        config:
          scrape_configs:
            - job_name: "kubernetes-metricsresource"
              # Default to scraping over https. If required, just disable this or change to
              # `http`.

Troubleshooting docs
@yantingqiu yantingqiu added the question The question issue label Aug 15, 2023
@steverweber
Copy link

steverweber commented Apr 15, 2024 •
edited

loki as of version 3 added support for exporter otlphttp perhaps their implementation might inspire some ideas https://grafana.com/docs/loki/latest/send-data/otel/ for a otlp endpoint in victorialogs.

that said.. endpoint: http://0.0.0.0:8427/insert/loki/api/v1/push is working for me with the otel loki exporter what jumps out to me is the quotas " perhaps they are unicode crap seeing that the error shows \" .. perhaps delete them.

@AndrewChubatiuk AndrewChubatiuk linked a pull request May 2, 2024 that will close this issue
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@valyala valyala

Labels
question The question issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vlinsert: added opentelemetry logs support VictoriaMetrics/VictoriaMetrics
3 participants
@valyala @steverweber @yantingqiu