context deadline exceeded error when writing from influx to mimir using influx2cortex proxy

[dhanvsagar]

Unable to send data from influx to mimir using influx2cortex proxy (with the configs shows below).

Telegraf output config

[[outputs.http]]
  url = "http://influx2cortex-proxy:8080"
  data_format = "influx"
  timeout = "30s"
  method = "POST"
  interval = "300s"
  flush_interval = "150s"
  insecure_skip_verify = true
  [outputs.http.headers]
    X-Scope-OrgID = "influx-proxy"

Proxy deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: influx2cortex-proxy
  labels:
    app.kubernetes.io/name: proxy
spec:
  selector:
    matchLabels:
      app: influx2cortex-proxy
  template:
    metadata:
      labels:
        app: influx2cortex-proxy
        app.kubernetes.io/name: proxy
    spec:
      containers:
        - name: influx2cortex
          image: 'influx2cortex-proxy:8d12d69'
          args:
            - "-auth.enable=false"
            - "-write-endpoint=http://mimir-distributor-headless.mimir.svc.cluster.local:8080/api/v1/push"
            - "-skip-label-validation=true"
            - "-max.request.size.bytes=0"
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
            - name: http2
              containerPort: 8081
              protocol: TCP
          resources:
            limits:
              cpu: '2'
              memory: 8Gi
            requests:
              cpu: 200m
              memory: 512Mi

Proxy Service

apiVersion: v1
kind: Service
metadata:
  name: influx2cortex-proxy
  labels:
    app: influx2cortex-proxy
spec:
  ports:
    - name: http
      port: 8000
      protocol: TCP
      targetPort: http
    - name: http2
      port: 8081
      protocol: TCP
      targetPort: http2
  selector:
      app.kubernetes.io/name: proxy

Logs and errors

Inlux2cortex proxy image was built using the Dockerfile provided in the official repository using the latest codebase.

No logs are available at the proxy other than

level=info caller=server.go:107 msg="server listening on address" addr=[::]:8000
level=info caller=server.go:127 msg="GRPC server listening on address" addr=[::]:9095
level=info caller=service.go:75 msg="Starting internal http server" addr=0.0.0.0:8081
level=info caller=server.go:166 msg="Starting grpc server" addr=[::]:9095
level=info caller=server.go:155 msg="Starting http server" addr=[::]:8000

Telegraf shows the following error message

2022-10-10T13:19:14Z E! [agent] Error writing to outputs.http: Post "http://influx2cortex-proxy:8080": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-10-10T13:19:44Z D! [outputs.http] Buffer fullness: 100015 / 10000000 metrics
2022-10-10T13:19:44Z E! [agent] Error writing to outputs.http: Post "http://influx2cortex-proxy:8080": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

When I tried changing the port number in telegraf output config from 8080 to 8081, shows an error as shown below

2022-10-10T13:08:06Z D! [outputs.http] Buffer fullness: 138855 / 10000000 metrics
2022-10-10T13:08:06Z E! [agent] Error writing to outputs.http: when writing to [http://influx2cortex-proxy:8081] received status code: 404

Please correct me if my configs are not proper. Also is there any option to enable debug logs in the proxy..?

[alexgreenbank]

Hello, based on your config above you need to configure telegraf to write to port 8000 not 8080, e.g.

[[outputs.http]]
  url = "http://influx2cortex-proxy:8000"
...

Also, port 8081 is an internal server that serves the /metrics and /healthz data that can be scraped as a Prometheus target, it's not an endpoint to be written to.

[alexgreenbank]

Also you might want to add the following setting to your telegraf outputs.http file:-

  skip_database_creation = true

as this might be needed (depending on your version of telegraf) to get rid of a database "telegraf" creation failed: 500 Internal Server Error error.

[dhanvsagar]

Hello, based on your config above you need to configure telegraf to write to port 8000 not 8080, e.g.

[[outputs.http]]
  url = "http://influx2cortex-proxy:8000"
...

Also, port 8081 is an internal server that serves the /metrics and /healthz data that can be scraped as a Prometheus target, it's not an endpoint to be written to.

Thanks for the response. Sorry the port number was a typo, Corrected it.
I have tried with 8000 also and shows the error

2022-10-10T13:08:06Z D! [outputs.http] Buffer fullness: 138855 / 10000000 metrics
2022-10-10T13:08:06Z E! [agent] Error writing to outputs.http: when writing to [http://influx2cortex-proxy:8081] received status code: 404

[dhanvsagar]

Also you might want to add the following setting to your telegraf outputs.http file:-

  skip_database_creation = true

as this might be needed (depending on your version of telegraf) to get rid of a database "telegraf" creation failed: 500 Internal Server Error error.

Tried adding the flag skip_database_creation = true, but that's causing another error

2022-10-10T14:09:55Z E! [telegraf] Error running agent: Error loading config file /usr/local/etc/telegraf/conf/telegraf.d/influx_output.conf: plugin outputs.http: line 1: configuration specified the fields ["skip_database_creation"], but they weren't used

I guess that flag is not supported in the version that I use.
Also, Please note that I am using [[outputs.http]] and I hope that is supported by the proxy.

[alexgreenbank]

Aha, missed that first time round. You need the full URL ending with /api/v1/push/influx/write, e.g.

[[outputs.http]]
  url = "http://influx2cortex-proxy:8000/api/v1/push/influx/write"
...

I've never used [[outputs.http]] but it should work if it is sending the metrics in the correct format. I've only ever used [[outputs.influxdb]] e.g.

[[outputs.influxdb]]
  urls = ["https://localhost:8000/api/v1/push/influx/write"]
  data_format = "influx"
  skip_database_creation = true

[dhanvsagar]

Aha, missed that first time round. You need the full URL ending with /api/v1/push/influx/write, e.g.

[[outputs.http]]
  url = "http://influx2cortex-proxy:8000/api/v1/push/influx/write"
...

I've never used [[outputs.http]] but it should work if it is sending the metrics in the correct format. I've only ever used [[outputs.influxdb]] e.g.

[[outputs.influxdb]]
  urls = ["https://localhost:8000/api/v1/push/influx/write"]
  data_format = "influx"
  skip_database_creation = true

The updated URL is working. Thanks..!

[alexgreenbank]

Excellent, tomorrow I'll add a section to the README.md with some notes on configuring telegraf and test it with [[outputs.http]] too.

[alexgreenbank]

Docs update: #72 now merged.