InfluxDB sink: 500 Internal Server Error

[r0bj]

InfluxDB version: 1.0.2

HTTP traffic on heapster node and influxdb node:

2016-10-26 11:32:05 10.14.30.112    10.14.30.35 >   POST    dev-graph-p1:8086   /write?consistency=&db=k8s&precision=&rp=default    HTTP/1.1    -   -
2016-10-26 11:32:05 10.14.30.35 10.14.30.112    <   -   -   -   HTTP/1.1    500 Internal Server Error
2016-10-26 11:32:05 10.14.30.112    10.14.30.35 >   GET dev-graph-p1:8086   /ping   HTTP/1.1    -   -
2016-10-26 11:32:05 10.14.30.35 10.14.30.112    <   -   -   -   HTTP/1.1    204 No Content

InfluxDB response is 500 Internal Server Error.

InfluxDB logs:

2016-10-26_11:35:05.03873 [httpd] 10.14.30.112 - root [26/Oct/2016:11:35:05 +0000] "POST /write?consistency=&db=k8s&precision=&rp=default HTTP/1.1" 500 72 "-" "heapster/1.2.0" 3d69d9ff-9b70-11e6-bc17-000000000000 1845

In the same time heapster logs:

I1026 11:32:05.000254       1 manager.go:79] Scraping metrics start: 2016-10-26 11:31:00 +0000 UTC, end: 2016-10-26 11:32:00 +0000 UTC
I1026 11:32:05.006473       1 manager.go:98] Querying source: kubelet:10.14.30.112:10255
I1026 11:32:05.011713       1 kubelet.go:232] successfully obtained stats for 8 containers
I1026 11:32:05.013481       1 manager.go:98] Querying source: kubelet:10.14.30.110:10255
I1026 11:32:05.015631       1 manager.go:98] Querying source: kubelet:10.14.30.111:10255
I1026 11:32:05.016861       1 kubelet.go:232] successfully obtained stats for 7 containers
I1026 11:32:05.022243       1 kubelet.go:232] successfully obtained stats for 13 containers
I1026 11:32:05.022365       1 manager.go:152] ScrapeMetrics: time: 22.016775ms size: 28
I1026 11:32:05.022388       1 manager.go:154]    scrape  bucket 0: 3
I1026 11:32:05.022394       1 manager.go:154]    scrape  bucket 1: 0
I1026 11:32:05.022398       1 manager.go:154]    scrape  bucket 2: 0
I1026 11:32:05.022400       1 manager.go:154]    scrape  bucket 3: 0
I1026 11:32:05.022403       1 manager.go:154]    scrape  bucket 4: 0
I1026 11:32:05.022407       1 manager.go:154]    scrape  bucket 5: 0
I1026 11:32:05.022411       1 manager.go:154]    scrape  bucket 6: 0
I1026 11:32:05.022414       1 manager.go:154]    scrape  bucket 7: 0
I1026 11:32:05.022417       1 manager.go:154]    scrape  bucket 8: 0
I1026 11:32:05.022420       1 manager.go:154]    scrape  bucket 9: 0
I1026 11:32:05.022422       1 manager.go:154]    scrape  bucket 10: 0
I1026 11:32:05.022910       1 manager.go:113] Pushing data to: Metric Sink
I1026 11:32:05.022925       1 manager.go:116] Data push completed: Metric Sink
I1026 11:32:05.022975       1 manager.go:113] Pushing data to: InfluxDB Sink
I1026 11:32:05.022992       1 manager.go:116] Data push completed: InfluxDB Sink
I1026 11:32:05.032602       1 influxdb.go:169] Exported 524 data to influxDB in 9.180614ms

Data isn't stored in influxdb.

heapster deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      containers:
      - name: heapster
        image: kubernetes/heapster:canary
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=influxdb:http://dev-graph-p1:8086
        - --vmodule=*=4

[gene-telligent]

This is directly related to #1143 .

The issue is related to the influxDB default retention policy being changed from "default" to "autogen".

You can fix this by running (in the influxDB instance)

CREATE RETENTION POLICY "default" ON "k8s" DURATION INF REPLICATION 1 DEFAULT

Which creates that old "default" retention policy.

[derekhound]

I met the same problem, but the above comment really resolve my problem. It works.

[piosz]

Currently we only support InfluxDB in version 0.12. The plan is to migrate to 1.0 for Kubernetes 1.5.

Closing in favor of #1143