transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF

[sw-samuraj]

Bug Report

What did you do?
Following Getting started documentation, I start Prometheus via ./prometheus --config.file=prometheus.yml command. The prometheus.yml config file is the default one in the distribution, without any modification.

What did you expect to see?
An initial log sequence with "ready to serve" message.

What did you see instead? Under which circumstances?
After the initial log sequence, there is endlessly repeating sequence of:

2018/06/12 14:04:44 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:04:44 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:04:44 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.

Environment

• System information:

$ uname -srm
Linux 4.15.0-22-generic x86_64

• Prometheus version:

$ prometheus --version
prometheus, version 2.3.0 (branch: HEAD, revision: 290d71791a507a5057b9a099c9d48703d86dc941)
  build user:       root@d539e167976a
  build date:       20180607-08:46:54
  go version:       go1.10.2

• Prometheus configuration file:

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

rule_files:

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
    - targets: ['localhost:9090']

• Logs:

level=info ts=2018-06-12T12:15:03.306533833Z caller=main.go:222 msg="Starting Prometheus" version="(version=2.3.0, branch=HEAD, revision=290d71791a507a5057b9a099c9d48703d86dc941)"
level=info ts=2018-06-12T12:15:03.306576156Z caller=main.go:223 build_context="(go=go1.10.2, user=root@d539e167976a, date=20180607-08:46:54)"
level=info ts=2018-06-12T12:15:03.306594198Z caller=main.go:224 host_details="(Linux 4.15.0-22-generic #24-Ubuntu SMP Wed May 16 12:15:17 UTC 2018 x86_64 sandman (none))"
level=info ts=2018-06-12T12:15:03.306610209Z caller=main.go:225 fd_limits="(soft=65535, hard=65535)"
level=info ts=2018-06-12T12:15:03.307704369Z caller=web.go:426 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-12T12:15:03.307702157Z caller=main.go:514 msg="Starting TSDB ..."
level=info ts=2018-06-12T12:15:03.313006036Z caller=main.go:524 msg="TSDB started"
level=info ts=2018-06-12T12:15:03.313043081Z caller=main.go:603 msg="Loading configuration file" filename=prometheus.yml
level=info ts=2018-06-12T12:15:03.313604856Z caller=main.go:500 msg="Server is ready to receive web requests."
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:03 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:04 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:04 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:04 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2018/06/12 14:15:04 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
^Clevel=warn ts=2018-06-12T12:15:04.375670642Z caller=main.go:377 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2018-06-12T12:15:04.375805748Z caller=main.go:402 msg="Stopping scrape discovery manager..."
level=info ts=2018-06-12T12:15:04.375869108Z caller=main.go:416 msg="Stopping notify discovery manager..."
level=info ts=2018-06-12T12:15:04.375916767Z caller=main.go:438 msg="Stopping scrape manager..."
level=info ts=2018-06-12T12:15:04.375900041Z caller=main.go:398 msg="Scrape discovery manager stopped"
level=info ts=2018-06-12T12:15:04.375991379Z caller=main.go:412 msg="Notify discovery manager stopped"
level=info ts=2018-06-12T12:15:04.376092166Z caller=main.go:432 msg="Scrape manager stopped"
level=info ts=2018-06-12T12:15:04.376181841Z caller=manager.go:464 component="rule manager" msg="Stopping rule manager..."
level=info ts=2018-06-12T12:15:04.376242794Z caller=manager.go:470 component="rule manager" msg="Rule manager stopped"
level=info ts=2018-06-12T12:15:04.376285546Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."
level=info ts=2018-06-12T12:15:04.37634038Z caller=main.go:588 msg="Notifier manager stopped"
level=info ts=2018-06-12T12:15:04.376445088Z caller=main.go:599 msg="See you next time!"

[brian-brazil]

This is likely the rpc gateway. Do you have proxy environment variables set?

[sw-samuraj]

Yes, I had following proxy set up:

export http_proxy="my.proxy.com:8080"
export https_proxy="${http_proxy}"
export no_proxy="127.0.0.1, localhost"

The error disappears after adding 0.0.0.0 to the no_proxy variable:

export no_proxy="127.0.0.1, localhost, 0.0.0.0"

Thanx for providing a clue.

[BehroozBahrameh]

I have the same issue on version 2.6.0.
Everything works correctly when running on 2.5.0 but I get unexpected end of data, got "EOF" exception with the same config on ver2.6.0.

scrape_configs:
  - job_name: 'my_service'
    metrics_path: '/health/p'
    scheme: http
    static_configs:
      - targets: ['172.16.10.97:7015']

I have no idea how can i fix this issue on 2.6.0

level=warn ts=2018-12-24T10:37:20.387481536Z caller=scrape.go:835 component="scrape manager" scrape_pool=my_service target=http://172.16.10.97:7015/health/p msg="append failed" err="unexpected end of data, got \"EOF\""

[dsg-anthony]

I have the same issue on version 2.6.0.
Everything works correctly when running on 2.5.0 but I get unexpected end of data, got "EOF" exception with the same config on ver2.6.0.

scrape_configs:
  - job_name: 'my_service'
    metrics_path: '/health/p'
    scheme: http
    static_configs:
      - targets: ['172.16.10.97:7015']

I have no idea how can i fix this issue on 2.6.0

level=warn ts=2018-12-24T10:37:20.387481536Z caller=scrape.go:835 component="scrape manager" scrape_pool=my_service target=http://172.16.10.97:7015/health/p msg="append failed" err="unexpected end of data, got \"EOF\""

I ran into this issue with a exporter I created in python.
The fix was to update the Prometheus pip package from 4.0 to 4.2