HTTP Error Bad Gateway when using prometheus

[aximo]

Read before posting:
Please include this information:

What Grafana version are you using?

5.4.2

What datasource are you using?

prometheus latest docke image

What OS are you running grafana on?

Linux 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

I create a prometheus datasource in grafana web ui

What was the expected result?

success create datasource

What happened instead?

Error with 'HTTP Error Bad Gateway'

I run grafana and prometheus in docker with almost default configuration. after both started, i check connection in grafana:

root@ia-test-middleware:/app/grafana# docker exec -it grafana bash
root@6ffdc3f7ef50:/usr/share/grafana# curl http://192.168.1.165:9090
<a href="/graph">Found</a>.

root@6ffdc3f7ef50:/usr/share/grafana#

i think it means that prometheus endpoint is ready for grafana backend.

then I creete data source in grafana ui like this:

so I think is a bug, or I miss some configuration?

[aximo]

sorry for my mistake

[ZhiruiFeng]

@aximo Hi, did you find any solution for this problem?

[mimarpe]

I also suffer the same issue, let me know if you found a solution, thanks!

[torkelo]

it's networking / connection issue. Since it's not a problem in Grafana we close it.

[SamD]

I have the same problem and if I curl the url from the Grafana back-end (pod) it works perfectly so it isn't networking related

edit

I'm new to Grafana & Prometheus my url is the /metrics uri on the service which works fine e.g.

$ curl http://nginx-ingress-controller-metrics.svc.ingressns:9913/metrics

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.8905e-05
go_gc_duration_seconds{quantile="0.25"} 2.7022e-05
...
# HELP nginx_ingress_controller_config_hash Running configuration hash actually running
# TYPE nginx_ingress_controller_config_hash gauge
# HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful
# TYPE nginx_ingress_controller_config_last_reload_successful gauge
# HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge
...
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 8
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

I have done the ingress controller deployment from the helmchart which creates a custom metrics service that exposes 9913 on the ingress controller pod

[AndrewWPhillips]

I had the same problem and found that changing access from Server to Browser fixes it. It appears that the grafana server if being prevented from accessing the address:port while the browser is not blocked.

@torkelo I guess you could argue this is a network issue but you could also consider it a Grafana config issue. I prefer to use the Server option for better security.

[souvikhaldar]

I too was facing it. I've solved it but the solution is a bit hilarious. The solution is-
In the URL field, you need to explicitly write the URL (in my case it was http://localhost:9090). Earlier I was not writing it and there already was the url as watermark!

[glitch-k8s]

@souvikhaldar you are absolutely right. Its a water mark and not actual content. Thanks.

[jimmycgz]

I still face this error after explicitly writing the URL (in my case it was http://localhost:9090), finally I resolved it by using http://prometheus:9090 or http:/172.17.0.2:9090 because localhost is not regonized as server host when I'm running both Prometheus and Grafana as containers and you should get the correct IP of prometheus container for grafana to connect to, below is the command line I used:

# Create a config file and spin up prometheus container
cat prometheus.yml
docker run -d --name prometheus -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

# link the prometheus container to grafana
docker run -d --name grafana --link prometheus -p 3000:3000 grafana/grafana:6.2.5

# Get the IP of prometheus 
docker inspect prometheus

[DaCao]

@jimmycgz Thank you for the answer. It now works.

[JoanGil]

Maybe it's too late to answer but I faced the same problem and I solved it by using: host.docker.internal instead of localhost.

[rafael-queiroz-cko]

host.internal.docker

just a note, you typed host.internal.docker but it is actually host.docker.internal

[JoanGil]

Changed! Thanks!

[tunkul]

I had the same issue being completely new to grafana, just following the tutorial. Changing from "Server" to "Browser" did the trick.

[sysadmin-exe]

Had same issue. First time on grafana. Solved issue by using URL with IP address of Prometheus Docker Pod (docker inspect ) like so url: http://172.17.0.2:9090
This worked instantly.

[manero6]

I mapped port 8010 to 9090 when running Prometheus (-p 8010:9090).
It finds itself on the default network just as the Grafana container does.

I'm able to connect to it either with:
http://172.17.0.4:9090 (connects to the container directly)
or
http://172.17.0.1:8010 (connects to the mapped port)
or
http://server-ip:8010 (connects to the mapped port)

[amitad16]

This solution worked for me

First check if port 9090 is working

sudo lsof -i:9090

then check the promethues logs

sudo docker-compose logs promethues

it showed me error global_scrape timeout greater than scrape_interval

In their docs https://bigbluebutton-exporter.greenstatic.dev/installation/all_in_one_monitoring_stack/#2-copy-configuration-files

they told to increase the scrape_timeout, but nothing about the scrape_interval in /root/bbb-monitoring/prometheus.yaml
So, I increased them to the same value

global:
  scrape_interval: 30s
  evaluation_interval: 30s
  scrape_timeout: 30s

then
cd ~/bbb-monitoring

docker-compose down

sudo docker-compose up -d

then to test

curl http://127.0.0.1:9090

this should return

<a href="/graph">Found</a>.

then I set the Data Source -> Promethues url to
http://localhost:9090

Hope this helps someone.

[neo7337]

On running this on local minikube cluster, use the cluster address provided in logs to connect to prometheus server from grafana as the Access is Server(default) and Host clearly states : Your access method is Server, this means the URL needs to be accessible from the grafana backend/server.

Using : http://prometheus-server.monitoring.svc.cluster.local:80
worked!

[lorezi]

I had the same problem and found that changing access from Server to Browser fixes it. It appears that the grafana server if being prevented from accessing the address:port while the browser is not blocked.

@torkelo I guess you could argue this is a network issue but you could also consider it a Grafana config issue. I prefer to use the Server option for better security.

@AndrewWPhillips changing the access to Browser worked for me. Thanks

[xsqian]

Had same issue. First time on grafana. Solved issue by using URL with IP address of Prometheus Docker Pod (docker inspect ) like so url: http://172.17.0.2:9090
This worked instantly.

also works on your host IP like this HTTP://your_host_ip:9090, note that it's your host IP, not "localhost", not "127.0.0.1"

[RMartires]

mats@agent-m:~$ kubectl get svc -n monitor
NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                  ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   82m
grafana                                ClusterIP   172.20.53.202    <none>        3000/TCP                     62m
monitor-kube-prometheus-alertmanager   ClusterIP   172.20.151.69    <none>        9093/TCP                     82m
monitor-kube-prometheus-operator       ClusterIP   172.20.239.6     <none>        8080/TCP                     82m
monitor-kube-prometheus-prometheus     ClusterIP   172.20.207.29    <none>        9090/TCP                     82m
monitor-kube-state-metrics             ClusterIP   172.20.87.134    <none>        8080/TCP                     82m
monitor-node-exporter                  ClusterIP   172.20.210.234   <none>        9100/TCP                     82m
prometheus-operated                    ClusterIP   None             <none>        9090/TCP                     82m

use the cluster ip's so http://172.20.207.29:9090

this is if grafana and prometeus are on kubernetes pods

[ahmetonurslmz]

host.docker.internal

exact solution for this and all related connection problems between docker images. Connection to prometheus to local app and from grafana to prometheus.
All of these issues is able to solved by using host.internal.docker instead of localhost or local ip

[cute-angelia]

i think it's problem http Access : select Browser not Server

[vuhuythanh1995]

I had the same issue being completely new to grafana, just following the tutorial. Changing from "Server" to "Browser" did the trick.

I did that but nothing shows up in dashboarh.

[Sulmon]

i had the same issue . In my case i think it happened because i was running graphana and prometheus in the same terminal so one would automatoically shutdown when the other will start. So obviouly graphana for example couldn't connect with prometheus because it was off. When i open two different terminals and launched each application seperately it worked

[stolsvik]

The tip from @souvikhaldar was spot on. 🤦. That is just an absurd hurdle for new users of Grafana to have to stumble across!
This github issue is Google's first hit for "grafana bad gateway prometheus".

Please, @torkelo, if you could just put something other there, like "{url}" or something, or actually let the help text there (http://localhost:9090/) be actual text instead, I guess you would help a lot of first time users like myself. @souvikhaldar's comment currently has 47 👍's, and that are only the ones that admit to this being the problem!

[stolsvik]

Wrt. the docker containers - if you run both Prometheus and Grafana with --network="host" (and then dropping the -p arguments for publishing the ports), it works with http://localhost:9090/.

[rthamrin]

i think it's problem http Access : select Browser not Server

this is saving my life :D

[kartikdc]

I was facing this problem because I was trying to access localhost:9090 while Grafana was running in a docker container in a bridge network. :)

[Rodert]

在显式写入 URL 后我仍然面临这个错误（在我的例子中是http://localhost:9090），最后我通过使用http://prometheus:9090或 http://172.17.0.2:9090 解决了它，因为 localhost 是当我将 Prometheus 和 Grafana 作为容器运行时，未将其重新注册为服务器主机，您应该获得 prometheus 容器的正确 IP 以供 grafana 连接，以下是我使用的命令行：

# Create a config file and spin up prometheus container
cat prometheus.yml
docker run -d --name prometheus -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

# link the prometheus container to grafana
docker run -d --name grafana --link prometheus -p 3000:3000 grafana/grafana:6.2.5

# Get the IP of prometheus 
docker inspect prometheus

this is right idea.

you are good ! i am debug in local my computer, use docker-compose , it create multi container. so i use my ip 192.168.1.35, it intranet host.
it: http://localhost:9090 -->> http://192.168.1.35:9090. now, i made it.