Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The status of federation target is DOWN #3609

Closed
ahrtr opened this Issue Dec 22, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@ahrtr
Copy link

ahrtr commented Dec 22, 2017

I setup a prometheus instance successfully, and I can get all the metrics as expected using the following two links,
http://10.136.4.67:8080/metrics
http://10.136.4.67:8080/federate?match[]={endpoint=%22http-metrics%22}

Afterwards, I setup another prometheus instance, which tried to scrape metrics from the above prometheus instance, the configuration file is as below,

global:
  scrape_interval:     15s 

scrape_configs:
  - job_name: 'federate'
    scrape_interval: 30s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{endpoint="http-metrics"}'

    static_configs:
      - targets:
        - '10.136.4.67:8080'

The second prometheus instance was successfully started using the following command,
docker run -d --env http_proxy=http://www-proxy.mycompany.com:80 --env https_proxy=http://www-proxy.mycompany.com:80 -p 9090:9090 -v pwd/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:v2.0.0

I could open the UI of the second prometheus instance, but I did not see any metrics which were available on the target instance, and the status of the target prometheus instance displayed on the targets page (as below) was DOWN.
http://<IP_ADDRESS_OF_SECOND_PROMETHEUS>:9090/targets

I am pretty sure that:
1. The first prometheus instance is always UP;
2. The two servers on which the two prometheus instances were deployed are always connectable.

What did I miss?

Both prometheus instances were deployed on Oracle Linux Server release 7.3.

@ahrtr

This comment has been minimized.

Copy link
Author

ahrtr commented Dec 22, 2017

When I checked the logs of the second prometheus using command "docker logs 42f0d0d104d2", I saw there were lots of error logs as below. What's the root cause?

2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2017/12/22 09:02:58 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.

@brian-brazil

This comment has been minimized.

Copy link
Member

brian-brazil commented Dec 22, 2017

I'd guess this is a problem with the proxy you're attempting to use, that needs to go in the config file.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

@lock

This comment has been minimized.

Copy link

lock bot commented Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.