Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Short hostname results in "context deadline exceeded" #2332

Closed
stuart-c opened this Issue Jan 10, 2017 · 8 comments

Comments

Projects
None yet
2 participants
@stuart-c
Copy link

stuart-c commented Jan 10, 2017

What did you do?

Added a target with a very short (4 char) hostname and port number combo

What did you expect to see?

Successful polling of the new hostname

What did you see instead? Under which circumstances?

No stats - error of "context deadline exceeded". Still didn't work even if the timeout was increased significantly.

Environment

Official Docker container of Prometheus (:latest)

The following config snippet didn't work:

  • job_name: 'jira'
    static_configs:
    • targets: ['jira:8080']

But the following does:

  • job_name: 'jira'
    static_configs:
    • targets: ['jirax:8080']
@brian-brazil

This comment has been minimized.

Copy link
Member

brian-brazil commented Jan 10, 2017

This sounds like a name resolution issue on your end. Have you verified this name resolves and is accessible from within the container?

@stuart-c

This comment has been minimized.

Copy link
Author

stuart-c commented Jan 10, 2017

Yep. I used "docker exec" to get into the prometheus container and then could both ping "jira" and telnet to the correct port sending a manual HTTP request and getting the metric data back.

@brian-brazil

This comment has been minimized.

Copy link
Member

brian-brazil commented Jan 10, 2017

Deadline exceeded implies that DNS worked. Can you access jira via IP address?

Incidentally ping uses non-standard resolution. I wouldn't fully trust it for testing something like this.

@stuart-c

This comment has been minimized.

Copy link
Author

stuart-c commented Jan 11, 2017

Yep accessing via IP address works. If I just change the container/link name in Docker to have an extra char it works, so it is just the 4 char hostname/port (jira:8080) which doesn't work. I don't see the callback on the jira exporter side being called at all (this is a Python script using the client library)

@brian-brazil

This comment has been minimized.

Copy link
Member

brian-brazil commented Jan 11, 2017

Something weird is going on here anyway. Several of my home systems happen to be 4 chars, so I don't think it's on the Prometheus side.

@stuart-c

This comment has been minimized.

Copy link
Author

stuart-c commented Jan 11, 2017

Could it be something about being 9 total chars (i.e. host:port = 4:4 chars)?

It was a very odd thing and as I say just changing the hostname fixed it

@brian-brazil

This comment has been minimized.

Copy link
Member

brian-brazil commented Mar 27, 2017

This is likely something odd with resolution on your end. If this is still occurring, it's best to discuss on https://groups.google.com/forum/#!forum/prometheus-users

@lock

This comment has been minimized.

Copy link

lock bot commented Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 23, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.