New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GELF output hostname resolution only performed during initialisation #882

Closed
gavllew opened this Issue Jan 15, 2015 · 5 comments

Comments

Projects
None yet
3 participants
@gavllew

gavllew commented Jan 15, 2015

graylog2-server 0.92.3

When configuring a new GELF output, a destination hostname is requested. Through testing, it would appear that this hostname is only resolved once (when the output is initialised), and any subsequent changes to the DNS record are ignored until the graylog service is restarted.

For the TCP transport, it would make sense to at least re-resolve the hostname when reconnecting. For the UDP transport, re-resolving the address periodically may be a good idea.

Use case: our graylog installation runs in a cloud environment, as does the service that receives the GELF output. To support fail-over and graceful upgrades, we would usually configure a DNS entry with a short TTL to point at the appropriate server.

@kroepke kroepke added this to the 1.0.0 milestone Jan 16, 2015

@kroepke

This comment has been minimized.

Member

kroepke commented Jan 16, 2015

There are JVM parameters for DNS caching, usually those defaults are geared towards not performing lookups again, IIRC.

Have a look at http://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html#nct to see if tweaking those settings helps to solve your problem.
If they do please let us know so we can put this into our documentation somewhere.

I'll leave this ticket open for that purpose.

@gavllew

This comment has been minimized.

gavllew commented Jan 16, 2015

Thanks for the info, but I don't think that's what is causing this particular behaviour. In graylog2-server/src/main/java/org/graylog2/outputs/GelfOutput.java, the buildTransport method constructs an InetSocketAddress to pass to the GelfConfiguration constructor. This means that the underlying GELF client only possesses an IP address (not the hostname), and since the client handles the reconnects all future connections will target that original address.

@kroepke

This comment has been minimized.

Member

kroepke commented Jan 16, 2015

I see. Yes we should change that, we'll see if we can get it done before the final.

@kroepke kroepke added bug and removed documentation labels Jan 16, 2015

@bernd bernd self-assigned this Jan 19, 2015

bernd added a commit to Graylog2/gelfclient that referenced this issue Jan 19, 2015

Avoid resolving the ip address early in the configuration.
Until this, we took an InetSocketAddress and reused that in
getRemoteAddress(). This is problematic if the ip address for the
hostname changes between re-connects.

We now store the hostname and port in the configuration object and
create a new InetSocketAddress for every call to getRemoteAddress().

See Graylog2/graylog2-server#882.

@bernd bernd closed this in 85fb369 Jan 20, 2015

@gavllew

This comment has been minimized.

gavllew commented Jan 20, 2015

Thanks!

@bernd

This comment has been minimized.

Member

bernd commented Jan 20, 2015

There is still an issue for the UDP transport. See Graylog2/gelfclient#5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment