Improved hostname resolution for Graphite connections

[spacecowboy]

This should fix a dead graphite metric in the case when the DNS
resolution to the graphite server updates while neo4j is running.

[spacecowboy]

@benbc @praveenag some proof that this works.

Java DNS TTL settings

See https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html

I have confirmed results when I set the java dns ttl, file

/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/java.security

on my system. It is possible that it works without this on openjdk but there was a significant delay before the metrics kick in again which I didn't expect. Basically, all connections in neo4j must timeout before a new resolution is attempted and that takes a couple of minutes.

#
# The Java-level namelookup cache policy for successful lookups:
#
# any negative value: caching forever
# any positive value: the number of seconds to cache an address for
# zero: do not cache
#
# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set. When a security
# manager is not set, the default behavior in this implementation
# is to cache for 30 seconds.
#
# NOTE: setting this to anything other than the default value can have
#       serious security implications. Do not set it unless
#       you are sure you are not exposed to DNS spoofing attack.
#
networkaddress.cache.ttl=10

# The Java-level namelookup cache policy for failed lookups:
#
# any negative value: cache forever
# any positive value: the number of seconds to cache negative lookup results
# zero: do not cache
#
# In some Microsoft Windows networking environments that employ
# the WINS name service in addition to DNS, name service lookups
# that fail may take a noticeably long time to return (approx. 5 seconds).
# For this reason the default caching policy is to maintain these
# results for 10 seconds.
#
#
networkaddress.cache.negative.ttl=10

Screenshots of graphite with this fix

Note that I tested it twice just to make sure I did it right, hence the two gaps. I did not restart neo4j, I only changed the Route53 dns and waited.

[benbc]

Cool. I'd absolutely expect there to be a gap if I'm messing around with the DNS. As long as it recovers, that's fine.

So do we need to document the JRE configuration?

[spacecowboy]

@benbc not sure where though. I don't think we have any specific graphite docs.