Java runtimes using SecurityManagers unpragmatically cache DNS hits *forever*

[rtyley]

This is the default behaviour of Java runtimes (in both Java 8 & 11 apparently), defined in the java.security file - they cache DNS hits forever:

# The Java-level namelookup cache policy for successful lookups:
#
# any negative value: caching forever
# any positive value: the number of seconds to cache an address for
# zero: do not cache
#
# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set. When a security
# manager is not set, the default behavior in this implementation
# is to cache for 30 seconds.
#
# NOTE: setting this to anything other than the default value can have
#       serious security implications. Do not set it unless
#       you are sure you are not exposed to DNS spoofing attack.
#
#networkaddress.cache.ttl=-1

Although intended as a security measure against DNS-spoofing, it's not very realistic in the AWS-based-world in which we operate - services do change their IP addresses from time to time, and so AWS recommend that a networkaddress.cache.ttl=60 setting be applied.

On the Ophan team @amyhughes & I have seen this result in us losing monitoring data as the Elastic-hosted monitoring cluster changed it's IP without our Elasticsearch nodes noticing- they attempted to continue sending data to the old IP addresses.

Elastic themselves recommend setting networkaddress.cache.ttl appropriately.

See also https://stackoverflow.com/q/1256556/438886

The java.security configuration file

Locating and patching the java.security file is a little fiddly - this is where you find it on today's images:

/etc/java-8-openjdk/security/java.security

Could we bake the updated setting into the AMI role?

[alexduf]

I don't know about other teams but on mobile I'd quite like to see that baked in the default image.