Caching private IPs in /etc/hosts

[nicpottier]

Hey hey, ran into this when I got bit by AWS's DNS blowing up as well.

Looks close to what we need, but wondering how you are resolving the RDS private IPs since that seems related (and I gather from your blog post comments you also rely on RDS).

Just switching the resolv to Google's DNS will get you public IPs no? Will that fly?

Wondering whether another solution would be to "cache" the private ips for a configurable set of hosts and then write those to /etc/hosts when the primary DNS fails.

Thoughts? If I added this would you be interested? Alternatively how do you deal with this?

[kvz]

We can also connect to public IP, but that might not fly depending on your setup indeed. It's worth testing though, since that would solve a lot of complexity.

If it doesn't, writing to /etc/hosts also is the first thing that comes to my mind without introducing many more moving parts / servers that will just add more vulnerable links to your chain.

I do worry a bit about atomicity (what If I'm also making a change to /etc/hosts using vim or ansible or aws is doing so via cloud-init for instance). It seems hard to do this is a failsafe way (not impossible).

[nicpottier]

I might be down with using the public IP in those rare cases. Did you manage to get a security group rule that restricted it to just your security group? Haven't managed to do that since I think once you go to the public IP you are treated as any other public traffic incoming. Definitely don't want RDS open to the world.