headless service discovery doesn't work on EKS 1.21

[andycmaj]

error seen when starting ES pods in statefulSet:

{"type": "server", "timestamp": "2021-12-21T20:34:18,952Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "production", "node.name": "prod-cluster-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [prod-cluster-master-1, prod-cluster-master-2, prod-cluster-master-3] to bootstrap a cluster: have discovered [{prod-cluster-master-0}{xDH5es11QgKEX0v1nLnVYA}{c391A-elStqo6dXv-6rUgg}{192.168.157.8}{192.168.157.8:9300}{lmr}{ml.machine_memory=7569141760, xpack.installed=true, transform.node=false, ml.max_open_jobs=20}, {prod-cluster-master-1}{75WHjYWFTs6zsefUuSazYQ}{C5SGMQdnQOiSuNy9c5sh5A}{192.168.91.238}{192.168.91.238:9300}{lmr}{ml.machine_memory=7569141760, ml.max_open_jobs=20, xpack.installed=true, transform.node=false}]; discovery will continue using [] from hosts providers and [{prod-cluster-master-0}{xDH5es11QgKEX0v1nLnVYA}{c391A-elStqo6dXv-6rUgg}{192.168.157.8}{192.168.157.8:9300}{lmr}{ml.machine_memory=7569141760, xpack.installed=true, transform.node=false, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2021-12-21T20:34:19,743Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "production", "node.name": "prod-cluster-master-0", "message": "failed to resolve host [prod-cluster-master-headless]", 
"stacktrace": ["java.net.UnknownHostException: prod-cluster-master-headless",

attempting to resolve headless service from a debug pod in the same NS:

root@sh:/# nslookup prod-cluster-master-headless
Server:		10.100.0.10
Address:	10.100.0.10#53

** server can't find prod-cluster-master-headless: NXDOMAIN

after changing service to add publishNotReadyAddresses: True

root@sh:/# nslookup prod-cluster-master-headless
Server:		10.100.0.10
Address:	10.100.0.10#53

Name:	prod-cluster-master-headless.runtime.svc.cluster.local
Address: 192.168.91.238
Name:	prod-cluster-master-headless.runtime.svc.cluster.local
Address: 192.168.157.8

deprecation of old annotation

apparently the annotation service.alpha.kubernetes.io/tolerate-unready-endpoints: 'true' has been deprecated or isn't used by EKS coreDNS.

[iamabhishek-dubey]

Fixed in #20