Decide on service discovery strategy for Boulder

[jsha]

Right now, Boulder discovers its gRPC backends via DNS, using gRPC's default "dns" resolver. This looks up a hostname, and sends traffic to all of the resulting IP addresses. Boulder's VA does not have gRPC backends, but has DNS backends. Right now these are statically configured via JSON and never updated except on restart.

We'd like to dynamically discover DNS backends for VA (for instance, to allow autoscaling). Also, we'd like to update our gRPC service discovery. We've noticed that gRPC never updates its backends after starting up.

According to grpc/grpc-go#1663 (comment), gRPC re-resolves on connection errors, and we can intentionally trigger periodic connection errors by setting keepalive.MaxConnectionAge. This should be non-disruptive since connections are closed cleanly, with extant RPCs given time to complete. Also relevant: grpc/grpc#12295

If that works, we can stick with DNS-based service discovery for gRPC. For consistency, we would probably also want to use DNS-based service discovery for DNS backends.

For autoscaling our remote VA DNS backends, we can use AWS Cloud Map. It supports DNS-based discovery. For on-prem, we would use our current DNS setup, which doesn't have any concept of auto-scaling. Using DNS for both would allow us to use the same code for on-prem and cloud.

Related: #5306

[jsha]

We decided to use DNS with SRV records for discovering our Unbound backends, and to continue to use DNS with A records to discovering our gRPC backends. For gRPC we're using MaxConnectionAge to discover changes faster.