Remote cluster info returns fatally for invalid hostnames

[jen-huang]

I added a cluster with an invalid hostname:

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "jentest": {
          "seeds": [
            "testname:1234"
          ]
        }
      }
    }
  }
}

this is causing GET _remote/info to return fatally

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "unknown host [testname]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "unknown host [testname]",
    "caused_by": {
      "type": "unknown_host_exception",
      "reason": "testname: nodename nor servname provided, or not known"
    }
  },
  "status": 400
}

For our Remote Clusters management UI feature, we're using _remote/info to fetch the list and info of clusters for display. Ideally it would still return the list of all clusters, maybe with the error attached to the corresponding cluster.

[elasticmachine]

Pinging @elastic/es-distributed

[cjcenizal]

A temporary UI-based solution would be to give the user a button that says "Remove remote clusters that have errors". Clicking this button will show a confirmation modal which lists the remote clusters containing seed nodes that match the offending host name. Confirming their removal will send a request to _cluster/settings that removes them.