Discovery: only retry join when other node is not (yet) a master

When a node tries to join a master, the master may not yet be ready to accept the join request. In such cases we retry sending the join request up to 3 times before going back to ping. To detect this the current logic uses ExceptionsHelper.unwrapCause(t) to unwrap the incoming RemoteTransportException and inspect it's source, looking for `ElasticsearchIllegalStateException`. However, local `ElasticsearchIllegalStateException` can also be thrown when the join process should be cancelled (i.e., node shut down). In this case we shouldn't retry.

Since we can't introduce new exceptions in a BWC manner, we are forced to check the message of the exception.

Relates to #8972
Closes #8979

src/main/java/org/elasticsearch/discovery/zen/ZenDiscovery.java

@@ -472,7 +472,9 @@ private boolean joinElectedMaster(DiscoveryNode masterNode) {
                 return true;
             } catch (Throwable t) {
                 Throwable unwrap = ExceptionsHelper.unwrapCause(t);
-                if (unwrap instanceof ElasticsearchIllegalStateException) {
+                // With #8972 we add an explicit exception to indicate we should retry. We can't do this in a bwc manner
+                // so we are forced to check for message text here.
+                if (unwrap instanceof ElasticsearchIllegalStateException && unwrap.getMessage().contains("not master for join request")) {
                     if (++joinAttempt == this.joinRetryAttempts) {
                         logger.info("failed to send join request to master [{}], reason [{}], tried [{}] times", masterNode, ExceptionsHelper.detailedMessage(t), joinAttempt);
                         return false;