Default to unicast discovery, with default host list of 127.0.0.1, [::1] #12999

rmuir · 2015-08-19T18:26:33Z

Multicast discovery has serious issues such as #12993
It is never going to work correctly and securely with a "default to localhost" scenario. It also has numerous issues with operating system and JDK support.

Instead we should encourage unicast discovery by default, since thats what we encourage for production, and configure it to look at "localhost" by default which will give the same experience for startup/development.

This PR changes the default unicast host list (if not specified) to be "127.0.0.1, [::1]", enables unicast by default, disables multicast by default, fixes the parsing of unicast hosts to be strict, fixes it to allow IPV6 addresses unambiguously, and fixes it to work with hostnames that resolve to multiple addresses.

Things that should be deferred to followups (trust me, this is a huge change on its own, and i know a bunch of tests are angry, which i need help with, but the change works):

Move multicast zen discovery out to a plugin, possibly deprecate it.
Review all uses + ban InetSocketAddress(java.lang.String, int), which is lenient and bad to use. I did some cleanups here already.

If anyone wants to help, let me know I will give commit access to my branch here.

rmuir · 2015-08-19T18:30:27Z

core/src/main/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPing.java

@@ -64,7 +64,7 @@

    public static final String ACTION_NAME = "internal:discovery/zen/unicast";

-    public static final int LIMIT_PORTS_COUNT = 1;
+    public static final int LIMIT_PORTS_COUNT = 10;


This is really a nocommit. We should only do 10 i think when its the localhost default. Just trying to start simple :)

Maybe when its localhost we dont need any limiting, not sure, we should look into it. At least any limiting should be "per-address", since a host here could resolve to multiple addresses (e.g. ipv4 and v6, among other possibilities).

Cool! If we merge it I think we can close #8833 ! It will also help in discovery plugins.

…t limit for node local disco

…de IP addresses

s1monw · 2015-08-19T22:38:31Z

ok we are down to 1 awaits fix and all other tests pass - there is some cleanup work needed here for sure but this is a great start!!

…ur logs and stats apis

…o sleep, sorry

We used static methods reading sys properties to define the node mode per cluster. this had lots of problems when tests couldn't cope with mixed or only local mode. Now we are passing it down to the cluster from the test which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force consistent node configurations.

We only run multicast tests now when we explicitly state it. A working multicast env is required which is not always the case.

s1monw · 2015-08-20T08:26:32Z

I fixed all the remaining tests. Everything now runs on unicast and passes with network and local. We still have some forbiddenAPI:

[ERROR] Forbidden method invocation: java.net.InetSocketAddress#getHostName() [Use getHostString() instead, which avoids a DNS lookup]
[ERROR]   in org.elasticsearch.common.transport.InetSocketTransportAddress (InetSocketTransportAddress.java:96)
[ERROR] Forbidden method invocation: java.net.InetSocketAddress#getHostName() [Use getHostString() instead, which avoids a DNS lookup]
[ERROR]   in org.elasticsearch.common.transport.InetSocketTransportAddress (InetSocketTransportAddress.java:132)
[ERROR] Scanned 5737 (and 1524 related) class file(s) for forbidden API invocations (in 1.90s), 2 error(s).

but those are the last remaining issues. We can SuppressForbidden them if we wanna push the decision out to a different issue?

bleskes · 2015-08-20T08:29:30Z

core/src/main/java/org/elasticsearch/cluster/node/DiscoveryNode.java

@@ -360,16 +360,16 @@ public int hashCode() {
    public String toString() {
        StringBuilder sb = new StringBuilder();
        if (nodeName.length() > 0) {
-            sb.append('[').append(nodeName).append(']');
+            sb.append('{').append(nodeName).append('}');


why is this change? did we decide to change our usage of [] to {}?

because it is part of a bracketed ipv6 address specification.

+1 for this change though RFCs are a good thing to follow here

Yes, when an IPv6 address is combined with a port number, you really need them. Its needed to make things unambiguous, its needed to work inside a user's browser, its needed to conform with numerous RFCs on the display of this stuff. If you need more justification, see ones like https://tools.ietf.org/html/rfc5952

On the other hand, the brackets in ES logging provide NOTHING. Part of me wanted to remove them across all logging across the board.

bleskes · 2015-08-20T11:16:05Z

core/src/main/java/org/elasticsearch/discovery/zen/ping/unicast/UnicastZenPing.java

@@ -64,7 +64,9 @@

    public static final String ACTION_NAME = "internal:discovery/zen/unicast";

-    public static final int LIMIT_PORTS_COUNT = 1;
+    public static final int LIMIT_FOREIGN_PORTS_COUNT = 1;
+    public static final int LIMIT_LOCAL_PORTS_COUNT = 25;


Not a big deal, but 25 ports feels a bit high - we will try to connect to all these ports all the time, per local address. Maybe lower to 5? note that people would still be able to start more than 5 nodes (but if they kill the first 5, the cluster will not form again).

+1. The most I have seen in dev/test was 4 nodes.
The most I have seen in PROD is 3 (2 data nodes + 1 master node on 128gb machines)

@dadoonet this is only for default config without any configured unicast hosts. I don't think this is used in production. I can change it to 5 for now I don't have strong feelings

I have strong feelings. I dont want this lowered unless its per address.

Wake up guys, computers have more than one address these days.

I'm changing this to per-address. Otherwise someone is just going to follow up with some other PR and break dual-stack machines.

s1monw · 2015-08-20T14:45:25Z

I am +1 on pushing this - it looks good and has alll the fixes needed. We can do all otehr things in folllowups

rmuir · 2015-08-20T14:51:51Z

Summarizing the current tradeoffs (versus multicast) so we are all aware:

out of box bin/elasticsearch will discover nodes on the local machine by default just like multicast did.
adding "localhost" as a default unicast discovery host has some advantages over multicast: for example it works with ipv6-only configurations.
however, unlike multicast, as soon as you start messing with configuration (changing port numbers, etc), you are going to need to deal with discovery. in most cases, multicast would continue to work all the way into production, regardless.

jaymode · 2015-08-20T17:06:36Z

+1, this looks good to me

@multicast

Fix unicast discovery to work when a host has multiple addresses. Ban dangerous methods in java.net with forbidden APIs. Fix ipv6 bugs and formatting of network addresses everywhere. Closes #12999 Closes #12993 Squashed commit of the following: commit 6c1aa00 Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 14:25:43 2015 -0400 Fix these to be correct with addresses just in case commit 6482156 Merge: d00561b 41d8fbe Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 13:23:09 2015 -0400 Merge branch 'master' into unicast_all_the_way_down commit d00561b Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 16:38:50 2015 +0200 limit local ports to 5 in UnicastZenPing commit e2e15c5 Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 10:32:47 2015 -0400 fix port limiting commit 10153cb Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 10:18:37 2015 -0400 don't serialize scopeids: that's broken commit 2aa63d4 Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 16:06:51 2015 +0200 restore @Network commit c840f1d Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 16:02:30 2015 +0200 Use NetworkAddress.formatAddress where applicable in plugins commit 374ce87 Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 15:34:06 2015 +0200 Use NetworkAddress.formatAddress where applicable commit e7a606d Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 10:17:57 2015 +0200 Add @multicast annotation to disable multicast tests by default. We only run multicast tests now when we explicitly state it. A working multicast env is required which is not always the case. commit 2d7d2d0 Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 09:51:28 2015 +0200 Remove extra check for local mode in InternalTestCluster commit dda59ac Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 09:37:03 2015 +0200 Handle node mode across entire test cluster We used static methods reading sys properties to define the node mode per cluster. this had lots of problems when tests couldn't cope with mixed or only local mode. Now we are passing it down to the cluster from the test which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force consistent node configurations. commit 058197b Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 03:19:14 2015 -0400 really ban InetSocketAddress's trappy method and break build and go to sleep, sorry commit ac87791 Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 03:16:52 2015 -0400 Ban methods that might surprisingly cause DNS lookups commit e64fe3d Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 02:59:05 2015 -0400 Add unit test commit f15434f Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 02:39:02 2015 -0400 fix ipv6 formatting bugs commit 05c2c74 Author: Robert Muir <rmuir@apache.org> Date: Thu Aug 20 02:12:05 2015 -0400 format addresses correctly so I can actually read what comes out of our logs and stats apis commit 4f9389d Author: Robert Muir <rmuir@apache.org> Date: Wed Aug 19 21:26:52 2015 -0400 ban dangerous methods in java.net commit 6aacd4d Author: Robert Muir <rmuir@apache.org> Date: Wed Aug 19 20:59:24 2015 -0400 ban lenient method commit f466a84 Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 00:29:00 2015 +0200 fix tests to not mix local transport and zen unicast disco commit 0de007a Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 00:10:07 2015 +0200 fix tests to not mix local transport and zen unicast disco commit 539f6ca Author: Simon Willnauer <simonw@apache.org> Date: Thu Aug 20 00:02:01 2015 +0200 fix tests to not mix local transport and zen unicast disco commit 004c288 Author: Robert Muir <rmuir@apache.org> Date: Wed Aug 19 17:51:45 2015 -0400 Fix multinode commit 54113af Author: Robert Muir <rmuir@apache.org> Date: Wed Aug 19 17:36:45 2015 -0400 fix integration tests commit 0156a77 Author: Simon Willnauer <simonw@apache.org> Date: Wed Aug 19 23:32:18 2015 +0200 enable multicast in MulticastZenPingIT.java commit 1791caa Author: Robert Muir <rmuir@apache.org> Date: Wed Aug 19 17:23:16 2015 -0400 Fix constant commit 22820b5 Author: Simon Willnauer <simonw@apache.org> Date: Wed Aug 19 22:59:09 2015 +0200 give it some extra ids for local transport crazyness commit b2138fa Author: Simon Willnauer <simonw@apache.org> Date: Wed Aug 19 22:51:42 2015 +0200 pass on local addresses from configured transport rather than hard code IP addresses commit 1bf5de1 Author: Simon Willnauer <simonw@apache.org> Date: Wed Aug 19 22:04:31 2015 +0200 fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco commit b6706ed Author: Robert Muir <rmuir@apache.org> Date: Wed Aug 19 14:16:03 2015 -0400 Default to unicast discovery, with default host list of 127.0.0.1, [::1]

Multicast has known issues (see elastic#12999 and elastic#12993). This change moves multicast into a plugin, and deprecates it in the docs. It also allows for plugging in multiple zen ping implementations. closes elastic#13019

…, to enable new multicast Related: * elastic/elasticsearch#12999 * elastic/elasticsearch#13274

Default to unicast discovery, with default host list of 127.0.0.1, [::1]

b6706ed

rmuir added >breaking WIP v2.0.0 labels Aug 19, 2015

rmuir reviewed Aug 19, 2015
View reviewed changes

s1monw and others added 10 commits August 19, 2015 22:04

fix PluggableTransportModuleIT.java to use local disco and detach por…

1bf5de1

…t limit for node local disco

pass on local addresses from configured transport rather than hard co…

b2138fa

…de IP addresses

give it some extra ids for local transport crazyness

22820b5

Fix constant

1791caa

enable multicast in MulticastZenPingIT.java

0156a77

fix integration tests

54113af

Fix multinode

004c288

fix tests to not mix local transport and zen unicast disco

539f6ca

fix tests to not mix local transport and zen unicast disco

0de007a

fix tests to not mix local transport and zen unicast disco

f466a84

rmuir and others added 10 commits August 19, 2015 20:59

ban lenient method

6aacd4d

ban dangerous methods in java.net

4f9389d

format addresses correctly so I can actually read what comes out of o…

05c2c74

…ur logs and stats apis

fix ipv6 formatting bugs

f15434f

Add unit test

e64fe3d

Ban methods that might surprisingly cause DNS lookups

ac87791

really ban InetSocketAddress's trappy method and break build and go t…

058197b

…o sleep, sorry

Remove extra check for local mode in InternalTestCluster

2d7d2d0

Add @multicast annotation to disable multicast tests by default.

e7a606d

We only run multicast tests now when we explicitly state it. A working multicast env is required which is not always the case.

bleskes reviewed Aug 20, 2015
View reviewed changes

s1monw added blocker v2.0.0-beta1 :Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure labels Aug 20, 2015

bleskes reviewed Aug 20, 2015
View reviewed changes

s1monw added 3 commits August 20, 2015 15:34

Use NetworkAddress.formatAddress where applicable

374ce87

Use NetworkAddress.formatAddress where applicable in plugins

c840f1d

restore @Network

2aa63d4

s1monw mentioned this pull request Aug 20, 2015

Remove support for address resolving in InetSocketTransportAddress #13014

Closed

rmuir and others added 3 commits August 20, 2015 10:19

don't serialize scopeids: that's broken

10153cb

fix port limiting

e2e15c5

limit local ports to 5 in UnicastZenPing

d00561b

Merge branch 'master' into unicast_all_the_way_down

6482156

rmuir mentioned this pull request Aug 20, 2015

make multicast zen discovery a plugin #13019

Closed

Fix these to be correct with addresses just in case

6c1aa00

rmuir closed this in e2ab625 Aug 20, 2015

rjernst mentioned this pull request Aug 20, 2015

Move multicast discovery to a plugin #13027

Merged

This was referenced Aug 21, 2015

ipv6 support for unicast hosts #10675

Closed

Fully support IPv6 #10535

Closed

brwe removed the v2.0.0 label Aug 21, 2015

s1monw deleted the unicast_all_the_way_down branch August 21, 2015 19:25

clintongormley removed the WIP label Aug 24, 2015

karmi added a commit to elastic/elasticsearch-ruby that referenced this pull request Sep 17, 2015

[EXT] Default the network host for the testing cluster to "localhost"…

92a8a5b

…, to enable new multicast Related: * elastic/elasticsearch#12999 * elastic/elasticsearch#13274

drekbour mentioned this pull request Jan 7, 2016

Correct documentation of discovery.zen.ping.unicast.hosts regarding support for port ranges #15816

Closed

clintongormley mentioned this pull request Jan 29, 2016

Deprecate and remove the multicast plugin #16310

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default to unicast discovery, with default host list of 127.0.0.1, [::1] #12999

Default to unicast discovery, with default host list of 127.0.0.1, [::1] #12999

rmuir commented Aug 19, 2015

rmuir Aug 19, 2015

rmuir Aug 19, 2015

dadoonet Aug 19, 2015

s1monw commented Aug 19, 2015

s1monw commented Aug 20, 2015

bleskes Aug 20, 2015

rmuir Aug 20, 2015

s1monw Aug 20, 2015

rmuir Aug 20, 2015

bleskes Aug 20, 2015

dadoonet Aug 20, 2015

s1monw Aug 20, 2015

rmuir Aug 20, 2015

rmuir Aug 20, 2015

s1monw commented Aug 20, 2015

rmuir commented Aug 20, 2015

jaymode commented Aug 20, 2015

Default to unicast discovery, with default host list of 127.0.0.1, [::1] #12999

Default to unicast discovery, with default host list of 127.0.0.1, [::1] #12999

Conversation

rmuir commented Aug 19, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented Aug 19, 2015

s1monw commented Aug 20, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1monw commented Aug 20, 2015

rmuir commented Aug 20, 2015

jaymode commented Aug 20, 2015