Default max local storage nodes to one

[jasontedor]

This commit defaults the max local storage nodes to one. The motivation
for this change is that a default value greather than one is dangerous
as users sometimes end up unknowingly starting a second node and start
thinking that they have encountered data loss.

Closes #19679, supersedes #19748

[nik9000]

Cute.

[rmuir]

The motivation
for this change is that a default value greather than one is dangerous
as users sometimes end up unknowingly starting a second node and start
thinking that they have encountered data loss.

Also that when locks are lost, the exception is masked and dropped on the floor and ES keeps on trucking. Oh and did i mention they are filesystem locks?

Seems legitimately unsafe.

[nik9000]

LGTM

[mikemccand]

Can we just remove this feature entirely, such that users must always be explicit on starting a node about precisely which path.data that node gets to use, and it must not be in use by any other node?

I think this is too much magic on ES's part, trying to have multiple nodes share path.data entries, and it has hit me (well, my brother, who then asked me WTF was happening) personally when he accidentally started up another node on the same box.

[jasontedor]

@mikemccand Doesn't this get us there, now you have to be explicit about wanting multiple nodes that share the same path.data? Removing the feature doesn't buy us much from a code perspective (we still need the locking code anyway), and it will complicate the build.

[jasontedor]

Thanks for reviewing @nik9000. I've merged this @mikemccand, but that shouldn't stop discussion on your proposal!

[mikemccand]

@mikemccand Doesn't this get us there, now you have to be explicit about wanting multiple nodes that share the same path.data?

This is definitely an improvement (thank you! progress not perfection!), but what I'm saying is I don't think such dangerous magic should even be an option in ES.

[rmuir]

Personally I feel the design is broken anyway, as i've said over, and over again. It relies on filesystem locking, which is unreliable by definition.

But worse, its lenient. Startup ES, index some docs, and go nuke that node.lock. You can continue about your business, continue indexing docs, and so on. The only thing you will see is this:

[2016-08-12 09:37:25,300][WARN ][env                      ] [_Oe9-bX] lock assertion failed
java.nio.file.NoSuchFileException: /home/rmuir/workspace/elasticsearch/distribution/zip/build/distributions/elasticsearch-5.0.0-alpha6-SNAPSHOT/data/nodes/0/node.lock

Why is such an important exception dropped on the floor and merely translated into a logger WARN?

This feature is 100% unsafe.

[nik9000]

I'm ok with removing the feature altogether. If we're already breaking backwards compatibility with the setting maybe we can just kill it? @clintongormley, what do you think?

I like that we did this now though because I have a feeling even if we do decide to remove the feature it'll take some time because lots of tests and the gradle build rely on it.

[rjernst]

The gradle build does not depend on it. Integ tests have unique installations per node, and even fantasy land tests create a unique temp dir per node for path.home iirc.