Add index.data_path setting

[dakrone]

This allows specifying the path an index will be at.

index.data_path is specified in the settings when creating an index,
and can not be dynamically changed.

An example request would look like:

POST /myindex
{
  "settings": {
    "number_of_shards": 2,
    "data_path": "/tmp/myindex"
  }
}

And would put data in /tmp/myindex/0/index/0 and /tmp/myindex/0/index/1

Since this can be used to write data to arbitrary locations on disk, it
requires enabling the node.enable_custom_paths setting in
elasticsearch.yml on all nodes.

I found that the NodeEnvironment abstraction works well for index-specific
data paths, and passing the index settings in to the various methods gives
us more flexibility in the future with regard to adding any other environment-
specific settings.

[dakrone]

I've left out documenting this feature, as I suspect the settings/configuration may look different after feedback on this PR, before merging this I will write documentation.

[rjernst]

Why is the templating needed? This seems like something a user should never mess with? For example, restoring snapshotted index would then not work on a cluster that didn't have the same template setup..

[dakrone]

Why is the templating needed? This seems like something a user should never mess with?

The templating is another feature because we originally discussed shard-level folder settings, it's not required for anything. I agree it does add complexity and I'd be totally fine with removing it. @clintongormley do you know if there's a reason we need to support custom templating for shard-specific folders, or can we stick with the default template all the time?

[s1monw]

can data_template be a constant?

[s1monw]

@dakrone I left a bunch of comments but everything minor. I also wonder if we should remove the templating for now and maybe make it a different PR just to push out the discussion? What do you think?

[dakrone]

@s1monw sounds reasonable to me, I will add a commit to remove it and address the feedback

[dakrone]

@s1monw I've removed the templating and added a custom data_path usage 30% of the time in ElasticsearchIntegrationTests, as well as addressing your other comments.

[s1monw]

left one comment other than that LGTM

[javanna]

why a single data node? If you need that, you can use numDataNodes=1 instead

[dakrone]

I've squashed, rebased (since master was heavily changed), and added a node-level setting node.enable_custom_paths (defaults to false) for allowing or disallowing custom data paths.

@s1monw can you take one last look? None of the original logic has been changed.

[rjernst]

@dakrone Why do we need one setting controlling another? They are both in yml, so anyone who can change that at the node level can change both?

[rjernst]

Oh I see...one is at the index level and one at yml...still, is it really necessary?

[dakrone]

Oh I see...one is at the index level and one at yml...still, is it really necessary?

index.data_path is set per-index, not in elasticsearch.yml at all. node.enable_custom_paths in elasticsearch.yml (as you mentioned, just clarifying).

As to whether it's necessary, yes, because this is intended for mixed clusters where some indices use the path.data directory, and others use a (perhaps slower or faster) disk.

[dakrone]

This is having trouble on Windows, I have reverted it for now while I look into it, I'll open a new PR when fixed.