Add the ability to set attributes on path.data entries

[bra-fsn]

Elasticsearch currently supports adding attributes to nodes. You can start a node with:
./bin/elasticsearch -Enode.attr.storage_type=ssd
and another with:
./bin/elasticsearch -Enode.attr.storage_type=hdd
and you can place indices to a hdd or ssd node:

PUT /logs_2016-12-26
{
  "settings": {
    "index.routing.allocation.require.storage_type": "hdd"
  }
}

The problem with this you can only place attributes onto nodes, but a(n elasticsearch) node can have multiple data paths and you can't use this feature to decide what index goes to what path.

So I propose a new feature, which makes it possible to add attributes to path.data entries.
Setting this could work like this:
path.data: /hdd/0;path.attr.storage_type=hdd, /hdd/1;path.attr.storage_type=hdd, /hdd/2;path.attr.storage_type=hdd, /hdd/3, /ssd/0;path.attr.storage_type=ssd, /ssd/1;path.attr.storage_type=ssd

path.attr could take precedence over node.attr, meaning if you've started the above node with:
./bin/elasticsearch -Enode.attr.storage_type=hdd
and set the routing.allocation for the logs_2016-12-26 to hdd, then it could be placed to any of the /hdd data paths (even onto /hdd/3, which doesn't have a path.attr set, because node.attr takes place there), but not to the ssd ones.

And if you have indices with routing.allocation requiring a storage_type of ssd, they could be placed onto this node (even if it had node.attr.storage_type=hdd), but only to /ssd/* paths, because those paths are marked as storage_type=ssd.

I think this is easy to do and offers much more flexibility than elasticsearch offers today.

[elasticmachine]

Pinging @elastic/es-distributed

[jasontedor]

Duplicates #24181. See the details there. You can control this already without the need for us to add path attributes. Node attributes and custom paths should be enough.

[bra-fsn]

By using index.data_path, you are limited to that path only. I think it's quite inflexible.
I mean, I have nodes with multiple paths (both ssds and hdds) and some nodes have fewer, others have more, but all have more than one. By implementing the above, you just don't care, it will work seamlessly without the need to hardcode paths, the same way with node attributes, which is a very elegant and flexible way to allocate shards/indices to nodes.

[bleskes]

@bra-fsn I understand the need and indeed the custom path is a work around. Sadly managing allocation on the path level is something that requires a lot of work on the allocation layer - all of our code now has a shard to node mapping. I don't think we can realistically implement these kind of features any time in the foreseeable future. I'm going to close the issue to communicate that fact.