Detecting exising indices and shards is broken

[s1monw]

Today we rely on FS operations to find the indices on disk or to find the shards for an index. This is super error prone and requires String parsing. We should on each level index -> shard know exactly what to expect and don't use directory listings which are expensive and subject to change. There should be a metadata file on each level that is atomically written that we open and see what we have to expect no matter of what's on the FS.

[dakrone]

+1 to moving away from parsing things from disk