Remove file based default mappings

[rjernst]

Using files that must be specified on each node is an anti-pattern
from the API based goal of ES. This change removes the ability
to specify the default mapping with a file on each node.

closes #10620

[jpountz]

LGTM. Let's just add a note to the 2.0 migration guide?

[clintongormley]

Also need to delete this page: http://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-conf-mappings.html

[nik9000]

Using files that must be specified on each node is an anti-pattern
from the API based goal of ES.

I appreciate the irony of having to use local files for scripting.

[clintongormley]

I appreciate the irony of having to use local files for scripting.

<sigh> yes </sigh>

[jpountz]

LGTM

[KlavsKlavsen]

so for those of us who actually want changelog based operations - that is no longer an option? We must "trust" whomever gets access to the API - to remember to also write to a changelog - so we know the change has been made.. ? (or write "yet another script" myself)

I - and many others - prefer to make centralized configuration changes, which we then commit and roll out into production.. using puppet f.ex. This change means its no longer possible to let puppet manage how ones indexes is setup.. a fairly major part of ES configuration :(

I can hack around it.. write some script, which is executed on the change of a template - which then loads it into ES using http.. but I would call it an anti-pattern to make sure users can't use configuration management to manage their ES installations (without scripting it themselves). I do realize the current ES handling of templates is subpar.. it would be better to set it up to f.ex. use inotify - so it is informed by the kernel when an index template file is updated - and then simply try to verify and load it.. and if this fails it could log it - and even mail it if need be..

[VascoVisser]

What exactly is the scope of this change? Is the ability to specify a default index template via files also removed? The docs are not clear:

The ability to specify mappings in configuration files has been removed. To specify default mappings that apply to multiple indexes, use index templates instead.

I am assuming for now template files are also subject to this change, because mine are being ignored.

From an operations perspective one would like to completely configure an ES node before bringing it up. Im thinking for example how to otherwise robustly prevent situations like where an index operation hits a node that hasn't got its mapping provided via the API yet. Is the idea nodes have to be "shielded" from the network until the mappings have been uploaded? Is there an easy way to prevent an ES instance from accepting operations until a template/mapping has been provided?

[rjernst]

@VascoVisser A node joining an existing cluster should not start accepting index/search operations until it has the necessary cluster state to perform those operation (eg knowing metadata of existing indexes). Do you have an example setup where that is not true? If so, please open a new issue and we will be happy to investigate.

[VascoVisser]

I have a single node setup, running ES from within a docker container. With 1.6 I ship the index templates with the container, allowing me to provision by just starting the container. With 2.0 I know of no other way but that I would have to make sure nothing is able to write to the a newly deployed instance, then "manually" send over the templates via the REST API, and then make the new ES instance available to clients.

[rjernst]

@VascoVisser file based templates and mappings have a key inherent problem: the resolution priority is undefined. What if you set a default file based mapping, and then set default in your mappings? Which one wins? What if you then restart the node? Mappings need to be exclusively part of the cluster state, so that the cluster controls how they are synchronized across the cluster as nodes go up and down.

As for your problem, I don't know docker, but could you send the index template to elasticsearch during the instance startup, after elasticsearch has started? Let's move any further discussion to https://discuss.elastic.co, as this PR is closed.

[brusic]

I am also disappointed that this feature was removed. Just like KlavsKlavsen, I use a file based approach so that my mappings can be place in source control. Is using source control an anti-pattern nowadays? Removing features because someone might break something is catering to newbies at the expenses of power users. We do not need kid gloves.

[s1monw]

@brusic this is not about kid gloves. At the stage of the project we really need to streamline approaches to maintain high quality. We are struggeling a lot with legacy code because of tons of different code-paths and ways to do the same thing. For us and I am convinced as well for the user going though the cluster and the clusterstate for things like this is superior to file based approaches. It's also less error-prone. If you use version control you can do the same thing, everytime you update your mapping just update it in the cluster and it will be consistent on all nodes.

[shayts7]

@s1monw, I do understand the need to reduce code paths, and I appreciate that way-of-thinking, really.
But, this change complicates the deployment. I believe there are more ES admins that would like to have a different defaults ("not analyzed", "date field", etc..), and the config file was the best way to achieve that, especially if you deploy your ES using docker (then you can prepare a preconfigured docker and use it whenever you need a new ES node).
Now I must have a post-deployment process that creates my template. Also, I need to make sure ES is out of the network until my template is created successfully (to avoid situations where a new index will be created with the wrong defaults before I applied my template).

[s1monw]

Now I must have a post-deployment process that creates my template. Also, I need to make sure ES is out of the network until my template is created successfully (to avoid situations where a new index will be created with the wrong defaults before I applied my template).

wait, you templates are stored in the clusterstate so if a new node joins it gets the templates first and then can handle indices creation etc. I can only see a problem when you are initially starting up your cluster which is a one off? That is actually the way to go, you have all your stuff in the clusterstate and all nodes are consistent.

[shayts7]

I'm referring to the initial starting up of the cluster.

[s1monw]

The way elasticsearch should be used is non-filebased configuration. If your cluster is ready to serve index creations before it's ready you have to prevent this. I am sorry that we are complicating your env. but we have to simplify for the majority here. The common case is to have a long running cluster and startup initially is not necessarily a huge concern.

[shayts7]

I think that in this particular use case, a file based configuration will simplify things. Anyway, thanks for the attention and the quick response. really appreciate that!