feature request: a way to require index creation from template.

[glycerine]

There should be a way to tell elastic search to not create an index without using a template for the mapping.

As we roll to a new index at the start of each day, I like and utilize the "automatic" index creation upon first write to a new index very useful.

However, and this may be subtle: I really, really don't want elasticsearch (ES) to be guessing about the types of fields. I always want it to be creating an index based on a pre-existing template for the mapping.

I've been bitten very badly by elasticsearch's guessing the types of fields in the record, and therefore we would find it highly desirable to have a way to turn off that particular feature.

So the feature request is: some new setting, whenever an index is created, it must be from a template mapping, and the mapping should never be guessed at. ES should fail to create the new index if a matching template is not available, and no data should be writable to that (absent) index.

In other words, I want ES to always be using a template, or to fail to write and create a new index if there is no appropriate template available.

[jasontedor]

@glycerine I think that #20640 would meet your requirements. In this case, you would remove the default template that matches * that would specify auto_create_index set to true, and then indices could only be created if they match a template pattern that has auto_create_index set to true. Do you agree with this? If so, I would like to close this feature request as a duplicate of #20640.

[glycerine]

Hi @jasontedor, thanks for pointing out #20640.

I think that the suggestion works... I'm just trying to piece it all together.

The docs are a little confusing because auto_create_index can apparently be either a bool or a string, depending on context. To avoid confusion, I highly recommend using a new name for the new setting that goes inside the template. i.e. avoid reusing auto_create_index again as it is a string now (or proposed) and in a different location.

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-creation says

Automatic Index Creation

The index operation automatically creates an index if it has not been created before (check out the create index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has not yet been created (check out the put mapping API for manually creating a type mapping).

The mapping itself is very flexible and is schema-free. New fields and objects will automatically be added to the mapping definition of the type specified. Check out the mapping section for more information on mapping definitions.

Automatic index creation can be disabled by setting action.auto_create_index to false in the config file of all nodes. Automatic mapping creation can be disabled by setting index.mapper.dynamic to false per-index as an index setting.

Automatic index creation can include a pattern based white/black list, for example, set action.auto_create_index to +aaa*,-bbb*,+ccc*,-* (+ meaning allowed, and - meaning disallowed).

Q: In the example just above, the aaa* would be referring to... is that a regex to match an index name?

It would help to be discussing a specific example. Lets ask this: to accomplish the desired goal expressed in this ticket, what would be the steps, the precise protocol, to activate the "no-new-index-without-a-template" Step 1, step 2, step 3...

[jasontedor]

Q: In the example just above, the aaa* would be referring to... is that a regex to match an index name?

Yes, it's a regex.

Lets ask this: to accomplish the desired goal expressed in this ticket, what would be the steps, the precise protocol, to activate the "no-new-index-without-a-template"

Given the proposal in #20640, it would be only to remove the default template that matches * that has auto_create_index set to true. The only way that an index can be created without that template existing is if there is a template existing whose template pattern matches the name of the index being created. As such, it would then be created from the relevant template(s). Given your other requirement:

I really, really don't want elasticsearch (ES) to be guessing about the types of fields.

you would also want to disable dynamic mapping (index.mapper.dynamic).

[jasontedor]

Duplicates #20640