Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace action.auto_create_index with a template setting #20640

Closed
clintongormley opened this issue Sep 23, 2016 · 3 comments · Fixed by #61858
Closed

Replace action.auto_create_index with a template setting #20640

clintongormley opened this issue Sep 23, 2016 · 3 comments · Fixed by #61858
Assignees
@pugnascotia
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >feature help wanted adoptme Team:Data Management Meta label for data/management team

Comments

@clintongormley
Copy link

In #9359 we discussed the possibility of extending the action.auto_create_index setting to automatically recognise index names which match templates, and to allow those to be auto-created without having to maintain both the template and the setting.

In FixitFriday we came up with a better solution:

  • Remove the action.auto_create_index setting
  • Add an auto_create_index setting to index templates - if true, then an index can be auto-created
  • Ship with a default match * index with auto_create_index: true - this can be overridden by the user
  • If multiple templates match, the auto_create_index setting is merged just like all other template settings - the highest order wins

When upgrading, the user will see an exception if they set action.auto_create_index and will need to update their templates to reflect the same logic.

Possibly, we could add a deprecation period where the action.auto_create_index setting continues to work, but also accepts a _template value which switches to use the template-based logic instead.
@clintongormley clintongormley mentioned this issue Sep 23, 2016
Closed
@clintongormley clintongormley mentioned this issue Nov 26, 2016
Closed
@jasontedor jasontedor mentioned this issue Jan 4, 2017
Closed
@jasontedor jasontedor mentioned this issue Mar 2, 2017
Closed
@jpountz jpountz mentioned this issue Mar 27, 2017
Closed
@clintongormley clintongormley added :Data Management/Indices APIs APIs to create and manage indices and templates and removed :Index Templates labels Feb 13, 2018
@hub-cap hub-cap added :Data Management/Indices APIs APIs to create and manage indices and templates and removed :Data Management/Indices APIs APIs to create and manage indices and templates labels Mar 21, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@elasticmachine elasticmachine mentioned this issue Apr 25, 2018
Closed
@original-brownbear original-brownbear self-assigned this Aug 10, 2018
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Aug 17, 2018
@original-brownbear
CORE: Allow Setting Index Autocreate to _template
c509131 
* Enables setting autocreate by template as described in elastic#20640
* Closes elastic#20640
@original-brownbear original-brownbear mentioned this issue Aug 17, 2018
Closed
@m9aertner m9aertner mentioned this issue Dec 10, 2019
Open
@jaymode jaymode mentioned this issue Feb 27, 2020
Open
23 tasks
@jamshid
Copy link

jamshid commented Apr 8, 2020

This would be a great feature, any chance?
I want the convenience of automatically creating an index but only if the cluster has the template defining the settings and schema that my application needs.

@rjernst rjernst added the Team:Data Management Meta label for data/management team label May 4, 2020
@droberts195 droberts195 mentioned this issue May 7, 2020
Closed
@gwbrown
Copy link
Contributor

gwbrown commented Jul 27, 2020

We discussed #56369 in today's Fix-It meeting, and came to the conclusion that the proposal in this issue is closer to what we want to do that the one described in #56369.

There are some new considerations, however. In particular, the suggestion of a wildcard template ("index_patterns" : ["*"]) with the setting to allow auto-creation as a means of backwards compatibility is not possible with the new component/composable templates. Because of this, the latest thinking on this is that we should implement a flag for component/composable templates which overrides action.auto_create_index. This way, most cases will continue to use the behavior of action.auto_create_index, but we can ensure that our internal indices will be able to be auto-created as necessary.

With this proposal, we have the option of deprecating and removing action.auto_create_index, but per our discussion today we are likely to allow them to coexist, with the index template flag overriding action.auto_create_index.

@pugnascotia pugnascotia mentioned this issue Sep 2, 2020
Merged
pugnascotia added a commit that referenced this issue Oct 26, 2020
@pugnascotia
Add a template parameter to override auto_create_index value (#61858)
dc855ad 
Closes #20640.

This PR introduces a new parameter to v2 templates, `allow_auto_create`,
which allows templates to override the cluster setting `auto_create_index`.
Notes:

   * `AutoCreateIndex` now looks for a matching v2 template, and if its
     `allow_auto_create` setting is true, it overrides the usual logic.
   * `TransportBulkAction` previously used `AutoCreateIndex` to check
     whether missing indices should be created. We now rely on
     `AutoCreateAction`, which was already differentiating between creating
     indices and creating data streams.  I've updated `AutoCreateAction` to
     use `AutoCreateIndex`. Data streams are also influenced by
     `allow_auto_create`, in that their default auto-create behaviour can
     be disabled with this setting.
   * Most of the Java file changes are due to introducing an extra
     constructor parameter to `ComposableIndexTemplate`.
   * I've added the new setting to various x-pack templates
   * I added a YAML test to check that watches can be created even when
     `auto_create_index` is `false`.
pugnascotia added a commit to pugnascotia/elasticsearch that referenced this issue Oct 27, 2020
@pugnascotia
Add a template parameter to override auto_create_index value (elastic…
6cd648e 
…#61858)

Closes elastic#20640.

This PR introduces a new parameter to v2 templates, `allow_auto_create`,
which allows templates to override the cluster setting `auto_create_index`.
Notes:

   * `AutoCreateIndex` now looks for a matching v2 template, and if its
     `allow_auto_create` setting is true, it overrides the usual logic.
   * `TransportBulkAction` previously used `AutoCreateIndex` to check
     whether missing indices should be created. We now rely on
     `AutoCreateAction`, which was already differentiating between creating
     indices and creating data streams.  I've updated `AutoCreateAction` to
     use `AutoCreateIndex`. Data streams are also influenced by
     `allow_auto_create`, in that their default auto-create behaviour can
     be disabled with this setting.
   * Most of the Java file changes are due to introducing an extra
     constructor parameter to `ComposableIndexTemplate`.
   * I've added the new setting to various x-pack templates
   * I added a YAML test to check that watches can be created even when
     `auto_create_index` is `false`.
@pugnascotia pugnascotia mentioned this issue Oct 27, 2020
Merged
pugnascotia added a commit that referenced this issue Nov 5, 2020
@pugnascotia
Add a template parameter to override auto_create_index value (#64208)
e21b2e0 
Backport of #61858.

Closes #20640.

This PR introduces a new parameter to v2 templates, `allow_auto_create`,
which allows templates to override the cluster setting `auto_create_index`.
Notes:

   * `AutoCreateIndex` now looks for a matching v2 template, and if its
     `allow_auto_create` setting is true, it overrides the usual logic.
   * `TransportBulkAction` previously used `AutoCreateIndex` to check
     whether missing indices should be created. We now rely on
     `AutoCreateAction`, which was already differentiating between creating
     indices and creating data streams.  I've updated `AutoCreateAction` to
     use `AutoCreateIndex`. Data streams are also influenced by
     `allow_auto_create`, in that their default auto-create behaviour can
     be disabled with this setting.
   * Most of the Java file changes are due to introducing an extra
     constructor parameter to `ComposableIndexTemplate`.
   * I've added the new setting to various x-pack templates
   * I added a YAML test to check that watches can be created even when
     `auto_create_index` is `false`.
@stevejgordon stevejgordon mentioned this issue Dec 17, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pugnascotia pugnascotia

Labels
:Data Management/Indices APIs APIs to create and manage indices and templates >feature help wanted adoptme Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
8 participants
@jamshid @clintongormley @rjernst @hub-cap @gwbrown @original-brownbear @pugnascotia @elasticmachine