Add a template parameter to override auto_create_index value #61858

pugnascotia · 2020-09-02T14:58:14Z

Closes #20640.

This PR introduces a new parameter to v2 templates, allow_auto_create, which allows templates to override the cluster setting auto_create_index. Notes:

AutoCreateIndex now looks for a matching v2 template, and if its allow_auto_create setting is true, it overrides the usual logic.
TransportBulkAction previously used AutoCreateIndex to check whether missing indices should be created. After talking to @martijnvg, we now rely on AutoCreateAction, which was already differentiating between creating indices and creating data streams. I've updated AutoCreateAction to use AutoCreateIndex.
Most of the Java file changes are due to introducing an extra constructor parameter to ComposableIndexTemplate.
I've added the new setting to various x-pack templates
I added a YAML test to check that watches can be created even when auto_create_index is false.

…-templates

pugnascotia · 2020-09-02T15:07:44Z

@martijnvg draft PR as discussed in Slack.

cc @jaymode for your interest.

martijnvg · 2020-09-03T12:59:45Z

server/src/main/java/org/elasticsearch/action/support/AutoCreateIndex.java

+
+        // Templates can override the AUTO_CREATE_INDEX_SETTING setting
+        final ComposableIndexTemplate template = findTemplate(index, state.metadata());
+        if (template != null && template.getAllowAutoCreate()) {


We talked about the auto create index logic and the allow auto create template flag yesterday. This logic is invoked from the bulk action and is evaluated on the coordinating node. While the decision the auto create an index or data stream is done in the AutoCreateAction on the elected master node.

The reason why we decided to make the decision to auto create an index or data stream in the AutoCreateAction and not here is, because if for some reason a node is lacking behind and doesn't have all the templates then an index is auto created instead of a data stream. Fixing the erroneous situation requires a lot of manual work, hence we moved making this decision to elected master node.

The question is whether the auto create index check (with this enhancement) should be moved to AutoCreateAction class too, so that all the decisions are made on the elected master node.

I'm currently thinking the following:

In the case a coordinating node doesn't have the latest action.auto_create_index setting values or composable index templates then the worst that can happen is a new few write requests fail, because the target index doesn't exist.

Currently if action.auto_create_index is set to false then also data streams are no longer created. I think this is a bug, because the name of the setting implies auto creation of indices only.

If this logic is moved to AutoCreateAction class then the overhead is that remote requests are made to the elected master node in the case target indices don't exist. Most requests will index into an existing data stream or index, so the cost is acceptable.

I remember that we eventually want to deprecate and remove the action.auto_create_index setting, but I may be mistaken.

If we are going to remove action.auto_create_index with the new allow auto create template flag then I think let's keep it here? And then maybe also add logic here that if a non existing index matches with a template that auto create data streams then checking the action.auto_create_index setting should be ignored?

…-templates

…n 7.10

…-templates

pugnascotia · 2020-09-18T11:08:52Z

@martijnvg can you take a look at what I've done here with AutoCreateAction?

martijnvg · 2020-09-18T14:24:10Z

@pugnascotia I like the overall direction of this PR. Moving the auto create index check to AutoCreateAction is a good change, since that fixes the problem that data streams are no longer auto created if action.auto_create_index is set to false, at the cost of an additional hop to elected master node (this cost is neglectable, as we only redirect to auto create action is index is missing and in order to create the missing index the extra hop to elected master node needs to be done anyway).

We have an `@After` annotated method in `AggregatorTestCase` that cleans up releasibles. But it was `private`! Confusingly, it seemed to be working! I'm not sure why. This makes it public which is a little more sensible.

Now that we've got regexes enabled by default (elastic#63029) this adds a test to runtime fields just to make sure that it works with regexes. It does, but this adds a test to make sure it continues to work.

This commit extends the set of default metrics for the data frame analytics evaluation API to all available metrics. The motivation is that if the user skips setting an explicit set of metrics, they get most of the evaluation offering.

Run time field that emits geo points from lat/lon values.

…ic#63970) Relates to elastic#63969

Currently when searching with an empty string as lower bound for a range query on text-based fields we return all documents when 'gte' is used (including the lower bound) but no documents when 'gt' is used. This might seem counterintuitive since every value should be greate than the empty string. The bug has been fixed in Lucene and this PR adds a test for assuring we observe the fixed behaviour on searches now. Closes elastic#63386

…c#63974) Relates to elastic#63973

Relates to elastic#63719

…-templates

gwbrown

This LGTM now, thanks for dealing with all the details around the interaction with data streams! I know multiple rounds of back-and-forth like that can be frustrating.

docs/reference/indices/put-component-template.asciidoc

Mention data streams in the docs changes. Co-authored-by: Gordon Brown <arcsech@gmail.com>

pugnascotia · 2020-10-22T08:30:25Z

@jaymode are you happy with these changes now? There's still a "changes requested" on the PR.

pugnascotia · 2020-10-22T09:49:14Z

@elasticmachine update branch

jaymode

LGTM

jaymode · 2020-10-22T16:20:10Z

There's still a "changes requested" on the PR.

If you don't mind, can you re-request a review once you feel the PR is ready for me to look at the next time.

martijnvg

Left one small comment. LGTM otherwise.

martijnvg · 2020-10-26T11:01:39Z

qa/smoke-test-http/src/test/java/org/elasticsearch/http/AutoCreateIndexIT.java

+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.hamcrest.Matchers.containsString;
+
+public class AutoCreateIndexIT extends ESRestTestCase {


Maybe distribution/archives/integ-test-zip is a better place for this test class? Given that it doesn't extend HttpSmokeTestCase and this test class doesn't seem to require to start a node with special settings or plugins.

I'm honestly not sure. I tried before to find out where is the "correct" place for REST tests, but couldn't get a clear answer. I think I'll leave this test alone and start an email thread about it. I feel that it ought to be easier to say where a REST test should live.

pugnascotia · 2020-10-26T14:44:53Z

This ought to be backported after #63122, I'm trying to find out what is happening with that PR.

…#61858) Closes elastic#20640. This PR introduces a new parameter to v2 templates, `allow_auto_create`, which allows templates to override the cluster setting `auto_create_index`. Notes: * `AutoCreateIndex` now looks for a matching v2 template, and if its `allow_auto_create` setting is true, it overrides the usual logic. * `TransportBulkAction` previously used `AutoCreateIndex` to check whether missing indices should be created. We now rely on `AutoCreateAction`, which was already differentiating between creating indices and creating data streams. I've updated `AutoCreateAction` to use `AutoCreateIndex`. Data streams are also influenced by `allow_auto_create`, in that their default auto-create behaviour can be disabled with this setting. * Most of the Java file changes are due to introducing an extra constructor parameter to `ComposableIndexTemplate`. * I've added the new setting to various x-pack templates * I added a YAML test to check that watches can be created even when `auto_create_index` is `false`.

Backport of #61858. Closes #20640. This PR introduces a new parameter to v2 templates, `allow_auto_create`, which allows templates to override the cluster setting `auto_create_index`. Notes: * `AutoCreateIndex` now looks for a matching v2 template, and if its `allow_auto_create` setting is true, it overrides the usual logic. * `TransportBulkAction` previously used `AutoCreateIndex` to check whether missing indices should be created. We now rely on `AutoCreateAction`, which was already differentiating between creating indices and creating data streams. I've updated `AutoCreateAction` to use `AutoCreateIndex`. Data streams are also influenced by `allow_auto_create`, in that their default auto-create behaviour can be disabled with this setting. * Most of the Java file changes are due to introducing an extra constructor parameter to `ComposableIndexTemplate`. * I've added the new setting to various x-pack templates * I added a YAML test to check that watches can be created even when `auto_create_index` is `false`.

pugnascotia added 5 commits August 3, 2020 16:15

Override auto-create index behaviour via a template flag

ee50924

WIP - trying to test watcher

f62c83b

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

097d9f5

…-templates

Fixes

3bb580c

Remove logging

42cfc00

pugnascotia marked this pull request as draft September 2, 2020 14:58

pugnascotia requested a review from martijnvg September 2, 2020 14:59

pugnascotia changed the title ~~Add a template parameter to override auto_create_index value~~ [Draft] Add a template parameter to override auto_create_index value Sep 2, 2020

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

8c17371

…-templates

martijnvg reviewed Sep 3, 2020

View reviewed changes

pugnascotia added 14 commits September 4, 2020 10:12

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

9a1e10a

…-templates

Use AutoCreateIndex in AutoCreateAction

961cfcf

Imports

41fdcb3

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

2228033

…-templates

Fixes

b00297e

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

ad5021d

…-templates

Fix compile error

ad9e699

Tweak ComposableIndexTemplate serialisation under code changes land i…

c42e3ce

…n 7.10

Fix auto create index bug

6a689b2

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

f81a510

…-templates

Test fix

2d54d2c

Handle null Boolean

85abcbc

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

eb299a0

…-templates

Imports

a3e3691

pugnascotia requested a review from martijnvg September 18, 2020 11:08

pugnascotia added 2 commits September 18, 2020 16:13

Checkstyle

9421140

Tweaks

ebb42d4

nik9000 and others added 10 commits October 21, 2020 12:10

Make agg test less confusing (elastic#63952)

9a9b96b

We have an `@After` annotated method in `AggregatorTestCase` that cleans up releasibles. But it was `private`! Confusingly, it seemed to be working! I'm not sure why. This makes it public which is a little more sensible.

Add a test for regex usage to runtime fields (elastic#63951)

7db889c

Now that we've got regexes enabled by default (elastic#63029) this adds a test to runtime fields just to make sure that it works with regexes. It does, but this adds a test to make sure it continues to work.

geo_point runtime field implementation (elastic#63164)

1376afd

Run time field that emits geo points from lat/lon values.

Mute CoreValuesSourceTypeTests.testDatePrepareRoundingWithDocs (elast…

dec6ea9

…ic#63970) Relates to elastic#63969

Mute DatafeedJobsIT#testDatafeedTimingStats_DatafeedRecreated (elasti…

473975c

…c#63974) Relates to elastic#63973

Mute FieldSortIT#testCastDate (elastic#63972)

f45ea30

Relates to elastic#63719

Only subject data streams to allow_auto_create, not auto_create_index

0482f5c

Merge remote-tracking branch 'upstream/master' into 20640-auto-create…

421679d

…-templates

gwbrown approved these changes Oct 21, 2020

View reviewed changes

docs/reference/indices/put-component-template.asciidoc Outdated Show resolved Hide resolved

Tweak docs

6bd5456

Mention data streams in the docs changes. Co-authored-by: Gordon Brown <arcsech@gmail.com>

Merge branch 'master' into 20640-auto-create-templates

0fd0f84

jaymode approved these changes Oct 22, 2020

View reviewed changes

martijnvg approved these changes Oct 26, 2020

View reviewed changes

pugnascotia merged commit dc855ad into elastic:master Oct 26, 2020

pugnascotia deleted the 20640-auto-create-templates branch October 26, 2020 12:35

pugnascotia added the backport pending label Oct 26, 2020

pugnascotia mentioned this pull request Oct 27, 2020

Add a template parameter to override auto_create_index value #64208

Merged

pugnascotia removed the backport pending label Oct 27, 2020

dakrone mentioned this pull request Nov 6, 2020

Include full list of recommended action.auto_create_index values for bootstrap check #29737

Closed

stevejgordon mentioned this pull request Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a template parameter to override auto_create_index value #61858

Add a template parameter to override auto_create_index value #61858

pugnascotia commented Sep 2, 2020 •

edited

Loading

pugnascotia commented Sep 2, 2020

martijnvg Sep 3, 2020

pugnascotia commented Sep 18, 2020

martijnvg commented Sep 18, 2020

gwbrown left a comment

pugnascotia commented Oct 22, 2020

pugnascotia commented Oct 22, 2020

jaymode left a comment

jaymode commented Oct 22, 2020

martijnvg left a comment

martijnvg Oct 26, 2020

pugnascotia Oct 26, 2020

pugnascotia commented Oct 26, 2020

Add a template parameter to override auto_create_index value #61858

Add a template parameter to override auto_create_index value #61858

Conversation

pugnascotia commented Sep 2, 2020 • edited Loading

pugnascotia commented Sep 2, 2020

martijnvg Sep 3, 2020

Choose a reason for hiding this comment

pugnascotia commented Sep 18, 2020

martijnvg commented Sep 18, 2020

gwbrown left a comment

Choose a reason for hiding this comment

pugnascotia commented Oct 22, 2020

pugnascotia commented Oct 22, 2020

jaymode left a comment

Choose a reason for hiding this comment

jaymode commented Oct 22, 2020

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Oct 26, 2020

Choose a reason for hiding this comment

pugnascotia Oct 26, 2020

Choose a reason for hiding this comment

pugnascotia commented Oct 26, 2020

pugnascotia commented Sep 2, 2020 •

edited

Loading