[Index management] Prepare support Index template V2 format #61588

sebelga · 2020-03-27T09:44:14Z

This PR prepares the index_management plugin to support the upcoming Index Template V2 (elastic/elasticsearch#53101).

Context

In v7.8 Elasticsearch is going to release a new API to read/create/update/delete index templates. Those index templates will have a different format (V2) that the current index template (V1).

The main difference being:

// V1 format

{
  "templateName": {
    "aliases: {},
    "mappings": {}.
    "settings": {},
    ...
  }
}

// V2 format
{
  "templateName: {
    "template": { // wrapped under _template_ parameter
      "aliases: {},
      "mappings": {}.
      "settings": {},
    },
    // new parameter. An array of template components (upcoming new API for 7.8),
    "composed_of": []
    // new parameter. An integer to set the priority of the index template to apply in case 2 index patterns matches an index name
    "priority: 1,
    ... // remaining parameters are the same
  }
}

There will be 2 different APIs to manage index templates:

PUT _template for the V1 format (supported until the end of 8.x cycle)
PUT _index_template for the V2 format (supported from 7.8)

This means that 2 index templates can have the same name with a different version.

Changes in this PR

This PR is a preparation work to support both formats in the UI. The client code has been refactored to work only with V2 index template format (it expects V2 format to be thrown at it) as it will be the future of index template. A _kbnMeta object is added from the server to let the client know the format version of the template.

Knowing the format version is important for the client as, when we will fetch or delete an index template, we need to tell our API server route the format version so we can proxy the call to the correct Elasticsearch API.

We will also, in a future PR, enhance the table view to clearly indicate to the user the format of the index template. And probably add a deprecation warning for v1 template and a button to convert V1 index templates to the V2 format.

This PR still creates V1 index templates but our goal for 7.8 is to create V2 index templates by default.

How to review

Create an index template and verify that the Request preview (in Review step) is the V1 format

Edit the index template (from the pencil icon in the table, the context menu in the table and the context menu in the details flyout), make sure it saves correctly
Clone an index template (from the context menu in the table and the context menu in the details flyout), make sure it saves correctly

Fixes #57288

… one

elasticmachine · 2020-03-30T14:58:24Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

sebelga · 2020-03-31T17:08:24Z

@elasticmachine merge upstream

alisonelizabeth

Great start with this @sebelga! Code LGTM. Tested locally and everything worked as expected. I did notice a test that was previously skipped failed on the latest run; there might be some more investigation needed there.

alisonelizabeth · 2020-03-31T22:07:40Z

x-pack/plugins/index_management/common/constants/index_templates.ts

+
+/**
+ * Up until the end of the 8.x release cycle we need to support both
+ * V1 and V2 index template formats. This constant keeps track of wether


Suggested change

* V1 and V2 index template formats. This constant keeps track of wether

* V1 and V2 index template formats. This constant keeps track of whether

alisonelizabeth · 2020-03-31T22:11:31Z

x-pack/plugins/index_management/common/types/mappings.ts

+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+// TODO: Move mappings type from Mappings editor here


Do you plan to address this as part of this PR?

I didn't plan on touching the mappings editor in this PR. As I plan on adding test coverage to the mappings editor I will do it then.

alisonelizabeth · 2020-03-31T22:15:02Z

.../plugins/index_management/public/application/components/template_form/steps/use_json_step.ts

    const isValid = validateContent();
    const value = isValid && content.trim() !== '' ? JSON.parse(content) : {};
-    const data = { [prop]: value };
-    return Promise.resolve({ isValid, data });
+    // If no key has been added to the JSON object, we strip it out so an empty objecct is not sent in the request


Suggested change

// If no key has been added to the JSON object, we strip it out so an empty objecct is not sent in the request

// If no key has been added to the JSON object, we strip it out so an empty object is not sent in the request

alisonelizabeth · 2020-03-31T22:31:00Z

x-pack/plugins/index_management/public/application/lib/index_templates.ts

+import { Location } from 'history';
+
+export const getFormatVersionFromQueryparams = (location: Location): 1 | 2 | undefined => {
+  const { v } = parse(location.search.substring(1));


nit: this might be more clear as const { v: version } = ...

alisonelizabeth · 2020-03-31T22:56:19Z

x-pack/plugins/index_management/__jest__/client_integration/template_edit.test.tsx

@@ -128,8 +130,7 @@ describe('<TemplateEdit />', () => {
      expect(nameInput.props().disabled).toEqual(true);
    });

-    // TODO: Flakey test
-    describe.skip('form payload', () => {
+    describe('form payload', () => {


Looks like this failed on the latest run. Might be more work that needs to be done. Related issue: #57288.

sebelga · 2020-04-01T11:51:33Z

Thanks for the review @alisonelizabeth! Indeed the flakiness was still there. I added a new helper in the Testbed utils: waitFor('myTestSubject') which should help us for this type of flakiness related to hooks cycle update. (no need to add an estimated guess in await nextTick(43); // wait 43 millisecond anymore 😄)

I will run the CI on this PR a few times to make sure.

sebelga · 2020-04-01T21:09:41Z

@elasticmachine merge upstream

cjcenizal · 2020-04-02T00:48:41Z

@elasticmachine merge upstream

sebelga · 2020-04-02T11:27:11Z

@elasticmachine merge upstream

sebelga · 2020-04-02T15:54:00Z

@elasticmachine merge upstream

kibanamachine · 2020-04-02T17:39:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 65451f5

History

💚 Build #38138 succeeded 6e95fd7
💛 Build #38095 was flaky 6176123
💚 Build #38081 succeeded d93ac4e
💚 Build #38053 succeeded 6176123
💚 Build #37995 succeeded 6176123

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: Switch to embeddable factory interface with optional override (elastic#61165) fix text error in diagrams (elastic#62101) [Index management] Prepare support Index template V2 format (elastic#61588) Updates dashboard images (elastic#62011) [Maps] remove MapBounds type (elastic#62332) [Uptime] Convert anomaly Jobs name to lowercase to comply with… (elastic#62293) Make d3 place nicely with object values (elastic#62004) EMT-287: update schema with elastic agent id (elastic#62252) [Maps] fix replaceLayerList to handle case where map is not intialized (elastic#62202) Remove support for deprecated xpack.telemetry configurations (elastic#51142) [Uptime] Remove static constant for index name completely (elastic#62256) [APM] E2E: install dependencies for vanilla workspaces (elastic#62178) [backport] Bump to 5.1.3 (elastic#62286) Show server name in Remote Cluster detail panel (elastic#62250) Rename some alert types (elastic#61693) changing duration type to ms, s, m (elastic#62265) [ML] Clear Kibana index pattern cache on creation or form reset. (elastic#62184) Move `src/legacy/server/index_patterns` to data plugin (server) (Remove step) (elastic#61618) [NP] Remove IndexedArray usage in Schemas (elastic#61410)

commit 5107ad7 Merge: 41881bc 408baf2 Author: Elastic Machine <elasticmachine@users.noreply.github.com> Date: Fri Apr 3 00:29:11 2020 -0400 Merge branch 'master' into reporting/np-migration-server-config commit 408baf2 Author: Nathan L Smith <nathan.smith@elastic.co> Date: Thu Apr 2 16:44:29 2020 -0500 Allow Enterprise license for service map (elastic#62371) We were previously only allowing "platinum" and "trial". Change this to allow `hasAtLeast('platinum')` which includes "platinum", "enterprise", and "trial". Fixes elastic#62243. commit d10889a Author: Brandon Morelli <brandon.morelli@elastic.co> Date: Thu Apr 2 14:19:40 2020 -0700 docs: updates to apm agent config (elastic#61893) commit 2ae566e Author: Jen Huang <its.jenetic@gmail.com> Date: Thu Apr 2 13:04:32 2020 -0700 [Ingest] Fix package info request returning 500 (elastic#61712) * Fix 500 package info request error * Fix installed packages page not loading Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> commit 4cb96ee Author: Mikhail Shustov <restrry@gmail.com> Date: Thu Apr 2 21:41:56 2020 +0200 move crypto to server utils (elastic#62344) * move crypto to server utils * fix mocks commit 7039aba Author: Mike Côté <mikecote@users.noreply.github.com> Date: Thu Apr 2 15:25:03 2020 -0400 Start indexing documents by default (elastic#62159) commit 6d8bdf6 Author: Davis Plumlee <56367316+dplumlee@users.noreply.github.com> Date: Thu Apr 2 14:59:50 2020 -0400 [Endpoint] Update host field accordion (elastic#61878) commit d639432 Author: Nicolas Ruflin <spam@ruflin.com> Date: Thu Apr 2 20:46:52 2020 +0200 Add more definitions about Ingest Management (elastic#62049) This should help to explain the different terms. The docs directory was also renamed from epm to ingest_manager. commit bb747ab Author: Stacey Gammon <gammon@elastic.co> Date: Thu Apr 2 14:27:51 2020 -0400 Switch to embeddable factory interface with optional override (elastic#61165) * wip * typescript map embeddable * More updates * Address code review comments and update some usages in SIEM and uptime to the new types * More clean up - carry over some of the SIEM types to maps for render tool tip * fixes * fixes * Address more review comments * fixes * fixes * fix jest test * Fix visualize embeddable * fixes after master merge * Fixes * Prefix variable with name "custom" to make it more obvious * Remove layerList from input state * fixes * Update src/plugins/dashboard/public/embeddable/dashboard_container_factory.tsx Co-Authored-By: Vadim Dalecky <streamich@users.noreply.github.com> * review updates * fixes * update maps readme Co-authored-by: Vadim Dalecky <streamich@users.noreply.github.com> commit 09f1bae Author: Peter Schretlen <peter.schretlen@elastic.co> Date: Thu Apr 2 14:23:09 2020 -0400 fix text error in diagrams (elastic#62101) commit 74d8413 Author: Sébastien Loix <sabee77@gmail.com> Date: Thu Apr 2 20:14:46 2020 +0200 [Index management] Prepare support Index template V2 format (elastic#61588) commit d6587d7 Author: Kaarina Tungseth <kaarina.tungseth@elastic.co> Date: Thu Apr 2 13:03:50 2020 -0500 Updates dashboard images (elastic#62011) commit 41881bc Merge: b965d81 8b31ce0 Author: Elastic Machine <elasticmachine@users.noreply.github.com> Date: Mon Mar 30 18:51:56 2020 -0400 Merge branch 'master' into reporting/np-migration-server-config commit b965d81 Merge: 3997034 9ff8be6 Author: Elastic Machine <elasticmachine@users.noreply.github.com> Date: Mon Mar 30 17:19:48 2020 -0400 Merge branch 'master' into reporting/np-migration-server-config commit 3997034 Merge: 0414532 8d539aa Author: Elastic Machine <elasticmachine@users.noreply.github.com> Date: Fri Mar 27 18:58:21 2020 -0400 Merge branch 'master' into reporting/np-migration-server-config commit 0414532 Merge: 412635e 878ab20 Author: Joel Griffith <joel.griffith@elastic.co> Date: Fri Mar 27 12:48:10 2020 -0700 Merge remote-tracking branch 'upstream/master' into reporting/np-migration-server-config commit 412635e Merge: b1b2bfa 5a537d1 Author: Joel Griffith <joel.griffith@elastic.co> Date: Fri Mar 27 11:32:11 2020 -0700 Merge remote-tracking branch 'upstream/master' into reporting/np-migration-server-config commit b1b2bfa Merge: 60220fb 9a53c08 Author: Joel Griffith <joel.griffith@elastic.co> Date: Thu Mar 26 09:22:52 2020 -0700 Merge remote-tracking branch 'upstream/master' into reporting/np-migration-server-config commit 60220fb Author: Joel Griffith <joel.griffith@elastic.co> Date: Wed Mar 25 10:10:24 2020 -0700 Revert "Revert "[Reporting/New Platform Migration] Use a new config service on server-side (elastic#55882)"" This reverts commit 5c04d9e.

cjcenizal · 2020-04-21T20:09:39Z

@sebelga Can you confirm that I understand the changes introduced by this PR correctly?

The GET /templates route currently fetches only v1 templates. In the future, we intend to update it to fetch both v1 and v2 index templates.
This route has been updated to normalize all fetched templates to the v2 shape before returning them to the client. As part of this normalization step, we add the _kbnMeta property to index templates to differentiate between those fetched from the v1 API and those fetched from the v2 API.
The client has been updated to accommodate the new v2 shape, and no longer supports the v1 shape.
The PUT /templates/{name} and POST /templates/{name} endpoints for creating and updating index templates have been updated to check the version stored in the _kbnMeta property, and either convert to v1 shape or throw an error. In the future, we intend to update it to save both v1 and v2 index templates using their respective APIs.

sebelga · 2020-04-22T07:19:30Z

@cjcenizal this is correct. Apart from the PUT and POST, the DELETE also will need to be updated to support both APIs.

sebelga force-pushed the index-template-v2/table-view branch from d805d58 to ac7efa0 Compare March 27, 2020 13:13

sebelga added 20 commits March 30, 2020 16:07

Add helper method to get template format version (1 or 2)

ec1bd97

Rename interface to make explicit deserialized template vs serialized…

1a91c00

… one

Refactor template list deserialization

c98c1a5

Move utils to common folder

57fb494

Differentiate serialisation to V1 or V2 template format

ac46764

Improve typings and make V2 format the default for client

0054cb0

Fix api integration tests

570e61f

Refactor client code to work with V2 template format (in and out)

4763cef

[useRequest] Fix TS type for data returned

20d24b7

Add constant to keep track of the default index template format

1415fb7

Fix TemplateList and TS issues

a96abc7

Fix component integration tests

b936e31

Set default template name to empty string

c2dc9f0

Add support for v2 format serialization in step review

199763c

Add deserializer for V2 template format (server side)

8b0caf2

Only allow template V1 to be created or edited

7fb48f8

Require version to be provided when fetching template

2a3fc01

Fix template V1 serialization

e3e3ccb

Update API delete template route to receive template format version

d220224

Fix API integration tests

ca0f0ec

sebelga force-pushed the index-template-v2/table-view branch from c2f7104 to ca0f0ec Compare March 30, 2020 14:07

Fix component integration tests

74dfe83

sebelga requested a review from alisonelizabeth March 30, 2020 14:53

sebelga marked this pull request as ready for review March 30, 2020 14:53

sebelga added Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.8 v8.0.0 labels Mar 30, 2020

Merge branch 'master' into index-template-v2/table-view

6afa4d8

alisonelizabeth approved these changes Mar 31, 2020

View reviewed changes

sebelga added 4 commits April 1, 2020 13:42

[TestBed] Add "waitFor" helper for flaxy tests

41efbc3

Make CR suggestions

b0f193c

[Form lib] Fix FormDataProvider initial state

ede49bf

Fix flaxy component integration test

0f12470

sebelga added 2 commits April 1, 2020 18:04

Fix component integration tests

d2ffe6c

Fix flaky test for <LoadMappingsProvider />

4b2c9bb

Merge branch 'master' into index-template-v2/table-view

6176123

Merge branch 'master' into index-template-v2/table-view

d93ac4e

shahzad31 requested a review from a team as a code owner April 2, 2020 06:32

Refactor TestBed.waitFor() and wait up to 30 sec

6e95fd7

Merge branch 'master' into index-template-v2/table-view

65451f5

sebelga merged commit 74d8413 into elastic:master Apr 2, 2020

sebelga deleted the index-template-v2/table-view branch April 2, 2020 18:14

sebelga mentioned this pull request Apr 3, 2020

[7.x] [Index management] Prepare support Index template V2 format (#61588) #62400

Merged

rayafratkina added v7.8.0 and removed v7.8 labels Apr 6, 2020

kibanamachine mentioned this pull request Feb 7, 2022

[IM] Add missing values to the index details flyout #124731

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Index management] Prepare support Index template V2 format #61588

[Index management] Prepare support Index template V2 format #61588

sebelga commented Mar 27, 2020 •

edited

Loading

elasticmachine commented Mar 30, 2020

sebelga commented Mar 31, 2020

alisonelizabeth left a comment •

edited

Loading

alisonelizabeth Mar 31, 2020

alisonelizabeth Mar 31, 2020

sebelga Apr 1, 2020

alisonelizabeth Mar 31, 2020

alisonelizabeth Mar 31, 2020

alisonelizabeth Mar 31, 2020

sebelga commented Apr 1, 2020

sebelga commented Apr 1, 2020

cjcenizal commented Apr 2, 2020

sebelga commented Apr 2, 2020

sebelga commented Apr 2, 2020

kibanamachine commented Apr 2, 2020

cjcenizal commented Apr 21, 2020

sebelga commented Apr 22, 2020

	* V1 and V2 index template formats. This constant keeps track of wether
	* V1 and V2 index template formats. This constant keeps track of whether

	// If no key has been added to the JSON object, we strip it out so an empty objecct is not sent in the request
	// If no key has been added to the JSON object, we strip it out so an empty object is not sent in the request

[Index management] Prepare support Index template V2 format #61588

[Index management] Prepare support Index template V2 format #61588

Conversation

sebelga commented Mar 27, 2020 • edited Loading

Context

Changes in this PR

How to review

elasticmachine commented Mar 30, 2020

sebelga commented Mar 31, 2020

alisonelizabeth left a comment • edited Loading

Choose a reason for hiding this comment

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

sebelga Apr 1, 2020

Choose a reason for hiding this comment

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

alisonelizabeth Mar 31, 2020

Choose a reason for hiding this comment

sebelga commented Apr 1, 2020

sebelga commented Apr 1, 2020

cjcenizal commented Apr 2, 2020

sebelga commented Apr 2, 2020

sebelga commented Apr 2, 2020

kibanamachine commented Apr 2, 2020

💚 Build Succeeded

History

cjcenizal commented Apr 21, 2020

sebelga commented Apr 22, 2020

sebelga commented Mar 27, 2020 •

edited

Loading

alisonelizabeth left a comment •

edited

Loading