ILM: restrict customization of alias to suffix. #3905

simitt · 2020-06-18T10:28:20Z

Motivation/summary

Change how rollover aliases can be configured to only allow customizing a suffix. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier.

Checklist

~~- [] I have signed the Contributor License Agreement.~~

I have updated CHANGELOG.asciidoc

I have considered changes for:

documentation
~~- [ ] logging (add log lines, choose appropriate log selector, etc.)~~
~~- [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)~~
automated tests (add tests for the code changes, all unit tests pass locally)
~~- [ ] telemetry~~
Elasticsearch Service (https://cloud.elastic.co)
~~- [ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)~~
~~- [ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)~~

How to test these changes

Configure index_suffix for event types in apm-server.ilm.setup.mappings and check that templates are created accordingly.

Related issues

Follow up on #3826
closes #3895

Change how rollover aliases can be configured to only allow customizingi a suffix.This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826 closes elastic#3895

apmmachine · 2020-06-18T11:13:57Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #3905 updated]
Start Time: 2020-06-23T07:50:13.010+0000
Duration: 45 min 59 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3217
Skipped	147
Total	3364

Steps errors

Expand to view the steps failures

Name: Compress
- Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage
- Duration: 0 min 0 sec
- Start Time: 2020-06-23T08:04:48.376+0000
- log
Name: Compress
- Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
- Duration: 0 min 0 sec
- Start Time: 2020-06-23T08:16:29.237+0000
- log
Name: Test Sync
- Description: ./script/jenkins/sync.sh
- Duration: 2 min 29 sec
- Start Time: 2020-06-23T07:59:51.203+0000
- log

jalvz · 2020-06-18T11:28:11Z

idxmgmt/ilm/config.go

 		if err != nil {
-			return Config{}, errors.Wrap(err, "variable part of rollover_alias cannot be resolved")
+			return Config{}, errors.Wrap(err, "variable part of index suffix cannot be resolved")


variable part of index suffix ==> isn't this just the "index"?

Technically yes, but everything except the suffix is fixed by us. The error message reuses the term index suffix as this is what the users can configure and are used to from the apm-server.yml.

jalvz · 2020-06-18T11:47:44Z

apm-server.yml

-      # need to be changed accordingly to match the created indices
+      # The configured event types, policies and index suffixes will be merged with the default setup.
+      # Only configure the mappings that should be customized.
+      # Index names will always start with `apm-%{[observer.version]}-%{[event.type]}, e.g. apm-7.9.0-span.


I find the phrasing in all this section a bit confusing/convoluted.
Can't we just say that index are named like apm-%{[observer.version]}-%{[event.type]}-{mapping.index_suffix}, where {mapping.index_suffix} is the setting bellow and it is optional?
Or something like that, I don't know. Just an idea. Or maybe seek assistance from professionals...

Also confused about "Only configure the mappings that should be customized": not sure if that is trivial or if I am missing something.
Also the last note, "ensure they are aligned with the ILM indices": what does "aligned" mean? what are "ILM indices" (aren't they all ILM)? etc.

I like Juan's idea here. What about something like this?

Customized mappings will be merged with the default setup, so you only need to configure mappings for the event types, policies, and index suffixes that you want to customize. Indices are named in this way: `apm-%{[observer.version]}-%{[event.type]}-{index_suffix}`, e.g., apm-7.9.0-span-custom*. The `index_suffix` is optional.

Thank you both, I changed the wording.

bmorelli25

I have a few docs suggestions.

docs/ilm-reference.asciidoc

docs/ilm.asciidoc

bmorelli25 · 2020-06-23T05:38:35Z

apm-server.yml

-      # need to be changed accordingly to match the created indices
+      # The configured event types, policies and index suffixes will be merged with the default setup.
+      # Only configure the mappings that should be customized.
+      # Index names will always start with `apm-%{[observer.version]}-%{[event.type]}, e.g. apm-7.9.0-span.


I like Juan's idea here. What about something like this?

Customized mappings will be merged with the default setup, so you only need to configure mappings for the event types, policies, and index suffixes that you want to customize. Indices are named in this way: `apm-%{[observer.version]}-%{[event.type]}-{index_suffix}`, e.g., apm-7.9.0-span-custom*. The `index_suffix` is optional.

Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

…uffix

Change how rollover aliases can be configured to only allow customizing a suffix. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on elastic#3826 closes elastic#3895 Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

Change how rollover aliases can be configured to only allow customizing a suffix. This change intends to make the ILM setup less error prone and to make the transition to the new index_templates and data streams easier. Follow up on #3826 closes #3895 Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

This reverts commit fc3b62a.

axw · 2020-07-21T05:51:16Z

Verified with BC2. Procedure:

Created stack with apm-it
Added apm-server config -E apm-server.ilm.setup.mapping=[{event_type: 'transaction', index_suffix: 'suffolks'}]
Started apm-server
Checked that (only) the transaction index has "-suffolks" appended:

green open apm-7.9.0-transaction-suffolks-000001 PnTAEchiQOy9aMK5wXSZXw 1 0   10 0   203kb   203kb
green open apm-7.9.0-metric-000001               5_5GltpHSrCVbTFuCe_NpA 1 0  107 0 137.8kb 137.8kb
green open apm-7.9.0-profile-000001              EaNGAmL9SyKfBjfHxKWaKg 1 0  388 0 753.5kb 753.5kb
green open apm-7.9.0-error-000001                EFqjo-qTSnuWLH-9vRSJLw 1 0    0 0    208b    208b
green open apm-7.9.0-span-000001                 su3VI2PQTIGUITaRFFHw_Q 1 0 2174 0 667.5kb 667.5kb
green open apm-7.9.0-onboarding-2020.07.21       zcJLiM4dRxGmGmhHMsoEUA 1 0    5 0    33kb    33kb

Checked that the transaction index has the expected ILM policy and rollover alias:

  "apm-7.9.0-transaction-suffolks-000001" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "apm-rollover-30-days",
          "rollover_alias" : "apm-7.9.0-transaction-suffolks"
        },
        ...

I initially started apm-server before updating config, which caused the template to be created without the suffix, restarted the server, and had two templates. The default template name ended up matching the created indices, so I had to delete everything and restart the server again.

simitt requested review from bmorelli25 and a team June 18, 2020 11:35

simitt added review test-plan v7.9.0 labels Jun 18, 2020

jalvz approved these changes Jun 18, 2020

View reviewed changes

axw approved these changes Jun 23, 2020

View reviewed changes

bmorelli25 reviewed Jun 23, 2020

View reviewed changes

simitt and others added 4 commits June 23, 2020 09:32

Apply suggestions from code review

98bb762

Co-authored-by: Brandon Morelli <bmorelli25@gmail.com>

rephrase docs according to code review

f6f8a00

Add changelog

7d6ab89

Merge remote-tracking branch 'elastic/master' into ilm-customizable-s…

c0ff0f9

…uffix

simitt merged commit fc3b62a into elastic:master Jun 23, 2020

simitt deleted the ilm-customizable-suffix branch June 23, 2020 09:01

simitt mentioned this pull request Jun 23, 2020

[7.x] ILM: restrict customization of alias to suffix. (#3905) #3911

Merged

jalvz added a commit to jalvz/apm-server that referenced this pull request Jul 2, 2020

Revert "ILM: restrict customization of alias to suffix. (elastic#3905)"

652bf35

This reverts commit fc3b62a.

axw mentioned this pull request Jul 21, 2020

Ilm alias #3826

Merged

4 tasks

axw added the test-plan-ok label Jul 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ILM: restrict customization of alias to suffix. #3905

ILM: restrict customization of alias to suffix. #3905

simitt commented Jun 18, 2020

apmmachine commented Jun 18, 2020 •

edited

Build stats

Test stats 🧪

jalvz Jun 18, 2020

simitt Jun 23, 2020 •

edited

jalvz Jun 18, 2020

bmorelli25 Jun 23, 2020

simitt Jun 23, 2020

bmorelli25 left a comment

bmorelli25 Jun 23, 2020

axw commented Jul 21, 2020

ILM: restrict customization of alias to suffix. #3905

ILM: restrict customization of alias to suffix. #3905

Conversation

simitt commented Jun 18, 2020

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Jun 18, 2020 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

jalvz Jun 18, 2020

Choose a reason for hiding this comment

simitt Jun 23, 2020 • edited

Choose a reason for hiding this comment

jalvz Jun 18, 2020

Choose a reason for hiding this comment

bmorelli25 Jun 23, 2020

Choose a reason for hiding this comment

simitt Jun 23, 2020

Choose a reason for hiding this comment

bmorelli25 left a comment

Choose a reason for hiding this comment

bmorelli25 Jun 23, 2020

Choose a reason for hiding this comment

axw commented Jul 21, 2020

apmmachine commented Jun 18, 2020 •

edited

simitt Jun 23, 2020 •

edited