[docs] Resolving failed Kibana upgrade migrations #80999

rudolf · 2020-10-19T14:42:13Z

Summary

Documentation for resolving failed Kibana upgrade migrations. Related to #80283

This provides instructions for retrying failed upgrade migrations, or rolling back to a previous migration using snapshots. Using snapshots makes this process significantly simpler and much harder to accidentally remove the wrong index or forget to add the correct alias etc etc.

TODO:

Does not touch on resolving the root cause of a migration failure. Sometimes failures happen because of transient problems, but if the cause is persistent (ES out of disk space, ES cluster doesn't have enough heap, a document throws an exception while being transformed) then retrying the migration won't resolve the problem.
Problems caused by the unsupported config value xpack.tasks.index: .tasks
Problems caused by index templates which change the mappings or the refresh_interval index setting for newly created indices.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-10-20T19:40:26Z

Pinging @elastic/kibana-platform (Team:Platform)

lcawl · 2020-10-20T22:10:27Z

@elasticmachine merge upstream

pgayvallet · 2020-10-21T09:55:37Z

@rudolf I see TODOs in the summary. Is this ready for review?

ppf2 · 2020-10-21T19:29:42Z

docs/setup/upgrade.asciidoc

-* Shut down all {kib} nodes. Running more than one {kib} version against the
-  same Elasticseach index is unsupported. If you upgrade while older {kib} nodes are
-  running, the upgrade can fail.
+* Shut down all {kib} instances. Running more than one {kib} version against


This bullet point about shutting down all Kibana instances (i.e., maintenance window/downtime required) is often missed. It might be helpful to highlight this in a warning statement and call out the expected downtime.

ppf2 · 2020-10-21T19:40:32Z

docs/setup/upgrade/upgrade-migrations.asciidoc

-the appropriate {kib} index. When you have the previous version running again,
-delete the more recent `.kibana_N` index or indices so that future upgrades are
-based on the current {kib} index. You must restart {kib} to re-trigger the migration.
+In order to rollback after a failed upgrade migration, the saved object indices might also have to be rolled back to be compatible with the previous {kibana} version. 


Do we have any guidelines on how the user can determine if there are objects that need to be rolled back to be compatible with previous Kibana version? (i.e., if they have to follow the "Rolling back to a previous version of ..." section in addition to the "Resolving migration failures" section)

yeah that would be useful to flesh out. In short, if you get unhandled exceptions it's worth retrying the migration to see if it would succeed. If a specific document fails to migrate retrying won't help, so you can either delete the document and retry, or rollback.

I've added a "preventing migration failures" section. In the rollback instructions I now added: if you tried the preventing migration failures and resolving migration failures and still can't upgrade you might choose to rollback.

docs/setup/upgrade/upgrade-migrations.asciidoc

ppf2 · 2020-10-21T20:06:06Z

docs/setup/upgrade.asciidoc

 [float]
 [[upgrade-before-you-begin]]
 === Before you begin

+WARNING: {kib} automatically runs upgrade migrations when required. To roll back to an earlier version in case of an upgrade failure, you **must** have a backup snapshot available. Use <<snapshot-repositories, Snapshot and Restore>> to back up {kib} data by targeting the `.kibana*` indices. For more information see <<upgrade-migrations, upgrade-migrations>>.
+
 Before you upgrade {kib}:


Thoughts on adding another item here to prevent custom index templates from interfering with .kibana* indices?
i.e., remove any custom index template that might accidentally "cover" .kibana* indices. This can help avoid saved objects migration issues due to custom index templates causing invalid/unexpected mappings.

Added the "preventing migration failures" section and linked to it from here.

…out stopping all instances in specific upgrade steps

…ation failures

rudolf · 2020-10-22T12:03:50Z

@rudolf I see TODOs in the summary. Is this ready for review?

@pgayvallet I've addressed the remaining todo's now.

rudolf · 2020-10-26T08:43:46Z

@elasticmachine merge upstream

pgayvallet

LGTM. That's some great doc improvements!

pgayvallet · 2020-10-26T10:10:19Z

docs/setup/upgrade/upgrade-migrations.asciidoc

+* `.kibana_N`, or if set, the `kibana.index` configuration setting
+* `.kibana_task_manager_N`, or if set, the `xpack.tasks.index` configuration setting


NIT: I would replace .kibana_N with .kibana_{N} and .kibana_N+1 with .kibana_{N+1}, but this is a developer point of view, not sure this is more readable for the target audience.

docs/setup/upgrade/upgrade-migrations.asciidoc

* Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc

* Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc

…astic#81810) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc

* master: (87 commits) [Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81778) [i18n] add get_kibana_translation_paths tests (elastic#81624) [UX] Fix search term reset from url (elastic#81654) [Lens] Improved range formatter (elastic#80132) [Resolver] `SideEffectContext` changes, remove `@testing-library` uses (elastic#81077) [Time to Visualize] Update Library Text with Call to Action (elastic#81667) [docs] Resolving failed Kibana upgrade migrations (elastic#80999) [ftr/menuToggle] provide helper for enhanced menu toggle handling (elastic#81709) [Task Manager] adds basic observability into Task Manager's runtime operations (elastic#77868) Doc changes for stack management and grouped feature privileges (elastic#80486) Added functional test for alerts list filters by status, alert type and action type. Did a code refactoring and splitting for alerts tests. (elastic#81422) [Security Solution][Endpoint][Admin] Malware Protections Notify User Version (elastic#81415) Revert "[Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81390)" [Maps] Use default format when proxying EMS-files (elastic#79760) [Discover] Deangularize context.html (elastic#81442) Use the displayName property in default editor (elastic#73311) adds bug label to Bug report for Security Solution template (elastic#81643) [ML] Transforms: Remove index field limitation for custom query. (elastic#81467) [Actions] Adding `hasAuth` to Webhook Configuration to avoid confusing UX (elastic#81390) [Task Manager] Mark task as failed if maxAttempts has been met. (elastic#80681) ...

…81814) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc

…81813) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…astic#81810) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc

) (#81882) * [docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc * Adapt upgrade migration documentation for <= 7.3

… (elastic#81810) (elastic#81882) * [docs] Resolving failed Kibana upgrade migrations (elastic#80999) (elastic#81810) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc * Adapt upgrade migration documentation for <= 7.3 # Conflicts: # docs/setup/upgrade.asciidoc

) (#81882) (#81985) * [docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) * Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # docs/setup/upgrade.asciidoc # docs/setup/upgrade/upgrade-migrations.asciidoc # docs/setup/upgrade/upgrade-standard.asciidoc * Adapt upgrade migration documentation for <= 7.3 # Conflicts: # docs/setup/upgrade.asciidoc

* Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* Resolving failed Kibana upgrade migrations * Move warning against rolling upgrades into upgrade-standard and call out stopping all instances in specific upgrade steps * Add preventing migration failures section * Add incompatible xpack.tasks.index: .tasks setting to preventing migration failures * Fix link Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

rudolf added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Saved Objects docs labels Oct 19, 2020

rudolf self-assigned this Oct 19, 2020

rudolf added this to Pending Review in kibana-core [DEPRECATED] via automation Oct 19, 2020

rudolf force-pushed the so-migration-docs branch from 4310b58 to d64aab8 Compare October 20, 2020 19:28

Resolving failed Kibana upgrade migrations

a958a2f

rudolf force-pushed the so-migration-docs branch from d64aab8 to a958a2f Compare October 20, 2020 19:31

rudolf marked this pull request as ready for review October 20, 2020 19:40

rudolf mentioned this pull request Oct 20, 2020

[DOC] Clarify compatibility of Saved Objects across versions #63090

Merged

1 task

Merge branch 'master' into so-migration-docs

1dc9c36

lcawl added release_note:skip Skip the PR/issue when compiling release notes v8.0.0 labels Oct 20, 2020

ppf2 reviewed Oct 21, 2020

View reviewed changes

docs/setup/upgrade/upgrade-migrations.asciidoc Outdated Show resolved Hide resolved

ppf2 reviewed Oct 21, 2020

View reviewed changes

rudolf added 3 commits October 22, 2020 11:10

Move warning against rolling upgrades into upgrade-standard and call …

e6f0c2d

…out stopping all instances in specific upgrade steps

Add preventing migration failures section

67997dd

Add incompatible xpack.tasks.index: .tasks setting to preventing migr…

db501d5

…ation failures

Merge branch 'master' into so-migration-docs

171c558

rudolf added the v7.10.0 label Oct 26, 2020

pgayvallet approved these changes Oct 26, 2020

View reviewed changes

rudolf added 2 commits October 27, 2020 13:23

Merge branch 'master' into so-migration-docs

3007ecc

Fix link

569f1ee

rudolf mentioned this pull request Oct 27, 2020

[7.5] [docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) #81813

Merged

rudolf mentioned this pull request Oct 27, 2020

[7.4] [docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) #81814

Merged

rudolf mentioned this pull request Oct 28, 2020

[7.3] [docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) #81882

Merged

This was referenced Oct 29, 2020

[7.2][docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) (#81882) #81985

Merged

[7.1] [docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) (#81882) #81986

Closed

rudolf mentioned this pull request Oct 29, 2020

[7.0][docs] Resolving failed Kibana upgrade migrations (#80999) (#81810) (#81882) #81988

Closed

rudolf mentioned this pull request Nov 27, 2020

Doc: Kibana upgrade failures resulting in alias not pointing to the latest .kibana* (and related) indices #80283

Closed

rudolf mentioned this pull request Jan 26, 2021

[7.x] [docs] Resolving failed Kibana upgrade migrations (#80999) #89294

Merged

rudolf mentioned this pull request Jan 26, 2021

[7.11] [docs] Resolving failed Kibana upgrade migrations (#80999) #89295

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[docs] Resolving failed Kibana upgrade migrations #80999

[docs] Resolving failed Kibana upgrade migrations #80999

rudolf commented Oct 19, 2020 •

edited

elasticmachine commented Oct 20, 2020

lcawl commented Oct 20, 2020

pgayvallet commented Oct 21, 2020

ppf2 Oct 21, 2020

ppf2 Oct 21, 2020

rudolf Oct 22, 2020

rudolf Oct 22, 2020

ppf2 Oct 21, 2020

rudolf Oct 22, 2020

rudolf commented Oct 22, 2020

rudolf commented Oct 26, 2020

pgayvallet left a comment

pgayvallet Oct 26, 2020

		* `.kibana_N`, or if set, the `kibana.index` configuration setting
		* `.kibana_task_manager_N`, or if set, the `xpack.tasks.index` configuration setting

[docs] Resolving failed Kibana upgrade migrations #80999

[docs] Resolving failed Kibana upgrade migrations #80999

Conversation

rudolf commented Oct 19, 2020 • edited

Summary

Checklist

For maintainers

elasticmachine commented Oct 20, 2020

lcawl commented Oct 20, 2020

pgayvallet commented Oct 21, 2020

ppf2 Oct 21, 2020

Choose a reason for hiding this comment

ppf2 Oct 21, 2020

Choose a reason for hiding this comment

rudolf Oct 22, 2020

Choose a reason for hiding this comment

rudolf Oct 22, 2020

Choose a reason for hiding this comment

ppf2 Oct 21, 2020

Choose a reason for hiding this comment

rudolf Oct 22, 2020

Choose a reason for hiding this comment

rudolf commented Oct 22, 2020

rudolf commented Oct 26, 2020

pgayvallet left a comment

Choose a reason for hiding this comment

pgayvallet Oct 26, 2020

Choose a reason for hiding this comment

rudolf commented Oct 19, 2020 •

edited