Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix mappings for Upgrade Assistant reindexOperationSavedObjectType. #71710

Conversation

cjcenizal
Copy link
Contributor

@cjcenizal cjcenizal commented Jul 14, 2020

Fixes #71642.

Release note

Some users who upgrade to 7.8.0 won't be able to start Kibana if previous upgrades resulted in errors with very large error messages. In this situation, they'll see a Kibana log error like this:

Document contains at least one immense term in field="upgrade-assistant-reindex-operation.errorMessage" (whose UTF8 encoding is longer than the max length 32766)

This fixes the problem so that users won't see this error and will be able to start Kibana as usual.

Steps to test

To test, you need to first reproduce the error in 7.8.0 and then verify that this change fixes it. Download ES 7.7.0, Kibana 7.7.0, ES 7.8.0, and Kibana 7.8.0.

  1. Start 7.7.0.
  2. Index a mock document into .kibana_1. Make sure to use the linked one because it has an error message that's large enough to produce the error.
  3. Stop ES and Kibana.
  4. Copy elasticsearch-7.7.0/data over to elasticsearch-7.8.0/data.
  5. Start 7.8.0 and you'll see Kibana fail to start with the fatal error below.
  6. Create a new branch off 7.8 and cherry-pick the changes from this PR.
  7. Use yarn es snapshot to install ES and kill the process. Copy elasticsearch-7.7.0/data to kibana/.es/7.8.1/data.
  8. Disable security in both ES and Kibana by adding xpack.security.enabled: false to their respective config files.
  9. Start ES with kibana/.es/7.8.1/bin/elasticsearch and start Kibana with yarn start. Kibana will start normally.
  10. Verify the mock document you indexed exists with GET .kibana_1/_doc/upgrade-assistant-reindex-operation:fb36e570-bca7-11e9-bbfd-f767236c4a9b.

Fatal error

image

To-do

In a subsequent PR, I'll add a test that verifies that we can ingest the mock document without any errors. Addressed in #72347.

@cjcenizal cjcenizal added regression release_note:fix v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more Feature:Upgrade Assistant v7.9.0 v7.8.1 labels Jul 14, 2020
@cjcenizal cjcenizal requested a review from a team as a code owner July 14, 2020 17:57
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui (Team:Elasticsearch UI)

@@ -15,13 +15,25 @@ export const reindexOperationSavedObjectType: SavedObjectsType = {
mappings: {
properties: {
reindexTaskId: {
type: 'keyword',
type: 'text',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that I made these other changes to regain parity with how the dynamic mapping was originally mapping these fields.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the default mappings for a dynamic field detected as being text, but is there any value in maintaining parity? With this mapping two fields will be added, text and keyword. It seems unlikely that we would do a match query on the keyword, so maybe having only a text field (with index: false) would add less indexing overhead without removing any validation by ES.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this and for suggesting a solution! I decided to maintain parity to minimize the risk of introducing another regression as much as possible. I have no idea if there's any value beyond that for maintaining parity (haven't looked into it yet). I agree that many of these fields probably don't need to be indexed as keyword (or possibly at all). #64547 tracks the work of updating these mappings to minimize the number of fields we're indexing and I'll keep your suggestion in mind when I begin working on it.

type: 'keyword',
ignore_above: 256,
},
},
},
reindexTaskPercComplete: {
type: 'float',
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was originally being dynamically mapped as long but I believe that was an error, since we expect floating point values where this field is consumed.

Copy link
Contributor

@alisonelizabeth alisonelizabeth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Tested using the instructions provided and was able to reproduce the error and verify the changes.

Error:
Screen Shot 2020-07-14 at 2 31 11 PM

With changes:
Screen Shot 2020-07-14 at 3 01 47 PM

@kibanamachine
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Upgrade Assistant regression release_note:fix Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.8.1 v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Immense Upgrade Assistant error messages are indexed, preventing Kibana from starting
5 participants