Immense Upgrade Assistant error messages are indexed, preventing Kibana from starting

[cjcenizal]

In #64332 we defined the mappings used by the Upgrade Assistant saved objects explicitly instead of depending upon dynamic: true. In particular this line defines errorMessage as type keyword.

This change shipped with 7.8 and causes Kibana to stop working when upgraded to 7.8. These error messages show up in the logs:

Document contains at least one immense term in field="upgrade-assistant-reindex-operation.errorMessage" (whose UTF8 encoding is longer than the max length 32766)

I need to verify this, but based on feedback from people using earlier versions, it looks like the new type is incorrect. Previously, the type was defined like this:

{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

The use of ignore_above meant that error messages longer than 256 bytes would not be indexed as keyword. Without this safeguard, I believe Upgrade Assistant generates and indexes very large error messages, which causes problems when Kibana tries to start up.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[rudolf]

@cjcenizal can you document a workaround here should anyone stumble upon this issue?

[cjcenizal]

Thanks for the reminder @rudolf!

Workaround

1. Stop the Kibana instance.
2. Check the current alias for the .kibana index. This should point to .kibana_X index.
3. Check the list of indices and you should see that a .kibana_X+1 exists (the one created during the upgrade and also reported in the Kibana logs). Delete this index:
4. DELETE .kibana_X+1
5. Backup .kibana_X index:

POST _reindex
{
  "source": {
    "index": ".kibana_X"
  },
  "dest": {
    "index": ".kibana_X_backup"
  }
}

5. Delete all documents containing failed in the upgrade-assistant-reindex-operation.errorMessage field from .kibana_X index:

POST .kibana_X/_delete_by_query
{
  "query": {
    "match": {
      "upgrade-assistant-reindex-operation.errorMessage": "failed"
    }
  }
}

6. Resume the instance. Kibana instance should be up and running.