Security index is created with custom mappings/settings resulting in upgrade issues

[romain-chanu]

Elasticsearch Version

7.17.4

Installed Plugins

No response

Java Version

bundled

OS Version

Deployment in ESS

Problem Description

1. Problem

1. This has been observed in Elasticsearch Service (ESS) and can be reproduced with the steps provided below.

2. User upgrades from version 7.17.4 to version 8.2.x and the Kibana upgrade fails with the following error message:

The role mapping elastic-cloud-sso-kibana-do-not-change could not be configured.

3. Deep-dive investigation reveals that the .security index was created with custom mappings/settings:

• Legacy index templates were defined, using wildcards (* or .*) and applied custom mappings/settings.
• User ran the Kibana Upgrade Assistant (i.e migrate system indices) while these legacy index templates were in place.

2. Workaround

Reindexing .security-6-reindexed-for-8 into a new index (e.g .security-7-reindexed-for-8) is the only workaround. However, depending on the setup, steps vary.

2.1 On-premise setup (self-managed)

1. (Recommended but optional) : take a snapshot

2. In each Elasticsearch node, add the below role definition in the roles.yml file:

system-indices-admin:
  cluster: [ 'all' ]
  indices:
    - names: [ '*' ]
      privileges: ['all']
      allow_restricted_indices: true

3. In each Elasticsearch node, create a file-based user and associate it with the above role:

bin/elasticsearch-users useradd fileuser
bin/elasticsearch-users roles fileuser -a system-indices-admin

4. Set the is_hidden property on the existing .security alias:

curl -u fileuser:<password> -XPOST "<ELASTICSEARCH_ENDPOINT>/_aliases?pretty" --json '
{
  "actions": [
    {
      "add": {
        "index": ".security-6-reindexed-for-8",
        "alias": ".security",
        "is_hidden": true
      }
    }
  ]
}
'

5. Reindex .security-6-reindexed-for-8 into a new index .security-7-reindexed-for-8

curl -u fileuser:<password> -XPOST "<ELASTICSEARCH_ENDPOINT>/_reindex?pretty" --json '
{
  "source": {
    "index": ".security-6-reindexed-for-8"
  },
  "dest": {
    "index": ".security-7-reindexed-for-8"
  }
}
'

6. Check the mappings for .security-7-reindexed-for-8 and verify the presence of _meta field.

curl -u fileuser:<password> -XGET "<ELASTICSEARCH_ENDPOINT>/.security-7-reindexed-for-8/_mapping"

7. If the reindexing operation succeeded:

a) Delete the old security index:

curl -u fileuser:<password> -XDELETE "<ELASTICSEARCH_ENDPOINT>/.security-6-reindexed-for-8"

b) In each Elasticsearch node, delete the file-based user:

bin/elasticsearch-users userdel fileuser

c) In each Elasticsearch node, remove the system-indices-admin role definition in the roles.yml file.

2.2 Elastic Cloud Enterprise / Elasticsearch Service

1. (Recommended but optional) : take a snapshot

2. Using the API Console, execute the below steps:

a) Set the is_hidden property on the existing .security alias:

POST _aliases
{
  "actions": [
    {
      "add": {
        "index": ".security-6-reindexed-for-8",
        "alias": ".security",
        "is_hidden": true
      }
    }
  ]
}

b) Reindex .security-6-reindexed-for-8 in a new index .security-7-reindexed-for-8

POST _reindex
{
  "source": {
    "index": ".security-6-reindexed-for-8"
  },
  "dest": {
    "index": ".security-7-reindexed-for-8"
  }
}

c) Check the mappings for .security-7-reindexed-for-8 and verify the presence of _meta field.

GET .security-7-reindexed-for-8/_mapping

d) If the reindexing operation succeeded, delete the old security index:

DELETE .security-6-reindexed-for-8

3. Related issues

It is in a way similar to #86801 with the below pull requests that should address and prevent this kind of scenarios:

• Don't migrate system indices that match user templates #87262
• System indices ignore all user templates #87260

Can we take a second look at this scenario and ensure that the above pull requests cover it?

4. Steps to Reproduce

1. Create a cluster version 6.8.23

2. Upgrade to version 7.17.4

3. Install the following legacy templates:

PUT _template/template_a
{
  "order": 11,
  "index_patterns": [
    ".*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": null
      }
    }
  },
  "mappings": {},
  "aliases": {}
}

PUT _template/template_b
{
  "order": 10,
  "index_patterns": [
    "*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "roll-delete"
      },
      "refresh_interval": "5s",
      "number_of_shards": "1",
      "number_of_replicas": "0"
    }
  },
  "mappings": {},
  "aliases": {}
}

PUT _template/template_c
{
  "order": 0,
  "index_patterns": [
    "*"
  ],
  "settings": {},
  "mappings": {
    "dynamic_templates": [
      {
        "message_field": {
          "path_match": "message",
          "mapping": {
            "norms": false,
            "type": "text"
          },
          "match_mapping_type": "string"
        }
      },
      {
        "string_fields": {
          "mapping": {
            "norms": false,
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "match_mapping_type": "string",
          "match": "*"
        }
      }
    ],
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "docMeta": {
        "dynamic": true,
        "type": "object",
        "properties": {
          "latitude": {
            "type": "float",
            "doc_values": true
          },
          "location": {
            "type": "geo_point"
          },
          "longitude": {
            "type": "float",
            "doc_values": true
          }
        }
      }
    }
  },
  "aliases": {}
}

4. Go to Kibana Upgrade Assistant and click migrate system indices

5. Check the Elasticsearch logs:

a) .security-6-reindexed-for-8 was created with the 3 above templates applied to it:

[instance-0000000000] [.security-6-reindexed-for-8] creating index, cause [migrate-system-index], templates [template_a, template_b, template_c], shards [1]/[0]

b) Shortly after, the following messages can be observed:

[instance-0000000000] Missing _meta field in mapping [_doc] of index [.security]	
...
...
instance-0000000000] failed to notify ClusterStateListener
java.lang.IllegalStateException: Cannot read security-version string in index .security

6. Upgrade to version 8.2.x

a) Elasticsearch upgrade succeeds but Kibana upgrade fails with the following error in the Cloud UI:

The role mapping elastic-cloud-sso-kibana-do-not-change could not be configured.

b) User can observe the below error message in the Kibana logs:

Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: status_exception: [status_exception] Reason: Cluster state has not been recovered yet, cannot write to the [null] index

Logs (if relevant)

No response

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[grcevski]

This issue should be resolved by this fix that went in 7.17.5 #87933