Kibana 6.5.x does not fully recover when kibana indices are deleted during operation

[jpcarey]

Kibana version: 6.5.x

Describe the bug: If you delete .kibana* while Kibana is running, Kibana will create a new .kibana index and place the config:6.5.2 document into this index. The space:default document is never recreated, which will then return a 400 when you try to visit Timelion, Canvas, Machine Learning, Infrastructure, Logs, APM, Graph, or Monitoring. These pages seem to try to load the base domain (https://{{server_name}}:5601/). Discover, Visualize, Dashboard, Dev Tools, and Management are not impacted, as they seem to load the domain + URI (ex. https://{{server_name:5601/app/kibana#/management). You must restart Kibana to fix this issue (or manually create the space:default document)

Steps to reproduce:

1. Startup a working Kibana and Elasticsearch 6.5.2 environment
2. Navigate to the Dev Console and use DELETE .kibana*
3. Try to load Timelion
4. This will try to load https://kibana.local:5601/, which will now produce a 400 Bad Request with a response body: {"statusCode":400,"error":"Bad Request","message":"[illegal_argument_exception] application privileges must refer to at least one resource"}

Provide logs and/or server output (if relevant):
log [23:08:18.354] [error][spaces] Unable to navigate to space "default", redirecting to Space Selector. Error: Saved object [space/default] not found

[elasticmachine]

Pinging @elastic/kibana-security

[jrplusone]

how do i manually create the space:default document?

[kobelb]

how do i manually create the space:default document?

If you stop the Kibana server and restart it, the default space document will be created.

[Bachib]

After stumbling on this issue for a few hours, I found a solution.
Works in Kibana 7.2. After deleting the kibana index with DELETE localhost:9200/.kibana_1, you need to :

1. Create the .kibana_1 index
   PUT localhost:9200/.kibana_1

{
        "aliases": {
            ".kibana": {}
        },
        "mappings": {
        }
}

2. Create the default:space. To do so:

POST localhost:9200/.kibana_1/_doc/space:default

{

    "space": {
        "name": "Default",
        "description": "This is your default space!",
        "color": "#00bfb3",
        "disabledFeatures": [],
        "_reserved": true
    },
    "type": "space",
    "references": [],
    "migrationVersion": {
        "space": "6.6.0"
    }
}

localhost:9200 being the Elasticsearch instance

[jpcarey]

@Bachib you might run into some issues down the road, since you explicitly created the .kibana_1 index with no mappings. I'm not entirely sure what Kibana will do in this case, but leaving elasticsearch to do automatic mapping of data will likely cause issues.

[janbernhardt]

I get the same error message, but my root cause is different. When I login as "elastic" super-user all looks fine. But when I try to login with my personal ldap account (with super-user role mappings), I get this error message. So for me the ldap integration seems to be unstable. (v7.5.1)

[efernandes-dev-ops]

What is the best way to get around this? We just experienced the same issue where a policy was created and applied to remove kibana indices, which then resulted in kibana going down and error being throw "application privileges must refer to at least one resource"

[kobelb]

@efernandes-ANDigital if you restart the Kibana server, on startup it will insert the ES documents that the policy potentially deleted.

[danksim]

Hm... What if downtime is not an option? @kobelb

[kobelb]

Hm... What if downtime is not an option? @kobelb

If downtime isn't an option, I'd recommend not deleting the .kibana index while Kibana is running. What is it that you're looking to accomplish by deleting the .kibana index?

[danksim]

I don't want to delete the .kibana index. I never have it in the first place. When I start up Kibana (v6.8.2), it runs some migration and creates the .kibana_2 index, reindexes .kibana to .kibana_1, migrates saved objects from .kibana_1 to .kibana_2, then the .kibana alias is pointed to this new .kibana_2 index.

I am trying to restore the .kibana_1 index (that is a snapshot of my prod cluster WITHOUT XPACK) to my dev cluster WITH XPACK to sync the saved objects. (I am in the process of adding xpack security, dev cluster first)

Once I do the restore (tried both Curator and manually), I overwrite the existing .kibana_1 (that was abandoned in favor of .kibana_2). Then I point the .kibana alias to this restored .kibana_1. At this point, I am missing the default:space document (doesn't exist in .kibana_1 index because the prod cluster does not have XPACK?) and once i kubectl port-forward, and browse to https://localhost:5601, it loads an error page saying 400 BAD REQUEST - [illegal_argument_exception] application privileges must refer to at least one resource. Which led me to this very issue.

What can I do at this point? Do I need to create this default:space document manually? If so, how? I have tried @Bachib's POST localhost:9200/.kibana_1/_doc/space:default but it throws a strict_dynamic_mapping_exception saying mapping set to strict, dynamic introduction of [space] within [doc] is not allowed.

I have a thread going here as well...

[kobelb]

Hey @danksim, this issue is tracking the behavior of when the .kibana index is explicitly deleted when Kibana is running. We reserve GitHub issues for confirmed bugs or feature requests.

Thanks for opening the discuss thread, we use discuss for questions like yours. However, your discuss thread was opened in the Elasticsearch topic. I've moved it to the Kibana topic, as it's more so a Kibana question than an Elasticsearch one.

[joggerjoel]

I can't believe creating a new fresh system 7.7.2 Kibana has the same issue...

I did this to resolve it:

curl -k -u elastic:elastic -X POST "https://192.168.1.132:9200/_aliases" -H 'Content-Type: application/json' -d'
{
    "actions" : [
        { "remove" : { "index" : ".kibana_1", "alias" : ".kibana" } },
        { "add" : { "index" : ".kibana_2", "alias" : ".kibana" } }
    ]
}
'

curl -k -u elastic:elastic -X POST "https://192.168.1.132:9200/_aliases" -H 'Content-Type: application/json' -d'
{
    "actions" : [
        { "remove" : { "index" : ".kibana_task_manager_1", "alias" : ".kibana_task_manager" } },
        { "add" : { "index" : ".kibana_task_manager_2", "alias" : ".kibana_task_manager" } }
    ]
}
'

[azasypkin]

Kibana 6.5.x does not fully recover when kibana indices are deleted during operation

@elastic/kibana-core do we actually intend to ever support deleting active .kibana index while Kibana is running or we can close this issue as wont-fix?

[pgayvallet]

No, we don't, especially now that our indices are system indices.