Kibana should not auto create it's system indices if these don't exist

[rudolf]

Kibana creates it's system indices with specific mappings. This includes:

• saved object indices: .kibana, .kibana_task_manager
• event_log plugin's .kibana-event-log and
• security plugin's .kibana_security_sessions_N.

When these indices are auto-created they don't infer the correct mappings and leads to unwanted behaviour and failure. Security sessions is the only index that gracefully handles being auto created, but this comes at the cost of added complexity to the plugin.

Kibana always creates these indices on startup and will only start indexing documents after they're created. However, users sometimes delete these indices to restore a snapshot without shutting Kibana down so Kibana keeps writing. Similarly, during development, developers often delete all indices without restarting Kibana. When this happens Kibana sometimes keeps on working for a short while. When Kibana eventually starts failing, it's not obvious what caused the problem.

ES introduced the require_alias request parameter to prevent ES from auto creating an index if one doesn't exist elastic/elasticsearch#58917 We can add this parameter to all our requests, but if we want this behaviour to apply to all kibana system indices, it might be better to make this the default behaviour through the Kibana System Indices Elasticsearch plugin.

@elastic/kibana-stack-services @elastic/kibana-reporting-services @elastic/kibana-security any objections with disabling auto create for all these indices?

[kobelb]

If we do make this the default behavior, will developers be able to determine whether or not the error that ES has thrown indicates that the index hasn't been created yet and be able to explicitly create the mappings and retry the operation?

[rudolf]

Yeah they'll get an "index_not_found_exception".

Using require_alias=true

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index [doesntexist] and [require_alias] request flag is [true] and [doesntexist] is not an alias",
        "index_uuid": "_na_",
        "index": "doesntexist"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index [doesntexist] and [require_alias] request flag is [true] and [doesntexist] is not an alias",
    "index_uuid": "_na_",
    "index": "doesntexist"
  },
  "status": 404
}

Using the ES config option action.auto_create_index=.watches,.triggered_watches,.watcher-history-*

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index [doesntexist] and [action.auto_create_index] ([.watches,.triggered_watches,.watcher-history-*]) doesn't match",
        "index_uuid": "_na_",
        "index": "doesntexist"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index [doesntexist] and [action.auto_create_index] ([.watches,.triggered_watches,.watcher-history-*]) doesn't match",
    "index_uuid": "_na_",
    "index": "doesntexist"
  },
  "status": 404
}

I guess we could customize the reason message to be more kibana system index specific.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[legrego]

any objections with disabling auto create for all these indices?

No objections on my end. I think this is worth doing! Especially if we can make a kibana-specific message to help our users out.

[kobelb]

TIL about action.auto_create_index 🤯

[azasypkin]

No objections on my end. I think this is worth doing! Especially if we can make a kibana-specific message to help our users out.

Same here.

Orthogonal topic/question: am I the only one who restarts ES (with yarn es snapshot) during development relatively frequently without restarting Kibana (e.g. when mistakenly chose wrong license, or specifically want to check license-upgrade/downgrade scenario, or forgot to add an additional CLI argument to enable API keys)? If it's common, then in addition to a clear error message I'm wondering if it'd make sense to change default behavior of yarn es snapshot to persist data or something along these lines (and instead add a new argument to wipe the data)?

[rudolf]

I'm wondering if it'd make sense to change default behavior of yarn es snapshot to persist data or something along these lines (and instead add a new argument to wipe the data)?

I agree, I think most developers will benefit from the data being persisted by default.

[gmmorris]

I'm wondering if it'd make sense to change default behavior of yarn es snapshot to persist data or something along these lines (and instead add a new argument to wipe the data)?

I agree, I think most developers will benefit from the data being persisted by default.

++

I definitely hadn't realised this was not a supported case and know other members of my team made the same mistake.
It is the most intuitive way of simulating a lack of connectivity with ES, so we should probably ensure it represents a realistic case or at least provide an easily understood alternative. :)

[rudolf]

#89992 will prevent Kibana from auto-creating saved object indices.

[pgayvallet]

Given it's resolved for SO indices should we close this?

@elastic/kibana-security is this issue also addressed for the security session indices?

[rudolf]

++ if security can confirm then we can close this issue.

[azasypkin]

@elastic/kibana-security is this issue also addressed for the security session indices?

Yes, we addressed this in #134900!

[pgayvallet]

Closing as resolved