Unable to upgrade kibana - forbidden

[mohmad-null]

Upgrading ES and Kibana from 6.3.2 to 7.2.0.

ES upgrade is fine, but when I try and run Kibana 7.2.0 I get an exception:

  log   [17:01:07.628] [fatal][root] { Error: [cluster_block_exception] index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
    at respond (C:\\elasticSearch\kibana-oss-7.2.0-windows-x86_64\kibana-7.2.0-windows-x86_64\node_modules\elasticsearch\src\lib\transport.js:315:15)
    at checkRespForFailure (C:\\elasticSearch\kibana-oss-7.2.0-windows-x86_64\kibana-7.2.0-windows-x86_64\node_modules\elasticsearch\src\lib\transport.js:274:7)
    at HttpConnector.<anonymous> (C:\\elasticSearch\kibana-oss-7.2.0-windows-x86_64\kibana-7.2.0-windows-x86_64\node_modules\elasticsearch\src\lib\connectors\http.js:166:7)

    at IncomingMessage.wrapper (C:\\elasticSearch\kibana-oss-7.2.0-windows-x86_64\kibana-7.2.0-windows-x86_64\node_modules\elasticsearch\node_modules\lodash\lodash.js:4935:
19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 403,
  displayName: 'AuthorizationException',
  message:
   '[cluster_block_exception] index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];',
  path: '/_aliases',
  query: {},
  body:
   { error:
      { root_cause: [Array],
        type: 'cluster_block_exception',
        reason:
         'index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];' },
     status: 403 },
  statusCode: 403,
  response:
   '{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_ex
ception","reason":"index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}',
  toString: [Function],
  toJSON: [Function] }

 FATAL  [cluster_block_exception] index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]; :: {"path":"/_aliases","query":{},"body":"{\"actions\":[{\"remove
_index\":{\"index\":\".kibana\"}},{\"add\":{\"index\":\".kibana_1\",\"alias\":\".kibana\"}}]}","statusCode":403,"response":"{\"error\":{\"root_cause\":[{\"type\":\"cluster_block_ex
ception\",\"reason\":\"index [.kibana] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"}],\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana] blocke
d by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}"}

No plugins.

This is using the OSS version of ES, so there shouldn't be any security stuff by default right?

If I then start Kibana again it says there's a database upgrade in progress and to delete it if it doesn't complete.

[cbuescher]

Hi @mohmad-null! Your question seems better suited for the support forums over at https://discuss.elastic.co. There is an elasticsearch and a Kibana section there. We prefer to use Github issues only for bug reports and feature requests, and we think it's more likely this is a specific problem than a bug report. I'm closing this as an issue here, but encourage you to ask this question in the forum instead. Many thanks.

[mohmad-null]

@cbuescher - This is a bug report. I don't need support with it because I simply worked around it by deleting all the kibana databases. It's a bug because a simple upgrade didn't work and it should.

[cbuescher]

My apologies, I read this as an isolated topic. I'll move this to the kibana repo though since it seems the .kibana index is somehow set to read_only when it shouldn't.

[elasticmachine]

Pinging @elastic/kibana-operations

[tylersmalley]

@mohmad-null, based on the error .kibana is set to read-only. Can you provide GET .kibana/_settings.

[mohmad-null]

@tylersmalley - it was whatever the default was; I'll not have changed it - I don't use kibana except the dev console to access ES. In fact I didn't even know you could change databases to read-only.

The current values (this is post-upgrade):

{
  ".kibana_1" : {
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".kibana_1",
        "creation_date" : "1562865154732",
        "number_of_replicas" : "0",
        "uuid" : "vqAH41I8SPS47QpNPjxH5A",
        "version" : {
          "created" : "7020099"
        }
      }
    }
  }
}

[tylersmalley]

I don't see index.blocks.read_only_allow_delete as I would have expected. Another possibility is It's hitting the high watermark threshold for disk usage of the ES cluster and preventing writes. Are you still experiencing this? If so, what is GET _cluster/health

[tylersmalley]

Closing as we have not received the feedback requested. The issue here is related to the index being read-only. In the future, these Kibana indices will be system indices and managed by the ES Kibana plugin (#49764)