kibana fails to come up with a fatal error if user sets the value of kibana.index in kibana.yml

[bhavyarm]

Kibana version: 7.16.0 BC5

Elasticsearch version: 7.16.0 BC5

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: As part of testing deprecations for 7.16 upgrade to 8.0.0 - I added in a different name for .kibana index in kibana config

kibana.index: ".thisisabadidea"

Now Kibana fails to come up with these logs:

 log   [09:54:47.752] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 1 in 2 seconds.
  log   [09:54:47.753] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 75ms.
  log   [09:54:47.757] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_READ. took: 72ms.
  log   [09:54:47.850] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_READ -> OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT. took: 93ms.
  log   [09:54:48.292] [info][savedobjects-service] [.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT -> UPDATE_TARGET_MAPPINGS. took: 442ms.
  log   [09:54:48.550] [info][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK. took: 258ms.
  log   [09:54:48.690] [info][savedobjects-service] [.kibana_task_manager] UPDATE_TARGET_MAPPINGS_WAIT_FOR_TASK -> DONE. took: 140ms.
  log   [09:54:48.690] [info][savedobjects-service] [.kibana_task_manager] Migration completed after 1074ms
  log   [09:54:49.759] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 2 in 4 seconds.
  log   [09:54:49.759] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 2007ms.
  log   [09:54:53.765] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 3 in 8 seconds.
  log   [09:54:53.765] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 4006ms.
  log   [09:55:01.772] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 4 in 16 seconds.
  log   [09:55:01.773] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 8007ms.
  log   [09:55:17.879] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 5 in 32 seconds.
  log   [09:55:17.879] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 16107ms.
  log   [09:55:49.885] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 6 in 64 seconds.
  log   [09:55:49.885] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 32006ms.
  log   [09:56:53.891] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 7 in 64 seconds.
  log   [09:56:53.891] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64006ms.
  log   [09:57:57.899] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 8 in 64 seconds.
  log   [09:57:57.900] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64008ms.
  log   [09:59:01.907] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 9 in 64 seconds.
  log   [09:59:01.907] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64008ms.
  log   [10:00:05.914] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 10 in 64 seconds.
  log   [10:00:05.914] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64007ms.
  log   [10:01:09.921] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 11 in 64 seconds.
  log   [10:01:09.921] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64007ms.
  log   [10:02:13.927] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 12 in 64 seconds.
  log   [10:02:13.927] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64006ms.
  log   [10:03:17.932] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 13 in 64 seconds.
  log   [10:03:17.932] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64005ms.
  log   [10:04:21.940] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 14 in 64 seconds.
  log   [10:04:21.940] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64008ms.
  log   [10:05:25.946] [error][savedobjects-service] [.thisisabadidea] Action failed with 'security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for user [kibana_system] with roles [kibana_system] on indices [.thisisabadidea_7.16.0_001], this action is granted by the index privileges [create_index,manage,all]'. Retrying attempt 15 in 64 seconds.
  log   [10:05:25.946] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 64006ms.
  log   [10:06:29.953] [info][savedobjects-service] [.thisisabadidea] CREATE_NEW_TARGET -> FATAL. took: 64007ms.
  log   [10:06:29.954] [fatal][root] Error: Unable to complete saved object migrations for the [.thisisabadidea] index: Unable to complete the CREATE_NEW_TARGET step after 15 attempts, terminating.
    at migrationStateActionMachine (/Users/bhavyarajumandya/Desktop/7_12_1_to_7.16/kibana-7.16.0-darwin-x86_64/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at SavedObjectsService.start (/Users/bhavyarajumandya/Desktop/7_12_1_to_7.16/kibana-7.16.0-darwin-x86_64/src/core/server/saved_objects/saved_objects_service.js:175:9)
    at Server.start (/Users/bhavyarajumandya/Desktop/7_12_1_to_7.16/kibana-7.16.0-darwin-x86_64/src/core/server/server.js:330:31)
    at Root.start (/Users/bhavyarajumandya/Desktop/7_12_1_to_7.16/kibana-7.16.0-darwin-x86_64/src/core/server/root/index.js:69:14)
    at bootstrap (/Users/bhavyarajumandya/Desktop/7_12_1_to_7.16/kibana-7.16.0-darwin-x86_64/src/core/server/bootstrap.js:120:5)
    at Command.<anonymous> (/Users/bhavyarajumandya/Desktop/7_12_1_to_7.16/kibana-7.16.0-darwin-x86_64/src/cli/serve/serve.js:229:5)
  log   [10:06:29.962] [info][plugins-system][standard] Stopping all plugins.
  log   [10:06:29.962] [info][kibana-monitoring][monitoring][monitoring][plugins] Monitoring stats collection is stopped
  log   [10:06:59.970] [warning][plugins-system][standard] "eventLog" plugin didn't stop in 30sec., move on to the next.

 FATAL  Error: Unable to complete saved object migrations for the [.thisisabadidea] index: Unable to complete the CREATE_NEW_TARGET step after 15 attempts, terminating.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[rudolf]

This behaviour is consistent with other 7.x versions. Users will have to use a setting starting with .kibana_ in order to use the default roles.