After upgrade 3.10 to 3.11 cluster-admin can't browse logs

[t0ffel]

Repro scenario:
3.10 cluster.
Authentication via htpasswd.

1. Add cluster-admin role to user2.
2. log in as user2 to Kibana, make sure you can see all the logs just fine.
3. upgrade just the logging bits to v3.11
4. user2 is still cluster-admin, but now after logging into Kibana he can't browse discover tab. the error is "Index Patterns: Please specify a default index pattern".
5. When trying to specify the default index pattern (any of .all, .operations., .orphaned., .project*) the error is: "Config: Error 404 Not Found: [doc][config:5.6.10]: document missing: [document_missing_exception] [doc][config:5.6.10]: document missing, with { index_uuid="5lgLBDtCSKipk1qJZvRdSQ" & shard="0" & …".

In ES log with trace enabled:

[2018-08-31T16:54:14,133][DEBUG][i.f.e.p.KibanaUserReindexAction] Response with Action 'indices:data/write/update'
[2018-08-31T16:54:14,134][TRACE][o.e.t.T.tracer           ] [logging-es-data-master-dwevbw1v] [504][indices:data/write/update[s]] sent to [{logging-es-data-master-dwevbw1v}{b3J6fw
BoQZGFyPOhGpLEDQ}{W5AsWQJfS2Cv5pNf0xhzOQ}{10.1.2.35}{10.1.2.35:9300}] (timeout: [null])
[2018-08-31T16:54:14,134][TRACE][o.e.t.T.tracer           ] [logging-es-data-master-dwevbw1v] [504][indices:data/write/update[s]] received request
[2018-08-31T16:54:14,134][TRACE][o.e.t.TaskManager        ] [logging-es-data-master-dwevbw1v] register 1557 [direct] [indices:data/write/update[s]] []
[2018-08-31T16:54:14,136][TRACE][o.e.t.TaskManager        ] [logging-es-data-master-dwevbw1v] unregister task for id: 1557
[2018-08-31T16:54:14,136][TRACE][o.e.t.T.tracer           ] [logging-es-data-master-dwevbw1v] [504][indices:data/write/update[s]] sent error response
org.elasticsearch.index.engine.DocumentMissingException: [config][5.6.10]: document missing
        at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:93) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:82) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:179) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:172) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:69) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:246) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:242) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.6.10.jar:5.6.10]
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178) ~[?:?]
        at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:107) ~[?:?]
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:92) ~[?:?]
        at com.floragunn.searchguard.SearchGuardPlugin$5$1.messageReceived(SearchGuardPlugin.java:493) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.10.jar:5.6.10]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.10.jar:5.6.10]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-08-31T16:54:14,138][TRACE][o.e.t.T.tracer           ] [logging-es-data-master-dwevbw1v] [504][indices:data/write/update[s]] received response from [{logging-es-data-master-dwevbw1v}{b3J6fwBoQZGFyPOhGpLEDQ}{W5AsWQJfS2Cv5pNf0xhzOQ}{10.1.2.35}{10.1.2.35:9300}]

If I create a new cluster-admin user, or don't log in to Kibana in v3.10 with user2 - I can browse the indices just fine.

/cc @jcantrill

[bchojnicki]

Same issue here with one exception: I'm using LDAP auth.

[mz118]

Same issue here - also with LDAP auth.

[yekaifeng]

Same issue with maiden install openshift logging 3.11. Authentication via htpasswd.

[mmariyan]

Same issue with the cluster-admin user ( Active-directory group).

[adawolfs]

The process to fix this issue is described at
https://bugzilla.redhat.com/show_bug.cgi?id=1652224#c4

Basically what it means is that as the user index (where all the custom options are stored) exists so kibana tries to fetch and save data from there but that document actually comes with the kibana version config:5.6.10 it makes unable to save new data as the expected document doesn't exists nor try to create it.

For this reason is necessary to delete the current user index at elasticsearch and login again to recreate the index and make things work.