Cannot Create Cross Site Index Patterns in Kibana 6.7.1

[hgoscenski-vail]

Kibana version:
6.7.1

Elasticsearch version:
6.7.1

Server OS version:
Running in Docker on Centos 7

Browser version:
Chrome 73, Firefox Nightly

Browser OS version:
MacOS 10.14

Original install method (e.g. download page, yum, from source, etc.):
Docker from docker.elastic.co

Describe the bug:
Showing failed to load remote clusters occasionally when going to the index pattern creation page.

Upgraded from Kibana/Elasticsearch 6.5.1 to 6.7.1 using a rolling upgrade. After the update all seemed fine but after a user went to create a new index patterns they were unable to use the : syntax.

The data is present in the cluster and can be matched against without the remote cluster component. When adding a *: or sf-prod: in front of an index pattern I know exists on all remote clusters the page churns for a moment then says "The index pattern you've entered doesn't match any indices." For a moment, immediately after adding the :, if I am sufficiently quick I can click the "Next Step" button and create the index pattern anyway.

I can also import index patterns with the : syntax from the Saved Objects tab of the management page.

Testing in the Kibana Dev Tools pane shows that I can see and search against those remote clusters/indices and see the results that I am expecting.

Remote Cluster Config:

GET _remote/info
{
  "sf-prod" : {
    "seeds" : [
      "splefkc01.vail:9300",
      "splefkc02.vail:9300"
    ],
    "http_addresses" : [
      "172.20.xxx.yyy:9201",
      "172.20.xxx.yyx:9201",
      "172.20.xxx.yxx:9201"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  },
  "au-prod" : {
    "seeds" : [
      "aplefkc01.vail:9300",
      "aplefkc02.vail:9300"
    ],
    "http_addresses" : [
      "172.25..xxx.xxx:9201",
      "172.25..xxx.xxy:9201",
      "172.25..xxx.xyy:9201"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

Querying to remote cluster:

GET /sf-prod:kubernetes-kube-system-dex-dex-*/_search?q=*
{
  "took" : 54,
  "timed_out" : false,
  "_shards" : {
    "total" : 12,
    "successful" : 12,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : 22480,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "sf-prod:kubernetes-kube-system-dex-dex-2019.01",
        "_type" : "doc",
        "_id" : "ZjE2OGNkMjAtN2NiMS00MTgxLTliZjEtYTMyODc0OTQ3YzAw",
        "_score" : 1.0,
        "_source" : { 
...

Steps to reproduce:

1. Have a remote cluster with data available
2. Add remote cluster to local cluster
3. Attempt to create index pattern in Kibana - find index and then use : syntax

Expected behavior:
Index pattern is created and able to be used in Kibana.

Screenshots (if relevant):

Errors in browser console (if relevant):
No errors in browser console.

Provide logs and/or server output (if relevant):
Kibana logs when attempting to create index pattern

{"type":"response","@timestamp":"2019-04-16T16:40:50Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true","method":"post","headers":{"host":"splefkc01.vail","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","content-length":"69","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","content-type":"application/json","dnt":"1","kbn-version":"6.7.1","origin":"https://splefkc01.vail","referer":"https://splefkc01.vail/app/kibana","x-forwarded-email":"hgoscenski@REDACTED","x-forwarded-for":"192.168.xxx.xxx","x-forwarded-user":"hgoscenski"},"remoteAddress":"172.20.xxx.xxx","userAgent":"172.20.xxx.xxx","referer":"https://splefkc01.vail/app/kibana"},"res":{"statusCode":200,"responseTime":816,"contentLength":9},"message":"POST /elasticsearch/kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true 200 816ms - 9.0B"}
{"type":"response","@timestamp":"2019-04-16T16:40:54Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/*kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true","method":"post","headers":{"host":"splefkc01.vail","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","content-length":"69","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","content-type":"application/json","dnt":"1","kbn-version":"6.7.1","origin":"https://splefkc01.vail","referer":"https://splefkc01.vail/app/kibana","x-forwarded-email":"hgoscenski@REDACTED","x-forwarded-for":"192.168..xxx.xxx","x-forwarded-user":"hgoscenski"},"remoteAddress":"172.20..xxx.xxx","userAgent":"172.20.xxx.xxx","referer":"https://splefkc01.vail/app/kibana"},"res":{"statusCode":200,"responseTime":711,"contentLength":9},"message":"POST /elasticsearch/*kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true 200 711ms - 9.0B"}
{"type":"response","@timestamp":"2019-04-16T16:40:54Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/*:kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true","method":"post","headers":{"host":"splefkc01.vail","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","content-length":"69","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","content-type":"application/json","dnt":"1","kbn-version":"6.7.1","origin":"https://splefkc01.vail","referer":"https://splefkc01.vail/app/kibana","x-forwarded-email":"hgoscenski@REDACTED","x-forwarded-for":"192.168..xxx.xxx","x-forwarded-user":"hgoscenski"},"remoteAddress":"172.20.xxx.xxx","userAgent":"172.20.xxx.xxx","referer":"https://splefkc01.vail/app/kibana"},"res":{"statusCode":200,"responseTime":1239,"contentLength":9},"message":"POST /elasticsearch/*:kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true 200 1239ms - 9.0B"}

Any additional context:
I am using Searchguard to provide access control to the clusters. Searchguard is configured to provide the user I am testing with full access to all indicies. The kibana_server user is given these permissions:

sg_kibana_server:
  readonly: true
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - cluster:admin/xpack/monitoring*
    - indices:admin/template*
    - indices:data/read/scroll*
    - cluster:monitor/remote/info
    - cluster:monitor/*
  indices:
    "?kibana":
      "*":
        - INDICES_ALL
    "?kibana-6":
      "*":
        - INDICES_ALL
    "?kibana_*":
      "*":
        - INDICES_ALL
    "?reporting*":
      "*":
        - INDICES_ALL
    "?monitoring*":
      "*":
        - INDICES_ALL
    "?tasks":
      "*":
        - INDICES_ALL
    "?management-beats*":
      "*":
        - INDICES_ALL
    "*":
      "*":
        - indices:admin/aliases*
        - indices:data/read/search
        - indices:admin/shards/search_shards

and for the kibana user which is automatically assigned to all signed in users:

sg_kibana_user:
  readonly: true
  cluster:
    - INDICES_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - cluster:monitor/*
    - indices:admin/shards/search_shards  
  indices:
    "?kibana":
      "*":
        - MANAGE
        - INDEX
        - READ
        - DELETE
    "?kibana-6":
      "*":
        - MANAGE
        - INDEX
        - READ
        - DELETE
    "?kibana_*":
      "*":
        - MANAGE
        - INDEX
        - READ
        - DELETE
    "?tasks":
      "*":
        - INDICES_ALL
    "?management-beats":
      "*":
        - INDICES_ALL
    "*":
      "*":
        - indices:data/read/field_caps*
        - indices:data/read/xpack/rollup*
        - indices:admin/mappings/get*
        - indices:admin/get
        - indices:admin/shards/search_shards

EDIT: Redacting.

[sebelga]

Hi,
Thanks for reporting. This indeed seems like a bug (@elastic/kibana-management ). We will look into it.
Cheers!

[hgoscenski-vail]

Thank you! Let me know if you need anything else to troubleshoot/debug.

[mattkime]

@hgoscenski-vail I'm going to work on reproducing this locally but it might be helpful if you pop open the inspector in your browser and see if there are any failed http requests.

[hgoscenski-vail]

I do not see any errors in the inspector.

[mattkime]

@hgoscenski-vail I set up a remote cluster but was able to create index patterns against it. You are using Searchguard - is there a way to see if the problem exists without Searchguard?

[hgoscenski-vail]

It very well may be an issue with Searchguard, from reading through some issue tickets here and the changelogs it looks like there may be different permissions needed in 6.7.1 vs 6.5.1 to access and use the cross cluster index pattern creation. Do you know what permissions are required to do so?

I could spin up 2 new clusters and set one up as a remote but I have a feeling I would just be duplicating your results.

[mattkime]

@hgoscenski-vail I'm not aware of any permissions changing in a minor version. That said, I'm happy to read what you've come across.

[hgoscenski-vail]

I was looking at this issue #27093 and the subsequent pull request #27345 and it seems like they did change some permissions.

[mattkime]

@hgoscenski-vail That issue and PR addressed a problem that was discovered and addressed before release and expressed itself via the error Failed to load remote clusters

[hgoscenski-vail]

Occasionally I get the same message when first loading up the Index Creation page, let me see if I can grab a snapshot. The message does not show up every time however. When that message appears there is no message in the console.

[mattkime]

When that message appears there is no message in the console.

It should correspond to a failed http request in the network tab in dev tools. To be honest, I'm not sure if failed network requests always display in the console. Anyway, seeing that failed request would be helpful.

[hgoscenski-vail]

I can try over the next few days and see if I can catch an error in the Dev tools console.

[hgoscenski-vail]

Looks like I am not getting an dev tools error.

[mattkime]

Anything in red in the network tab when that happens?

[hgoscenski-vail]

Yes. I checked and it looks like there are two _search?ignore_unavailable=true and one of them fails.

During one attempt it failed and showed the Failed to Connect to Remote Clusters message the other time it did not but I was still unable to create the index pattern.

[mattkime]

Click on one of those network request entries to see the POST params in the request.

[pukhanov]

Looks like Searchguard had a bug with CCS index creation. Should be fixed in 25.0 floragunncom/search-guard/pull/675

[hgoscenski-vail]

@pukhanov Thank you! I updated my instances and I was able to create the remote index patterns.