Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot Create Cross Site Index Patterns in Kibana 6.7.1 #35229

Closed
hgoscenski-vail opened this issue Apr 17, 2019 · 18 comments

Comments

Projects
None yet
4 participants
@hgoscenski-vail
Copy link

commented Apr 17, 2019

Kibana version:
6.7.1

Elasticsearch version:
6.7.1

Server OS version:
Running in Docker on Centos 7

Browser version:
Chrome 73, Firefox Nightly

Browser OS version:
MacOS 10.14

Original install method (e.g. download page, yum, from source, etc.):
Docker from docker.elastic.co

Describe the bug:
Showing failed to load remote clusters occasionally when going to the index pattern creation page.

Upgraded from Kibana/Elasticsearch 6.5.1 to 6.7.1 using a rolling upgrade. After the update all seemed fine but after a user went to create a new index patterns they were unable to use the : syntax.

The data is present in the cluster and can be matched against without the remote cluster component. When adding a *: or sf-prod: in front of an index pattern I know exists on all remote clusters the page churns for a moment then says "The index pattern you've entered doesn't match any indices." For a moment, immediately after adding the :, if I am sufficiently quick I can click the "Next Step" button and create the index pattern anyway.

I can also import index patterns with the : syntax from the Saved Objects tab of the management page.

Testing in the Kibana Dev Tools pane shows that I can see and search against those remote clusters/indices and see the results that I am expecting.

Screen Shot 2019-04-16 at 11 19 55 AM
Screen Shot 2019-04-16 at 11 40 59 AM

Remote Cluster Config:

GET _remote/info
{
  "sf-prod" : {
    "seeds" : [
      "splefkc01.vail:9300",
      "splefkc02.vail:9300"
    ],
    "http_addresses" : [
      "172.20.xxx.yyy:9201",
      "172.20.xxx.yyx:9201",
      "172.20.xxx.yxx:9201"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  },
  "au-prod" : {
    "seeds" : [
      "aplefkc01.vail:9300",
      "aplefkc02.vail:9300"
    ],
    "http_addresses" : [
      "172.25..xxx.xxx:9201",
      "172.25..xxx.xxy:9201",
      "172.25..xxx.xyy:9201"
    ],
    "connected" : true,
    "num_nodes_connected" : 3,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

Querying to remote cluster:

GET /sf-prod:kubernetes-kube-system-dex-dex-*/_search?q=*
{
  "took" : 54,
  "timed_out" : false,
  "_shards" : {
    "total" : 12,
    "successful" : 12,
    "skipped" : 0,
    "failed" : 0
  },
  "_clusters" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0
  },
  "hits" : {
    "total" : 22480,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "sf-prod:kubernetes-kube-system-dex-dex-2019.01",
        "_type" : "doc",
        "_id" : "ZjE2OGNkMjAtN2NiMS00MTgxLTliZjEtYTMyODc0OTQ3YzAw",
        "_score" : 1.0,
        "_source" : { 
...

Steps to reproduce:

  1. Have a remote cluster with data available
  2. Add remote cluster to local cluster
  3. Attempt to create index pattern in Kibana - find index and then use : syntax

Expected behavior:
Index pattern is created and able to be used in Kibana.

Screenshots (if relevant):

Errors in browser console (if relevant):
No errors in browser console.

Provide logs and/or server output (if relevant):
Kibana logs when attempting to create index pattern

{"type":"response","@timestamp":"2019-04-16T16:40:50Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true","method":"post","headers":{"host":"splefkc01.vail","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","content-length":"69","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","content-type":"application/json","dnt":"1","kbn-version":"6.7.1","origin":"https://splefkc01.vail","referer":"https://splefkc01.vail/app/kibana","x-forwarded-email":"hgoscenski@REDACTED","x-forwarded-for":"192.168.xxx.xxx","x-forwarded-user":"hgoscenski"},"remoteAddress":"172.20.xxx.xxx","userAgent":"172.20.xxx.xxx","referer":"https://splefkc01.vail/app/kibana"},"res":{"statusCode":200,"responseTime":816,"contentLength":9},"message":"POST /elasticsearch/kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true 200 816ms - 9.0B"}
{"type":"response","@timestamp":"2019-04-16T16:40:54Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/*kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true","method":"post","headers":{"host":"splefkc01.vail","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","content-length":"69","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","content-type":"application/json","dnt":"1","kbn-version":"6.7.1","origin":"https://splefkc01.vail","referer":"https://splefkc01.vail/app/kibana","x-forwarded-email":"hgoscenski@REDACTED","x-forwarded-for":"192.168..xxx.xxx","x-forwarded-user":"hgoscenski"},"remoteAddress":"172.20..xxx.xxx","userAgent":"172.20.xxx.xxx","referer":"https://splefkc01.vail/app/kibana"},"res":{"statusCode":200,"responseTime":711,"contentLength":9},"message":"POST /elasticsearch/*kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true 200 711ms - 9.0B"}
{"type":"response","@timestamp":"2019-04-16T16:40:54Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/*:kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true","method":"post","headers":{"host":"splefkc01.vail","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36","content-length":"69","accept":"application/json, text/plain, */*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","content-type":"application/json","dnt":"1","kbn-version":"6.7.1","origin":"https://splefkc01.vail","referer":"https://splefkc01.vail/app/kibana","x-forwarded-email":"hgoscenski@REDACTED","x-forwarded-for":"192.168..xxx.xxx","x-forwarded-user":"hgoscenski"},"remoteAddress":"172.20.xxx.xxx","userAgent":"172.20.xxx.xxx","referer":"https://splefkc01.vail/app/kibana"},"res":{"statusCode":200,"responseTime":1239,"contentLength":9},"message":"POST /elasticsearch/*:kubernetes-kube-system-dex-dex-*/_search?ignore_unavailable=true 200 1239ms - 9.0B"}

Any additional context:
I am using Searchguard to provide access control to the clusters. Searchguard is configured to provide the user I am testing with full access to all indicies. The kibana_server user is given these permissions:

sg_kibana_server:
  readonly: true
  cluster:
    - CLUSTER_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - cluster:admin/xpack/monitoring*
    - indices:admin/template*
    - indices:data/read/scroll*
    - cluster:monitor/remote/info
    - cluster:monitor/*
  indices:
    "?kibana":
      "*":
        - INDICES_ALL
    "?kibana-6":
      "*":
        - INDICES_ALL
    "?kibana_*":
      "*":
        - INDICES_ALL
    "?reporting*":
      "*":
        - INDICES_ALL
    "?monitoring*":
      "*":
        - INDICES_ALL
    "?tasks":
      "*":
        - INDICES_ALL
    "?management-beats*":
      "*":
        - INDICES_ALL
    "*":
      "*":
        - indices:admin/aliases*
        - indices:data/read/search
        - indices:admin/shards/search_shards

and for the kibana user which is automatically assigned to all signed in users:

sg_kibana_user:
  readonly: true
  cluster:
    - INDICES_MONITOR
    - CLUSTER_COMPOSITE_OPS
    - cluster:monitor/*
    - indices:admin/shards/search_shards  
  indices:
    "?kibana":
      "*":
        - MANAGE
        - INDEX
        - READ
        - DELETE
    "?kibana-6":
      "*":
        - MANAGE
        - INDEX
        - READ
        - DELETE
    "?kibana_*":
      "*":
        - MANAGE
        - INDEX
        - READ
        - DELETE
    "?tasks":
      "*":
        - INDICES_ALL
    "?management-beats":
      "*":
        - INDICES_ALL
    "*":
      "*":
        - indices:data/read/field_caps*
        - indices:data/read/xpack/rollup*
        - indices:admin/mappings/get*
        - indices:admin/get
        - indices:admin/shards/search_shards

EDIT: Redacting.

@sebelga

This comment has been minimized.

Copy link
Contributor

commented Apr 17, 2019

Hi,
Thanks for reporting. This indeed seems like a bug (@elastic/kibana-management ). We will look into it.
Cheers!

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 17, 2019

Thank you! Let me know if you need anything else to troubleshoot/debug.

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

@hgoscenski-vail I'm going to work on reproducing this locally but it might be helpful if you pop open the inspector in your browser and see if there are any failed http requests.

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 18, 2019

I do not see any errors in the inspector.

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 18, 2019

@hgoscenski-vail I set up a remote cluster but was able to create index patterns against it. You are using Searchguard - is there a way to see if the problem exists without Searchguard?

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 19, 2019

It very well may be an issue with Searchguard, from reading through some issue tickets here and the changelogs it looks like there may be different permissions needed in 6.7.1 vs 6.5.1 to access and use the cross cluster index pattern creation. Do you know what permissions are required to do so?

I could spin up 2 new clusters and set one up as a remote but I have a feeling I would just be duplicating your results.

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

@hgoscenski-vail I'm not aware of any permissions changing in a minor version. That said, I'm happy to read what you've come across.

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 19, 2019

I was looking at this issue #27093 and the subsequent pull request #27345 and it seems like they did change some permissions.

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

@hgoscenski-vail That issue and PR addressed a problem that was discovered and addressed before release and expressed itself via the error Failed to load remote clusters

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 19, 2019

Occasionally I get the same message when first loading up the Index Creation page, let me see if I can grab a snapshot. The message does not show up every time however. When that message appears there is no message in the console.
Screen Shot 2019-04-19 at 11 00 15 AM

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2019

When that message appears there is no message in the console.

It should correspond to a failed http request in the network tab in dev tools. To be honest, I'm not sure if failed network requests always display in the console. Anyway, seeing that failed request would be helpful.

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 22, 2019

I can try over the next few days and see if I can catch an error in the Dev tools console.

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 22, 2019

Looks like I am not getting an dev tools error.
Screen Shot 2019-04-22 at 2 39 25 PM
Screen Shot 2019-04-22 at 2 39 27 PM

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2019

Anything in red in the network tab when that happens?

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented Apr 23, 2019

Yes. I checked and it looks like there are two _search?ignore_unavailable=true and one of them fails.

Screen Shot 2019-04-23 at 8 33 48 AM

During one attempt it failed and showed the Failed to Connect to Remote Clusters message the other time it did not but I was still unable to create the index pattern.

@mattkime

This comment has been minimized.

Copy link
Contributor

commented Apr 26, 2019

Click on one of those network request entries to see the POST params in the request.

@pukhanov

This comment has been minimized.

Copy link

commented Apr 30, 2019

Looks like Searchguard had a bug with CCS index creation. Should be fixed in 25.0 floragunncom/search-guard/pull/675

@hgoscenski-vail

This comment has been minimized.

Copy link
Author

commented May 1, 2019

@pukhanov Thank you! I updated my instances and I was able to create the remote index patterns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.