Update index names for endpoint Datastream permissions.

[pjhampton]

The Kibana System user can't read out of the .ds-metrics-endpoint.policy-* data streams to get the Endpoint metrics document + fleet policy responses. I'm wondering if this fixes it by referencing the data stream name directly.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[albertzaharovits]

@pjhampton Given the current definition for the kibana_system role, can you please clarify what type of search requests don't work but should?

[albertzaharovits]

(From my vantage, I would always prefer fixed names to wildcards for built-in roles, but it might not be what you need.)

[pjhampton]

Hey @albertzaharovits

The Kibana system user has to now query 2 data streams daily:

.ds-metrics-endpoint.metrics*
.ds-metrics-endpoint.policy*

See more details on this here: elastic/kibana#102171

The queries in my description are what is implemented in Kibana:

https://github.com/elastic/kibana/pull/102171/files#diff-42eb463cfaca4b263f4d6543fd550415c343c5f004a3c72312c8e0dd704ef1caR149

https://github.com/elastic/kibana/pull/102171/files#diff-42eb463cfaca4b263f4d6543fd550415c343c5f004a3c72312c8e0dd704ef1caR210

---

I've confirmed this is read access related by doing a reindex op into a fleet managed index .fleet

POST _reindex
{
  "source": {
    "index": "metrics-endpoint.metrics-default"
  },
  "dest": {
    "index": ".fleet-metrics-endpoint.metrics-test"
  }
}

Please note I am testing against the ES snapshot

[albertzaharovits]

@pjhampton I've tried reproducing the problem, no dice:

workspace/elasticsearch [master] » curl -u elastic:password -X POST "localhost:9200/_security/user/jacknich?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "password",
  "roles" : [ "kibana_system" ]
}
'
{
  "created" : true
}
workspace/elasticsearch [master] » curl -u elastic:password -X PUT "localhost:9200/_data_stream/metrics-endpoint.policy-default?pretty"
{
  "acknowledged" : true
}
workspace/elasticsearch [master] » curl -u elastic:password -X POST "localhost:9200/metrics-endpoint.policy-default/_doc" -H 'Content-Type: application/json' -d'
{
  "category": "click", "tag": "three", "@timestamp": "1270001"
}
'

{"_index":".ds-metrics-endpoint.policy-default-2021.07.07-000001","_id":"D_3jgXoB13ZEC5gM3H6Y","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}%
workspace/elasticsearch [master] » curl -u elastic:password -X POST "localhost:9200/metrics-endpoint.policy-default/_rollover/?pretty"

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "old_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000001",
  "new_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000002",
  "rolled_over" : true,
  "dry_run" : false,
  "conditions" : { }
}
workspace/elasticsearch [master] » curl -u elastic:password -X POST "localhost:9200/metrics-endpoint.policy-default/_doc" -H 'Content-Type: application/json' -d'
{
  "category": "click", "tag": "three", "@timestamp": "1270001"
}
'

{"_index":".ds-metrics-endpoint.policy-default-2021.07.07-000002","_id":"E_3kgXoB13ZEC5gMBn6w","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}%
workspace/elasticsearch [master] » curl -u elastic:password -X POST "localhost:9200/metrics-endpoint.policy-default/_rollover/?pretty"

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "old_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000002",
  "new_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000003",
  "rolled_over" : true,
  "dry_run" : false,
  "conditions" : { }
}
workspace/elasticsearch [master] » curl -u jacknich:password -X POST "localhost:9200/metrics-endpoint.policy-default/_search?pretty"
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000001",
        "_id" : "D_3jgXoB13ZEC5gM3H6Y",
        "_score" : 1.0,
        "_source" : {
          "category" : "click",
          "tag" : "three",
          "@timestamp" : "1270001"
        }
      },
      {
        "_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000002",
        "_id" : "E_3kgXoB13ZEC5gMBn6w",
        "_score" : 1.0,
        "_source" : {
          "category" : "click",
          "tag" : "three",
          "@timestamp" : "1270001"
        }
      }
    ]
  }
}
workspace/elasticsearch [master] » curl -u jacknich:password -X POST "localhost:9200/.ds-metrics-endpoint.policy*/_search?pretty"
{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000001",
        "_id" : "D_3jgXoB13ZEC5gM3H6Y",
        "_score" : 1.0,
        "_source" : {
          "category" : "click",
          "tag" : "three",
          "@timestamp" : "1270001"
        }
      },
      {
        "_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000002",
        "_id" : "E_3kgXoB13ZEC5gMBn6w",
        "_score" : 1.0,
        "_source" : {
          "category" : "click",
          "tag" : "three",
          "@timestamp" : "1270001"
        }
      }
    ]
  }
}
workspace/elasticsearch [master] » curl -u jacknich:password -X POST "localhost:9200/.ds-metrics-endpoint.policy*/_search?expand_wildcards=open,hidden&pretty"
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 3,
    "successful" : 3,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000001",
        "_id" : "D_3jgXoB13ZEC5gM3H6Y",
        "_score" : 1.0,
        "_source" : {
          "category" : "click",
          "tag" : "three",
          "@timestamp" : "1270001"
        }
      },
      {
        "_index" : ".ds-metrics-endpoint.policy-default-2021.07.07-000002",
        "_id" : "E_3kgXoB13ZEC5gMBn6w",
        "_score" : 1.0,
        "_source" : {
          "category" : "click",
          "tag" : "three",
          "@timestamp" : "1270001"
        }
      }
    ]
  }
}
workspace/elasticsearch [master] » curl -u jacknich:password -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "metrics-endpoint.policy*"
  },
  "dest": {
    "index": ".fleet-metrics-endpoint.policy-test"
  }
}
'

{
  "took" : 1089,
  "timed_out" : false,
  "total" : 2,
  "updated" : 0,
  "created" : 2,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}

Can you clarify which is the request that does not work, and what the error is?

[pjhampton]

Thanks, @albertzaharovits. I was able to reproduce your testing.
The error turned out to be on my end - It was how the reindexing changed how we can aggregate.

...
        aggs: {
          endpoint_agents: {
            terms: {
              field: 'agent.id', <-- reindexed documents required .keyword
              size: this.max_records,
            },

Thanks again for helping on this one