Scrolling a remote index should not require cluster:monitor/state privilege

[droberts195]

At the moment it seems that starting a scroll on a remote index requires just the index permissions you'd expect, but continuing that scroll involving a remote index additionally requires that you have permission to read the entire cluster state.

To demonstrate:

1. Install two local single node clusters on your machine with security enabled: cluster 1 and cluster 2
2. Configure cluster 1 as a remote of cluster 2
3. Import some data into both clusters
4. Create a role cluster_one_data in cluster 1 that gives all access to the index containing the imported data
5. Create a role cluster_one_data in cluster 1 that gives all access to the remote index containing the imported data in cluster 1
6. Create a role cluster_two_data in cluster 2 that gives all access to the index containing the imported data in cluster 2
7. Create a user in cluster 2 with the following roles:
   a. kibana_user
   b. cluster_one_data
   c. cluster_two_data
8. From cluster 2, try scrolling the local and remote data in both clusters using the newly created user:

POST /farequote/_search?scroll=1m

POST /_search/scroll
{
  "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAALfEWZGhpcUxXeENST1dZRUZSMTF1T2Vidw==",
  "scroll" : "1m"
}

POST /cluster_one:farequote/_search?scroll=1m

POST /_search/scroll
{
  "scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAABvQiY2x1c3Rlcl9vbmU6Qm1hRFIzOWdRT0d1bjI5OV9hZE5zdw==",
  "scroll" : "1m"
}

(Obviously the exact scroll IDs will vary from run to run.)

You will note that the local scroll can be started and continued with just privileges on the index being scrolled. The scroll of the remote index can also be started successfully, returning the first few hits as expected. But the attempt to continue the remote scroll returns this:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "action [cluster:monitor/state] is unauthorized for user [dave]"
      }
    ],
    "type": "security_exception",
    "reason": "action [cluster:monitor/state] is unauthorized for user [dave]"
  },
  "status": 403
}

I think that if the remote scroll continuation needs to get some portion of the cluster state as an internal implementation detail then it should switch to system context to get this information, because logically you'd expect to be able to scroll an index when you have been granted permission to scroll that index. You should not additionally need permission to read the entire cluster state.

[elasticmachine]

Pinging @elastic/es-search

[droberts195]

I think the problem is in the RemoteClusterConnection class:

elasticsearch/server/src/main/java/org/elasticsearch/transport/RemoteClusterConnection.java

Line 210 in b9ce649

transportService.sendRequest(connection, ClusterStateAction.NAME, request, TransportRequestOptions.EMPTY,

I think that call to get the remote cluster state for the purpose of checking which nodes exist should be done in the system context, like in

elasticsearch/server/src/main/java/org/elasticsearch/cluster/metadata/TemplateUpgradeService.java

Line 134 in b9ce649

threadContext.markAsSystemContext();