[ftr] implement support for accessing ES through CCS

[spalger]

Closes #124865

In order to support testing CCS compatibility we have decided to extend the FTR config schema to support esTestCluster.ccs which, when enabled, changes the way that we start ES to start two single-node clusters:

1. ftr-remote

This cluster is started first, it listens to HTTP requests at esTestCluster.ccs.remoteClusterUrl, and transport requests on a port determined at runtime using the get-port module. The remoteEs and remoteEsArchiver services are setup to communicate directly with this node for loading data into and managing the remote cluster. Roles on this cluster can be setup with the esTestCluster.remoteRoles config.

2. ftr-local

This cluster is initialized second, it is the cluster that Kibana and all the default services will talk to, including es and esArchiver. This cluster also has a single remote cluster configured at startup, ftr-remote which can be used to query the ftr-remote cluster.

---

In the test/functional_ccs/config.ts file we setup the ccs_remote_search role on the remote cluster, and then also set this as a default role for all users created on via the testUser service. When calling testUser.setRoles() people will need to define this role if they want that user to have CCS permissions on the remote cluster. We could change the logic to be out-out, but I'd prefer to make it explicit when we call setRoles() that the list of roles provided will be the roles that user has.

Additionally, in order to support cloud testing this config supports defining the entire esTestCluster.ccs.remoteClusterUrl via the REMOTE_CLUSTER_URL environment variable, which could point to a completely different machine.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[spalger]

@elasticmachine merge upstream

[liza-mae]

Yes set like this:

_pass1=<password>
_pass2=<password>
export TEST_ES_URL=https://elastic:${_pass1}@<endpoint>
export TEST_KIBANA_URL=https://elastic:${_pass1}@<endpoint>
export REMOTE_CLUSTER_URL=https://elastic:${_pass2}@<endpoint>

The curl command for those URLs comes back successful.

curl $TEST_ES_URL
{
  "name" : "instance-0000000001",
  "cluster_name" : "2695877d320c4bcdbfbc81223be4401d",
  "cluster_uuid" : "blV5szVNTqSrGuDHDVbv6Q",
  "version" : {
    "number" : "8.1.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "3700f7679f7d95e36da0b43762189bab189bc53a",
    "build_date" : "2022-03-03T14:20:00.690422633Z",
    "build_snapshot" : false,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

curl $REMOTE_CLUSTER_URL
{
  "name" : "instance-0000000001",
  "cluster_name" : "a8c8efc71c6d4229831104be818f246b",
  "cluster_uuid" : "ezfFY-PZSEyECZWfX0xtSw",
  "version" : {
    "number" : "7.17.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "bee86328705acaa9a6daede7140defd4d9ec56bd",
    "build_date" : "2022-01-28T08:36:04.875279988Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

[liza-mae]

@spalger your fix worked for the superuser security issue. Thanks!

I only ran the data_view_css.ts test but it failed, it appears to be a problem with privileges for the test_user role, so I reverted back to superuser elastic and the test passed.

[liza-mae]

LGTM, tested FTR changes against cloud CCS setup.

[LeeDr]

@spalger can you explain how I would (in a test) get the test_user created with the correct roles on the remote cluster? The existing functional_ccs tests copied to x-pack and inheriting x-pack/test/functional/config.js won't pass for me locally unless it also created test_user on the remote cluster.

I'm not sure we can use the existing test_user service if it's hitting the Kibana api. We need to use the remote Elasticsearch cluster to create the user.

[spalger]

@LeeDr My understanding of https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters-privileges.html#remote-clusters-privileges-ccs was that we just needed to have a known role name in sync between the remote cluster and the local user, so the testUser would still be created on the local cluster but it would need to get a role name that is in place on the remote cluster and gives it the read_cross_cluster permission. In the test/function_ccs config I setup the security.remoteEsRoles config to create the ccs_remote_search role on the remote cluster, users created with the testUsers service should just need the ccs_remote_search role to be able to search on the remote cluster too... My reading of the docs might be incorrect though, do we really need to create user accounts on all clusters?

[LeeDr]

LGTM - Spencer is correct that we don't need the test_user on the remote cluster. Only the role including view_index_metadata. Works great!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: f69b556
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/test	208	210	+2

Unknown metric groups

API count

id	before	after	diff
@kbn/test	244	248	+4

History

• 💚 Build #28041 succeeded 845a8bb
• 💚 Build #27825 succeeded b561d19
• 💚 Build #27773 succeeded 5bb9c96
• 💚 Build #27540 succeeded 34c4c0a
• 💔 Build #27520 failed 29ed085
• 💚 Build #26986 succeeded 0498ec0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.1	Backport failed because of merge conflicts You might need to backport the following PRs to 8.1: - Added CCS test for data view functionality. (#124586)
❌	7.17	Backport failed because of merge conflicts You might need to backport the following PRs to 7.17: - [kbn/es] add support for --ready-timeout (#126217) - Added CCS test for data view functionality. (#124586)
❌	8.0	Backport failed because of merge conflicts You might need to backport the following PRs to 8.0: - Added CCS test for data view functionality. (#124586)

Manual backport

To create the backport manually run:

node scripts/backport --pr 126547

Questions ?

Please refer to the Backport tool documentation

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 126547 or prevent reminders by adding the backport:skip label.

[spalger]

Backport blocked by #124586, going to skip it for now. If you get the backports done and would like me to backport this please let me know @cuff-links

[cuff-links]

Will do.