Add RCS Strong Verification Documentation

[jfreden]

This adds documentation for the RCS Strong Verification feature added in elastic/elasticsearch#136299, elastic/elasticsearch#134137, elastic/elasticsearch#134893, elastic/elasticsearch#135674 and elastic/elasticsearch#134604.

Related settings docs PR: elastic/elasticsearch#137822

[github-actions]

🔍 Preview links for changed docs

• deploy-manage/remote-clusters/remote-clusters-api-key.md

[shainaraskas]

couple of comments.

consider swapping steps 4 and 5 in the list of tasks (so all of the relationship tasks are done before the user/role tasks):

we might hint that these steps can be undertaken paired with steps 2 and 3 (because you'll be on those machines anyway setting up the trust relationship)

tagging in @eedugon as well because this is in the remote clusters space and Edu has been doing some work to refactor these docs. Just want to make sure we're not doing anything that would conflict with that work (edu: do you think this needs to be added to any of the other tutorials? not sure which ones piggyback on core elasticsearch remote cluster features)

[shainaraskas]

There are a lot more settings specified in your reference PR. how does someone know from this procedure when to use those additional settings?

[jfreden]

Good point, I wasn't sure if it made sense to include all settings. This is the most basic configuration to keep it simple, would it make sense to link the other settings here you think?

This is a setup the mirrors our SSL configuration so I could either copy some of the instructions from there or link that too? WDYT?

[eedugon]

Including just some settings in the configuration guide should be ok, but probably we should include all settings we want users to know in the Elasticsearch reference docs. Somewhere like here: https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/remote-clusters, or here: https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/networking-settings#remote-cluster-network-settings (we have settings in 2 places).

We have an opened internal issue to review these settings as some of them (remote_cluster_client) are missing in the reference docs.

For your consideration.

[eedugon]

Update: I see you have created elastic/elasticsearch#137822! So I'd just wait until the Elasticsearch PR is merged first and then in this PR just link to the reference doc for reference about all available settings, with something like:

For a full list of available strong verification settings for remote clusters, refer to xxxxx.

[gmjehovich]

Overall instructions were clear and didn't see anything major missing. I left some comments with some suggestions and questions

[gmjehovich]

Wondering if this should be more specific by including the actual output/examples from logs

[jfreden]

Looks like there isn't one right now, my bad. Thinking more about this, having an info log for every successful request will be too verbose so I'll just remove this.

[gmjehovich]

Wondering if this should link back to the section above that shows how to use elasticsearch-certutil

[gmjehovich]

Also wondering: for ECH if a customer wants to use their own certs, they would need to use the bundles feature. Where would we document instructions for that?

[jfreden]

Yes, good point! I think that should be specifically for the ECH documentation. I'll investigate where we should add that. Thanks!

[eedugon]

@gmjehovich , @jfreden , I've been working lately in all docs related with remote clusters, for all deployment types (ECH, ECE, ECK, self-managed). Feel free to ping me in private and we can review where to add information related with this PR (Strong verification configuration) for other deployment types, and I could do that in another PR.

[gmjehovich]

This paragraph could be clearer about which cluster each component lives on, and it should mention how trust is established. The phrase "provide a trusted certificate" doesn't explain that the remote cluster validates trust based on its configured certificate authorities.

Something like this:

"Cross-cluster API keys can be configured with strong verification to provide an additional layer of security. To enable this feature, a cross-cluster API key is created on the remote cluster with a certificate identity pattern that specifies which certificates are allowed to use it. The local cluster must then sign each request with its private key and include a certificate whose subject Distinguished Name (DN) matches the pattern. The remote cluster validates both that the certificate is trusted by its configured certificate authorities and that the certificate's subject matches the API key's identity pattern."

This makes it explicit that

1. The API key lives on the remote cluster
2. Trust is validated against the remote cluster's CA configuration

[gmjehovich]

Also consider mentioning that this is configured per-connection (each remote cluster alias can have different signing certs) and per-API-key (each API key can have different certificate identity patterns). This helps users understand the granularity of the feature.

[gmjehovich]

LGTM from my side, my feedback has been addressed. Thank you!

I will hold off on the formal approval while you and @shainaraskas finalize the decision on which settings to include. I personally prefer keeping docs focused with links to advanced stuff if possible, but happy with whatever you two decide.

[eedugon]

@jfreden : All this looks great to me.

I haven't started with the refinement and restructure of the self-managed remote clusters docs (currently working on ECK use cases for remote clusters), so adding this information in the way it's being added in these docs for self-managed looks ok.

What we need to clarify is the implications of this feature when the local cluster is on ECE, ECK or ECH, so we can properly update and link from the affected documents, but we can do that in a separate thread / issue or in private before raising an issue for an extra PR.

One quick comment, @jfreden and @gmjehovich : Recently we included (and refined) the main descriptions and presentations of security models (API key + the deprecated TLS cert based authentication) to this document: https://www.elastic.co/docs/deploy-manage/remote-clusters/security-models. The plan is for that document to be the source of truth when it comes to explaining API key authentication (for all deployment types).

Eventually (soon) I was planning to remove redundant sections of the self-managed document that you are updating (deploy-manage/remote-clusters/remote-clusters-api-key.md) and linking to the mentioned doc.

Anyway as I said, if you are ok with me moving a bit part of the information (feature description) to another doc in the near future (and proper cross-linking), I'm totally ok with the current proposal.

[eedugon]

LGTM!
I've added 2 minor suggestions with a couple of links.
The second link will only work after merging elastic/elasticsearch#137822

I'd also suggest to put the entire new sub-section just before configure users and roles, and leave that at the end of the doc.

And as we talked in private I could add in a next iteration some comments about other types of deployments, like using bundles for ECH or ECE.

[bytebilly]

LGTM, I left a couple of comments

[bytebilly]

I see that the improved functionlity is named "strong verification", while we (I?) generally use "strong identity verification" elsewhere.

Was there a discussion about that?

[jfreden]

No, this is an oversight from me. I'll update it! Thanks for reviewing!

[bytebilly]

I would explicitly mention that the private key can be encrypted and the password must be stored securely in the keystore. People may be wondering otherwise.