[META] Secure communication between OpenSearch and Extensions

[peternied]

OpenSearch and Extensions are establishing connections to one another. The Security Plugin provides TLS for OpenSearch, OpenSearch <-> Extensions connections should be established via TLS. Handshaking between both endpoints need to be robust to ensure man-in-the-middle attacks are mitigated.

Open Questions:

• Do TLS settings need to be refactored into the OpenSearch cluster, such as ciphers/protocols?
• Can outbound requests from extensions be identifiable as from a given extension in a tamperproof way?
• If an OpenSearch cluster has TLS disabled, are extensions allowed to connect?
• Are preventative measured needed at the connection level for authorization, or will a Subject association be enough?

[peternied]

@cwperks I know you were looking into this space, but I figured I would file this to capture the overall effort.

[cwperks]

Thank you Peter. I linked this issue to the open PR in the SDK repo: opensearch-project/opensearch-sdk-java#619

The extension bootstrap process utilizes the transport layer to register the extension, settings and actions on node bootstrap and is also used for RestSentToExtensionAction to forward an action to an extension. Transport-layer communication cannot currently happen with an extension and an opensearch node with the security plugin installed and enabled. The Security plugin does not permit turning off transport-layer encryption so extensions need a way to setup TLS. The PR I opened up in the SDK repo takes a lot of the Security plugin TLS setup with the DefaultSecurityKeyStore, but removes anything having to do with http since the extension is not acting as an HTTP server itself.

One of the challenges in implementing the PR was that it started failing the handshake process with the security plugin installed. Inside of the ExtensionsInitRequestHandler it calls on connectToNode which initiates a handshake.

The security plugin forbids this from happening because it gets a transport request from something that isn't a node in the cluster. The companion PRs I linked to in the SDK PR solve the issue of the handshake where the extension can identify itself using a threadcontext header and the security plugin can verify the header's presence and also check with the ExtensionManager that its aware of the extension based on its unique id.

[reta]

I have somewhat developer friendly view on that:

• If an OpenSearch cluster has TLS disabled: I think we should allow the extensions to communicate over plain HTTP, I suspect that would be the primary development flow
• Are preventative measured needed at the connection level for authorization: the first option which comes to mind is to use mTLS to establish mutual trust between extensions and other cluster nodes (but that would complicate the deployments due to certificate management)
• Can outbound requests from extensions be identifiable as from a given extension in a tamperproof way: that's a tough one I think, extensions are free to do whatever they want, I do not know how we could limit that :(
• Do TLS settings need to be refactored into the OpenSearch cluster, such as ciphers/protocols: probably, so the extensions and nodes could use the secure settings and protocols

[peternied]

Thanks for your thoughts @reta

[scrawfor99]

[Triage] This issue is part of the ongoing work with the extensions project.

[expani]

When TLS is enabled, OpenSearch Core ( Sec Plugin ) should enforce a cipher suite and not downgrade based on the one provided by Extension.
This ensures that untrusted extensions don't use weaker TLS