Proposal: TLS authentication/encryption support for gRPC

[longquanzheng]

Is your feature request related to a problem? Please describe.
Similar to this closed issue #2018
We should provide TLS options/configuration for clients talking to Cadence server.

Proposed Solution
There could be two steps:

• For external clients talking to server, It seems pretty straightforward if following https://grpc.io/docs/guides/auth/ This is highest priority.
• For Cadence server nodes talking to each others and cross cluster/XDC also require to configure private key to Cadence server

Additional context
This is only for authentication.
Authorization is in #3920 and we are working on JWT/OAuth based authorization.

[tfcace]

Hi @longquanzheng, the definition of "clients talking to Cadence server" may be ambiguous.
For starters, I came here from our Slack discussion over gRPC and AuthN, and expected to also see something relating to AuthN in cadence-web (falls under client to Cadence server).

Additionally, your second bullet confuses me, as AFAIK, clients communicate with the Cadence server over the frontend service at 7933, yet you state that this port will continue to use TChannel. In that regard, I would have expected Workers to also be able to authenticate against the frontend.

Thanks in advance :)

p.s. Regarding TLS, who / what is responsible for generating the certificates?
I think that once the proposal is finalized, it would be crucial to have a proper guide on how to setup AuthN in new and existing clusters.

[longquanzheng]

@tfcace thanks for the feedback.

Yeah, the "clients" here include cadence-web and SDK starter/worker. As long as they use gRPC library to send requests with TLS, server will accept the requests.

To understand the second point -- All services including Frontend/Matching/History/Worker will continue to use TChannel for intercommunication. As we don't have a replacement for Ringpop yet, which is implemented based on TChannel. Therefore each service will have to expose two port -- a gRPC port and a TChannel port. External traffic from "clients" -- worker/starter/web will only need to talk to the gRPC port, which TChannel port is for internal purpose.

For certificates -- It's all up to users to manage certificates(how to create them).

Option 1 in the proposal here is that Cadence server will only need to be configured with public key for validation and doesn't need private keys -- if the TChannel port can continue to be used for serving traffic, instead of just ringpop communication. However, if the team decided to TChannel not serving any traffic, then private key will also be required to be configured with server.

Option 2 in the proposal is to also configure private key.