ISSUE-19673: Support TLS, Authn/z for a function's gRPC server

[sijie]

Original Issue: apache#19673

---

Search before asking

• I searched in the issues and found nothing similar.

Motivation

A function pod (k8s runtime) or process (process runtime) starts a gRPC server that is accessed by the function worker for a number of operations does not currently support TLS, authentication, or authorization. The endpoints exposed can modify the function's state with the following endpoints defined:

service InstanceControl {
    rpc GetFunctionStatus(google.protobuf.Empty) returns (FunctionStatus) {}
    rpc GetAndResetMetrics(google.protobuf.Empty) returns (MetricsData) {}
    rpc ResetMetrics(google.protobuf.Empty) returns (google.protobuf.Empty) {}
    rpc GetMetrics(google.protobuf.Empty) returns (MetricsData) {}
    rpc HealthCheck(google.protobuf.Empty) returns (HealthCheckResult) {}
}

It is known that functions have this weakness, but it might be valuable to support securing this gRPC server.

For example, here are the two places we start the plaintext clients:

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/process/ProcessRuntime.java#L178-L180

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntime.java#L327-L329

Solution

The TLS piece would be pretty easy to implement with gRPC, but the authentication and authorization might be more challenging. The question might be whether granular permissions are required. If they are, perhaps there would be a read role and a write role?

Alternatives

No response

Anything else?

I am not able to work on this now, but I wanted to expose this as a potential feature to see if others are interested in it.

Are you willing to submit a PR?

• I'm willing to submit a PR!