Add the ability to specify metadata

[amrmahdi]

We have a use case where we need to send custom metadata to the server for the health check. This change enables specifying metadata on the command line.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[amrmahdi]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[ahmetb]

Can you elaborate your use case further? What do you use metadata for in this case?
I am trying to see if this could be a feature used by someone other than you.

[amrmahdi]

It is for a proxy service, where we will run the health check request through the proxy to ensure service health. The service needs metadata for routing requests.

[ahmetb]

So let me rephrase what I understood and you tell me if I got it right: You have multiple gRPC server instances behind a load balancer (what you referred as "proxy") and you use metadata to tell the proxy to route the request to a particular server instance.

If I got it right, this sounds like an anti-pattern as you shouldn't be going through a proxy/LB to check health. You should be directly target the instances.

For that reason, I'd love to know what sort of environment you're running on (VMs, Kubernetes, something else?) as this is probably not the right way to do health checks on that platform.

Again sorry for digging deep, but for any feature we add, we need to make sure there's a valid use case that we can get behind as a best practice.

[amrmahdi]

We are running on K8s. The requests are not going through an LB. The health check is run per pod (service instance).

health probe -> proxy service instance -> health service

The health service will always be returning Serving and we will rely on the request being successful that the proxy is alive.

And the metadata is needed to control/modify the behavior of the proxy service.

Hope this clarifies the scenario.

[amrmahdi]

Hi @ahmetb, any other concerns ? Don't you think metadata is generic enough that could be used for different scenarios ?

[ahmetb]

Can you give some concrete examples? I am still unclear.

Why can’t your Kubernetes health check directly hit the grpc service port it’s running on inside the Pod? I still don’t get what this proxy instance is and how it comes to play.

[amrmahdi]

Maybe I wasn't clear enough. The service we are developing and trying to health check in this scenario is a proxy service. And the way we are checking the service health is running a request to the health check service though our service (the proxy).

[ahmetb]

Sorry, it's still not clear enough. Any photos of drawings, or diagrams are appreciated.

What is proxy service? Is it a gRPC service itself, or is it simply a http/2 proxy?

Why are you simply not using a liveness probe on the proxyservice pod? Does it forward the request to another grpc service because the pod is a proxy itself? I still don't get your situation fully.

[amrmahdi]

It is a GRPC proxy deployed as a sidecar as shown in the diagram above. The GRPC proxy forwards requests to other GRPC services based on metadata. And to test if that the proxy is alive we will simply forward a request to the health check service (which the proxy service is implementing) to make sure it is alive. The health probe command will be similar to:

grpc-health-probe --addr <proxy_address> --metadata forward_to=<health_service_address> --metadata foo=bar

And since the proxy needs additional metadata other than the forwarding address the http_proxy option is not enough in our case.

[ahmetb]

Closing due to inactivity.