[Serve] gRPC proxy server is not respecting Ray's gRPC options

[GeneDer]

What happened + What you expected to happen

Re: https://discuss.ray.io/t/received-message-larger-than-max-error-when-sending-request-with-grpc/13599

When user submits a large payload, even when configured client correctly, is not able to send it through Serve's gRPC proxy.

Versions / Dependencies

2.9.0

Reproduction script

# client.py
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage


url = "localhost:9000"
options = [
    ('grpc.max_message_length', 1024 * 1024 * 1024),
    ('grpc.max_send_message_length', 1024 * 1024 * 1024),
    ('grpc.max_receive_message_length', 1024 * 1024 * 1024),
]
channel = grpc.insecure_channel(url, options=options)

print("\n\n____________test calling __call__ ____________")
stub = UserDefinedServiceStub(channel)
test_in = UserDefinedMessage(
    name="genes" * 840_000,
    num=88,
    foo="bar",
)
metadata = (
    ("application", "app3"),
)
response, call = stub.__call__.with_call(request=test_in, metadata=metadata)
print(call.trailing_metadata())  # Request id is returned in the trailing metadata
print("Output type:", type(response))  # Response is a type of UserDefinedResponse
print("Full output:", response)
print("Output greeting field:", response.greeting)
print("Output num_x2 field:", response.num_x2)

Issue Severity

High: It blocks me from completing my task.