gRPC response body and trailers stripped through tunnel even with TLS+ALPN+h2 origin (cloudflared 2026.3.0)

[laurentpellegrino]

gRPC response body and trailers stripped through Cloudflare Tunnel even with TLS+ALPN+h2 origin (cloudflared 2026.3.0)

Summary

Through a fully-documented gRPC-tunnel configuration (origin uses TLS, advertises h2 via ALPN, originRequest.http2Origin: true, noTLSVerify: true), responses lose their body and trailers between the origin and the external client. The same origin returns full responses when reached directly inside the cluster — only the path through the tunnel is affected.

Environment

• cloudflared: 2026.3.0 (also reproduced on 2025.8.1)
• Tunnel type: locally-managed (config.yaml, not Dashboard)
• Origin: nginx 1.27 with grpc_pass to a Tonic (Rust) gRPC server, listening on TLS:50443 with self-signed cert and h2 ALPN
• Client: grpcurl 1.9.1 and grpc-java-netty 1.80.0
• Tunnel transport: QUIC (default), 4 connections to fra03/fra16/fra17/fra18 edge

cloudflared config

tunnel: <id>
credentials-file: /etc/cloudflared/creds/credentials.json
metrics: 0.0.0.0:2000
no-autoupdate: true
ingress:
  - hostname: grpc.example.com
    service: https://grpc-proxy.namespace.svc.cluster.local:50443
    originRequest:
      http2Origin: true
      noTLSVerify: true
  - service: http_status:404

The origin nginx config:

events { worker_connections 1024; }
http {
  client_max_body_size 64m;
  grpc_read_timeout 600s;
  grpc_send_timeout 600s;
  server {
    listen 50443 ssl;
    http2 on;
    ssl_certificate     /etc/nginx/tls/tls.crt;
    ssl_certificate_key /etc/nginx/tls/tls.key;
    location / {
      grpc_pass grpc://tonic-backend.namespace.svc.cluster.local:50051;
      grpc_set_header authorization $http_authorization;
    }
  }
}

Symptom

External client through Cloudflare:

$ curl -s -o /dev/null -w "status=%{http_code} ct=%{content_type} size=%{size_download}\n" \
       --http2 -X POST 'https://grpc.example.com/some.Service/Method' \
       -H 'content-type: application/grpc' -H 'te: trailers'
status=200 ct=application/grpc size=0

$ grpcurl -import-path /p -proto x.proto -H "authorization: Bearer …" \
          -d '{"lat":48.8566,"lng":2.3522}' grpc.example.com:443 some.Service/Method
ERROR:
  Code: Internal
  Message: server closed the stream without sending trailers

The HTTP/2 response status is 200 with the correct content-type: application/grpc, but the body is empty (0 bytes) and no grpc-status / grpc-message trailers are sent. The gRPC client treats this as Internal: server closed the stream without sending trailers (or Received unexpected EOS on empty DATA frame from server in grpc-java).

Same origin works directly

kubectl port-forward to the in-cluster proxy Service, then identical grpcurl call locally:

$ grpcurl -insecure -import-path /p -proto x.proto -H "authorization: Bearer …" \
          -d '{"lat":48.8566,"lng":2.3522}' localhost:50443 some.Service/Method
{
  "continentCode": "EU",
  "countryCode": "FR"
}

Full response body, proper grpc-status: 0 trailer.

In-cluster pod (different from the cloudflared pod) calling the same Service DNS name with TLS:

$ kubectl run … --image fullstorydev/grpcurl:v1.9.1 \
    -- -insecure -import-path /proto -proto geocoding.proto … \
       grpc-proxy.namespace.svc.cluster.local:50443 some.Service/Method
{ "continentCode": "EU", "countryCode": "FR" }

So the proxy is correctly emitting HTTP/2 responses with body + trailers. Confirmed in nginx access log:

… "POST /some.Service/Method HTTP/2.0" status=200 sent=71 upstream=…

What we ruled out

• Origin not speaking HTTP/2: nginx access log shows it received the request as HTTP/2.0 from cloudflared and emitted a 71-byte response. The in-cluster grpcurl test confirms the proxy returns full bodies + trailers.
• http2Origin: true not taking effect: with this config (HTTPS origin + the flag), nginx access log explicitly shows HTTP/2.0. (For comparison: with service: http://… + same flag, nginx access log shows HTTP/1.1 from cloudflared, matching the documented "HTTP/2 origin requires SSL" behavior — that's a separate finding, not the one in this report.)
• Origin failing mid-stream: backend returns proper responses to in-cluster clients; nginx is healthy, no errors.
• gRPC zone setting: API token doesn't have permission to read it, but the response status is 200 (not 403 Forbidden as the docs say a disabled zone returns).
• cloudflared version: same behavior on 2025.8.1 and 2026.3.0.

Hypothesis

cloudflared (or the Cloudflare edge over the tunnel transport) is dropping HTTP/2 DATA frames and/or HEADERS-with-trailers when proxying gRPC responses back to the client. The headers (:status, content-type: application/grpc) are preserved but DATA / trailers are not.

Repro steps

1. Deploy a Tonic gRPC server (or any HTTP/2-prior-knowledge server) on :50051 cleartext.
2. Deploy nginx with the config above (grpc_pass to the Tonic backend, listen TLS+h2 on :50443).
3. Configure cloudflared to point at https://nginx:50443 with originRequest.http2Origin: true and noTLSVerify: true.
4. From outside the cluster, run grpcurl against the public hostname.
5. Observe: status 200, content-type application/grpc, but empty body and no trailers.
6. From inside the cluster (port-forward or in-cluster pod), run the same grpcurl against the same nginx Service.
7. Observe: full response body + grpc-status: 0 trailer.

What we'd like

Either confirmation that this is a known bug with a fix in flight, or guidance on what we should configure differently. If the answer is "this can't currently work for HTTP/2-prior-knowledge origins," that's worth documenting on the gRPC docs page (which currently implies any HTTP/2 origin should work).