tailscale serve can't provide tls termination for grpc

[asg0451]

What is the issue?

i have a grpc service on kubernetes with a tailscale sidecar, and i want to expose it to the internet with a Funnel.

This is the serving setup;

# tailscale --socket=/tmp/tailscaled.sock serve https:443 / http://127.0.0.1:8080; tailscale --socket=/tmp/tailscaled.sock funnel 443 on
...
 # tailscale --socket=/tmp/tailscaled.sock serve status

# Funnel on:
#     - https://XX.XXXXX.ts.net

https://XX.XXXXX.ts.net (Funnel on)
|-- / proxy http://127.0.0.1:8080

(if i do a serve with regular tcp, or if i connect through tailscale directly to port 8080, the service works fine)

on my client, when i try and send a request, it returns:

Error: status: Unavailable, message: "grpc-status header missing, mapped from HTTP status code 502", details: [], metadata: MetadataMap { headers: {"content-length": "0", "date": "Sun, 16 Apr 2023 18:27:57 GMT"} }

and i see in the sidecar logs:

 2023/04/16 18:27:57 http: proxy error: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x04\x00\x10\x00\ │
│ x00\x00\x05\x00\x00@\x00\x00\x06\x01\x00\x00\x00"        

does tailscale serve https termination break with grpc/http2?

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Kubernetes/Ubuntu

Tailscale version

1.38.4 tailscale commit: 043a345-dirty go version: go1.20.3-tsddff070

Other software

No response

Bug report

BUG-8563b715f0eed5a6fdcff86867a434981502c65eb5a5071bb0693bb8e3f38793-20230416183258Z-2c89c5881f07d89d

[shayne]

Hi there 👋 , thanks for all of the info!

For gRPC to work over the Internet via Funnel, you must wrap it in TLS. Funnel uses the TLS Client Hello message to determine how to route the request.

An example: https://grpc.io/docs/guides/auth/#with-server-authentication-ssltls

[asg0451]

Thanks for the reply! What certfile should I use for Funnel?

…

On Tue, 18 Apr 2023 at 11:23, shayne ***@***.***> wrote: Hi there 👋 , thanks for all of the info! For gRPC to work over the Internet via Funnel, you must wrap it in TLS. Funnel uses the TLS Client Hello message to determine how to route the request. An example: https://grpc.io/docs/guides/auth/#with-server-authentication-ssltls — Reply to this email directly, view it on GitHub <#7893 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBIDEIC3FS5AIS7S6ETKKDXB2WYTANCNFSM6AAAAAAXAJCXME> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Miles Frankel | Software Engineer | Mixpanel

[shayne]

We have a $ tailscale cert command. Here is a link to those docs:
https://tailscale.com/kb/1153/enabling-https/#provision-tls-certificates-for-your-devices

[shayne]

@asg0451, were you able to sort things out? I believe this was an issue in using a non-TLS protocol over Funnel.

[asg0451]

I actually ended up going in a different direction to unblock myself and never looked back. There was also an issue where I couldn't guarantee the k8s ts sidecar would get the name I gave it - - even though the machine was ephemeral, the old one would hang around and on restart it would choose NAME-1,-2 etc. I notice you've fixed this issue in the new ssh session recorder however..

…

On Tue, May 16, 2023, 10:31 PM shayne ***@***.***> wrote: @asg0451 <https://github.com/asg0451>, were you able to sort things out? I believe this was an issue in using a non-TLS protocol over Funnel. — Reply to this email directly, view it on GitHub <#7893 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBIDEJGG6XSNGPMVGGC7UDXGQ2AXANCNFSM6AAAAAAXAJCXME> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shayne]

Thanks for the feedback. I'll close this, but if there's an outstanding bug, please comment! 😄