Websockets do not work with app_service if the upstream web server is using HTTP/2

[webvictim]

Expected behavior: Teleport's application access service should be able to connect to an upstream server using HTTP/2, and using websockets over this connection should work.

Current behavior: Using Teleport application access to connect to an upstream server speaking HTTP/2 and using websockets does not work. An Unable to read websocket upgrade response: malformed HTTP response message appears over in the app_service logs whenever a websocket connection is attempted and the browser is unable to open the socket:

Nov 14 20:03:04 k700 teleport[1047682]: 2022-11-14T20:03:04Z             INFO Round trip: GET /discovery/api/v1/about, code: 200, duration: 8.678635ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:04 k700 teleport[1047682]: 2022-11-14T20:03:04Z             INFO Round trip: GET /lidar-hub/api/v1/about, code: 200, duration: 12.96765ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:04 k700 teleport[1047682]: 2022-11-14T20:03:04Z             ERRO Unable to read websocket upgrade response: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01" forward/fwd.go:309
Nov 14 20:03:04 k700 teleport[1047682]: 2022-11-14T20:03:04Z             INFO Round trip: GET /perception/api/v1/telemetry, code: 200, duration: 4.053955ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:04 k700 teleport[1047682]: 2022-11-14T20:03:04Z             INFO Round trip: GET /perception/api/v1/sensor, code: 200, duration: 3.568626ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:05 k700 teleport[1047682]: 2022-11-14T20:03:05Z             ERRO Unable to read websocket upgrade response: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01" forward/fwd.go:309
Nov 14 20:03:05 k700 teleport[1047682]: 2022-11-14T20:03:05Z             INFO Round trip: GET /perception/api/v1/telemetry, code: 200, duration: 3.860804ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:05 k700 teleport[1047682]: 2022-11-14T20:03:05Z             INFO Round trip: GET /perception/api/v1/point_zones, code: 200, duration: 4.412703ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:05 k700 teleport[1047682]: 2022-11-14T20:03:05Z             INFO Round trip: GET /event/api/v1/event_zones/, code: 200, duration: 4.107315ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:05 k700 teleport[1047682]: 2022-11-14T20:03:05Z             INFO Round trip: GET /perception/api/v1/sensor, code: 200, duration: 3.654156ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:06 k700 teleport[1047682]: 2022-11-14T20:03:06Z             ERRO Unable to read websocket upgrade response: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01" forward/fwd.go:309
Nov 14 20:03:06 k700 teleport[1047682]: 2022-11-14T20:03:06Z             INFO Round trip: GET /perception/api/v1/telemetry, code: 200, duration: 4.031548ms tls:version: 304, tls:resume:true, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:06 k700 teleport[1047682]: 2022-11-14T20:03:06Z             INFO Round trip: GET /perception/api/v1/sensor, code: 200, duration: 3.84757ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182
Nov 14 20:03:07 k700 teleport[1047682]: 2022-11-14T20:03:07Z             ERRO Unable to read websocket upgrade response: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01" forward/fwd.go:309
Nov 14 20:03:07 k700 teleport[1047682]: 2022-11-14T20:03:07Z             INFO Round trip: GET /event/api/v1/event_zones/, code: 200, duration: 5.208286ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:6f75737465722e74656c65706f72742e7368.teleport.cluster.local forward/fwd.go:182

Disabling HTTP/2 on the upstream web server worked around this issue - websocket upgrades work fine with application access when using HTTP/1.1.

cc @tigrato who had theories about this being related to the reuse of tls.Config internally.

Bug details:

• Teleport version: 10.3.6

[Clinton-Groundspeed]

I'm experiencing the same error when attempting to use Application Access service (Teleport Cloud) to connect to an HTTP/2 app using WebSockets.

@WilliamLoy - This is a critical internal app. Could you please research this internally? Thanks!

[gs-rickchristy]

bump

[Clinton-Groundspeed]

FYI, this issue (exact same error) also occurs with version 11.1.1 - tested with a standalone Enterprise cluster as well:

2022-12-06T20:49:18Z ERRO             Unable to read websocket upgrade response: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x80\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\x7f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01" forward/fwd.go:309

If it helps, the web app I am attempting to configure Application Access for is a MeteorJS app hosted on EKS through Istio. Please let me know if I can provide any more details. Thanks!