ALPN SNI Proxy #7524

smallinsky · 2021-07-12T09:52:06Z

ALPN SNI proxy service listener

ALPN SNI proxy service allows to handle incoming TLS connections and route received connections base on ServerName (SNI) NextProtos (ALPN) values to proper Teleport Proxy service.

CLI UX changes

tsh proxy ssh command that can be used as openSSH ProxyCommand in order to wrap ssh protocol into TLS connection and connect to a remote proxy.
```
ssh -o "ForwardAgent yes" \
-o "ProxyCommand tsh proxy ssh --user=%r %h:%p" \
user@node.example.com
```
tsh proxy db [-p port] example starts local db proxy and wraps db protocol into TLS connection and establishes the connection to remote proxy.

Other Changes:

tsh db connect example (introduce in #7213) will detect if the remote proxy supports ALPN SNI listener and under the hood will spawn on demand local db proxy that will wrap db protocol into TLS connection and connect to remote proxy.
Extended KUBECONFIG file generated by tsh kube login command by cluster.server field use to set SNI value by kubectrl tool.
Extended Proxy Ping response proxy message by alpn_sni_listener_enabled filed.
Introduced a new alpn_sni_listener_enabled field in the tsh user profile to distinguish if proxy support ALPN SNI listener.

Backward compatibility

Client:

tsh proxy client uses proxy.alpn_sni_listener_enabled proxy response ping field to distinguish if remote Teleport Proxy supports TLS SNI ALPN listener. If ALPN SNI Proxy support is detected by the proxy client the traffic is wrapped into TLS and send to Web port where ALPN SNI Teleport Proxy service listen on.
If former version of a client is used to establish a connection to the remote Teleport Proxy service

Server

In order to support former proxy clients Teleport Proxy by default exposed Teleport Proxy Services in an unchanged way. Additional TLS SNI Teleport Proxy service is exposed on Web port.

Load Test IoT (connection through teleport proxy reverse tunnel)

Load Test 1K nodes teleport 7.0.0-beta.5 (For reference)

Soak 1K nodes teleport 7.0.0-beta.5 (For reference)

tsh bench --interactive --duration=30m root@loadtest-5ddf88b955-w7sbn ps uax

* Requests originated: 17918
* Requests failed: 110
* Last error: ssh: unexpected packet in response to channel open: <nil>

Histogram

Percentile Response Duration
---------- -----------------
25         3745 ms
50         8223 ms
75         8367 ms
90         8679 ms
95         9071 ms
99         13039 ms
100        59455 ms

tsh bench --duration=30m root@loadtest-5ddf88b955-tq5d4 ls


* Requests originated: 17918
* Requests failed: 104
* Last error: connection error: desc = "transport: authentication handshake failed: EOF"

Histogram

Percentile Response Duration
---------- -----------------
25         3743 ms
50         8223 ms
75         8367 ms
90         8679 ms
95         9039 ms
99         12663 ms
100        60255 ms

Load Test 1K nodes SNI proxy:

Soak 1K nodes teleport SNI Proxy

tsh bench --duration=30m root@loadtest-fbdb47d9f-zgs64 ls


* Requests originated: 17925
* Requests failed: 81
* Last error: ssh: unexpected packet in response to channel open: <nil>

Histogram

Percentile Response Duration
---------- -----------------
25         3807 ms
50         7455 ms
75         7507 ms
90         7735 ms
95         8839 ms
99         14551 ms
100        60607 ms

tsh bench --interactive --duration=30m root@loadtest-fbdb47d9f-xdm6v ps uax


* Requests originated: 17923
* Requests failed: 89
* Last error: ssh: unexpected packet in response to channel open: <nil>

Histogram

Percentile Response Duration
---------- -----------------
25         3815 ms
50         7755 ms
75         7811 ms
90         8035 ms
95         9079 ms
99         14655 ms
100        60479 ms

r0mant

@smallinsky Don't have a lot of other feedback, just a few remaining cleanups/questions.

api/client/webclient/webclient.go

r0mant · 2021-08-26T15:55:50Z

integration/integration_test.go

+		// add proxy as a backend second time to prevent situations where there are only 2 backend and internal
+		// reverse tunnel dialer makes 2 calls to connect to the tunnel (ProxyPing, factual Dial call). In that
+		// situations the second lb backend reverse tunnel address is actually never dialed.
+		lb.AddBackend(*utils.MustParseAddr(net.JoinHostPort(Loopback, strconv.Itoa(proxyReverseTunnelPort))))


Sorry I somehow missed this during previous passes. @smallinsky I'm not sure I fully understand why we're adding a second B backend here. How does it help with the test and the proxy?

lib/reversetunnel/transport.go

lib/reversetunnel/agent.go

lib/srv/alpnproxy/conn.go

r0mant · 2021-08-26T16:17:43Z

lib/srv/alpnproxy/proxy.go

+		return nil, nil, trace.Wrap(err)
+	}
+
+	// Following TLS handshake fails on the server side with error: "no certificates configured" after server


Ohh I see now, it's cause you're using that fake tlsConn above. A couple of thoughts:

Can this failed fake handshake somehow affect the client and/or the future "real" handshake?

Can your fake tlsConn return a proper tls.Config so this handshake succeeds also? We have p.cfg.TLSConfig here?

tool/tsh/db.go

tool/tsh/tsh.go

r0mant · 2021-08-26T16:30:51Z

lib/service/service.go

@@ -2587,6 +2588,14 @@ func (process *TeleportProcess) setupProxyListeners() (*proxyListeners, error) {
 				listeners.web = listener
 			}
 		}
+
+		// Even if web service API was disabled create a web listener used for ALPN/SNI service as the master port
+		if cfg.Proxy.DisableWebService && !cfg.Proxy.DisableTLS && listeners.web == nil {


Nit: Isn't cfg.Proxy.DisableWebService basically a redundant condition here?

r0mant · 2021-08-26T16:34:03Z

lib/service/service.go

@@ -3030,6 +2948,24 @@ func (process *TeleportProcess) initProxyEndpoint(conn *Connector) error {
 		if err != nil {
 			return trace.Wrap(err)
 		}
+
+		if alpnRouter != nil && !cfg.Proxy.DisableDatabaseProxy {


I don't think we'll ever be able drop other listeners since we'll still need to have support for separate listeners, right? But ok, that's fair.

r0mant

lgtm once we address the issue with the hacky integration test

@awly / @nklaassen: since you guys looked at this before, could you give this another look so we can get a second approval? Thanks.

nklaassen

Looks alright to me overall, my remaining question is related to the issue with multiple proxies behind a load balancer and clients choosing the port based on the wrong ping response. Would this be a serious problem for people turning on this feature or is there a workaround? Update all proxies at once?

smallinsky · 2021-09-02T15:44:54Z

@nklaassen

Looks alright to me overall, my remaining question is related to the issue with multiple proxies behind a load balancer and clients choosing the port based on the wrong ping response. Would this be a serious problem for people turning on this feature or is there a workaround? Update all proxies at once?

The proxy update should work the proxy behind LB should have same config the issue is rated to proxy stagnation when even number of proxies are configured behind reverse tunnel LB. I have create a separate PR with fix for this case:
#8106

smallinsky force-pushed the smallinsky/alpnsniproxy branch 9 times, most recently from cf67e96 to 28131a2 Compare July 13, 2021 09:20

smallinsky added 2 commits July 14, 2021 22:18

ALPN SNI Proxy

cf6775d

revert debug

ac9e265

smallinsky force-pushed the smallinsky/alpnsniproxy branch from 00ceaf0 to ac9e265 Compare July 14, 2021 20:19

smallinsky added 8 commits July 15, 2021 11:36

update

c47f976

update

95216de

update

3d71842

set SNI value

378778a

logs

1b076ed

update

d32b3ad

update comments

703c2d3

update

598573c

smallinsky force-pushed the smallinsky/alpnsniproxy branch from 7196bcd to 598573c Compare July 15, 2021 18:44

smallinsky marked this pull request as ready for review July 15, 2021 19:58

smallinsky requested review from klizhentas, r0mant and russjones as code owners July 15, 2021 19:58

smallinsky requested review from awly, andrejtokarcik and nklaassen July 15, 2021 19:59

fix kube

8953dec

smallinsky force-pushed the smallinsky/alpnsniproxy branch from 32f5e70 to 8953dec Compare July 16, 2021 11:03

smallinsky force-pushed the smallinsky/alpnsniproxy branch from adfeeaa to 8c8ffa7 Compare August 23, 2021 10:25

smallinsky added 3 commits August 23, 2021 15:42

Remove proxy sync call

4eb6595

Fix err race

b6695a4

Move mkLocalProxy

db8dbb3

smallinsky requested review from nklaassen, awly and r0mant August 23, 2021 15:20

update

8f7529c

r0mant reviewed Aug 26, 2021

View reviewed changes

Update

dff400a

smallinsky force-pushed the smallinsky/alpnsniproxy branch from 26f5b59 to f69576b Compare August 30, 2021 13:40

update

3ccddeb

smallinsky force-pushed the smallinsky/alpnsniproxy branch from f69576b to 3ccddeb Compare August 30, 2021 14:27

r0mant approved these changes Aug 31, 2021

View reviewed changes

nklaassen approved these changes Sep 1, 2021

View reviewed changes

smallinsky requested a review from Joerger September 2, 2021 13:06

Add support for ALPN SNI proxy teleport-auth protocol (#7991)

8550ac1

smallinsky requested a review from alex-kovoy as a code owner September 3, 2021 13:40

smallinsky added 6 commits September 3, 2021 15:44

Merge branch 'master' into smallinsky/alpnsniproxy

e148c20

Update

dcb86f7

Fix LB proxy dial issue (#8106)

72c3ffb

Merge branch 'master' into smallinsky/alpnsniproxy

39b027d

Fix err chan size

898f401

Merge branch 'master' into smallinsky/alpnsniproxy

66cf7e7

smallinsky merged commit c142b65 into master Sep 13, 2021

smallinsky deleted the smallinsky/alpnsniproxy branch September 13, 2021 09:54

zmb3 pushed a commit that referenced this pull request Sep 23, 2021

ALPN SNI Proxy (#7524)

1d0152a

pschisa mentioned this pull request Dec 8, 2021

Merge SSH reverse tunnel and SSH proxy port #4843

Closed

programmerq mentioned this pull request Mar 31, 2023

Prompt for Teleport credentials when using an OpenSSH client #1183

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ALPN SNI Proxy #7524

ALPN SNI Proxy #7524

smallinsky commented Jul 12, 2021 •

edited

Loading

r0mant left a comment

r0mant Aug 26, 2021

r0mant Aug 26, 2021

r0mant Aug 26, 2021

r0mant Aug 26, 2021

r0mant left a comment

nklaassen left a comment

smallinsky commented Sep 2, 2021

ALPN SNI Proxy #7524

ALPN SNI Proxy #7524

Conversation

smallinsky commented Jul 12, 2021 • edited Loading

ALPN SNI proxy service listener

CLI UX changes

Other Changes:

Backward compatibility

Client:

Server

Load Test IoT (connection through teleport proxy reverse tunnel)

Load Test 1K nodes teleport 7.0.0-beta.5 (For reference)

Soak 1K nodes teleport 7.0.0-beta.5 (For reference)

Load Test 1K nodes SNI proxy:

Soak 1K nodes teleport SNI Proxy

r0mant left a comment

Choose a reason for hiding this comment

r0mant Aug 26, 2021

Choose a reason for hiding this comment

r0mant Aug 26, 2021

Choose a reason for hiding this comment

r0mant Aug 26, 2021

Choose a reason for hiding this comment

r0mant Aug 26, 2021

Choose a reason for hiding this comment

r0mant left a comment

Choose a reason for hiding this comment

nklaassen left a comment

Choose a reason for hiding this comment

smallinsky commented Sep 2, 2021

smallinsky commented Jul 12, 2021 •

edited

Loading