[v13] Proxy Templates update: cluster switching and tsh ssh parity

rfd/0062-tsh-proxy-template.md

@@ -1,5 +1,5 @@
 ---
-authors: Roman Tkachenko (roman@goteleport.com)
+authors: Roman Tkachenko (roman@goteleport.com), Brian Joerger (bjoerger@goteleport.com)
 state: implemented
 ---
 
@@ -17,6 +17,10 @@ prefer to use plain `ssh` client to connect to nodes using jump hosts i.e.
 connecting directly via leaf cluster's proxy rather than going through the
 root cluster's proxy.
 
+## Details
+
+### Proxy switching
+
 Consider the following scenario:
 
 - User has multiple leaf clusters in different regions, for example
@@ -45,6 +49,34 @@ This is not ideal because users need to maintain complex SSH config, update it
 every time a new leaf cluster is added, and use non-trivial bash logic in the
 proxy command to correctly separate node name from the proxy address.
 
+### Cluster switching
+
+Alternatively, consider the following scenario:
+
+- User has multiple leaf clusters, for example `leaf1.us.acme.com`, `leaf2.eu.acme.com`, etc.
+- The leaf cluster do not have public proxies, so all requests must pass through
+  the root cluster proxy.
+- Each leaf cluster has nodes like `node-1`, `node-2`, etc.
+- User wants to log into a root cluster to connect to any node in the trusted cluster.
+
+In order to `ssh root@node01.leaf1.us.acme.com` you can create an SSH config similar to this:
+
+```
+Host *.leaf1.us.acme.com
+    HostName %h
+    Port 3022
+    ProxyCommand ssh -p 3023 %r@root.us.acme.com -s proxy:%h:%p@$(echo %h | cut -d '.' -f2)
+
+Host *.leaf2.eu.acme.com
+    HostName %h
+    Port 3022
+    ProxyCommand ssh -p 3023 %r@root.us.acme.com -s proxy:%h:%p@$(echo %h | cut -d '.' -f2)
+```
+
+This strategy suffers from the same challenges as proxy switching noted above.
+
+### Solution
+
 An ideal state would be where user's SSH config is as simple as:
 
 ```
@@ -57,11 +89,11 @@ Host *.acme.com
 Where `<some proxy command>` is smart enough (and user-controllable) to
 determine which host/proxy/cluster user is connecting to.
 
-## UX
+### UX
 
 The proposal is to extend the existing `tsh proxy ssh` command with support for
-parsing out the node name and proxy address from the full hostname `%h:%p` in
-SSH config.
+parsing out the node name, proxy address, and cluster from the full hostname `%h:%p`
+in SSH config.
 
 Specifically, the syntax for the `<some proxy command>` would look like:
 
@@ -77,10 +109,10 @@ instead of the default behavior of connecting to the proxy of the current
 client profile. This usage of the `-J` flag is consistent with the existing
 proxy jump functionality (`tsh ssh -J`) and [Cluster Routing](https://github.com/gravitational/teleport/blob/master/rfd/0021-cluster-routing.md).
 
-When a template variable `{{proxy}}` is used, the desired hostname and proxy address
-are extracted from the full hostname in the `%r@%h:%p` spec. Users define the
-rules of how to parse node/proxy from the full hostname in the tsh config file
-`$TELEPORT_HOME/config/config.yaml` (or global `/etc/tsh.yaml`). Group captures
+When a template variable `{{proxy}}` is used, the desired hostname, proxy address,
+and cluster are extracted from the full hostname in the `%r@%h:%p` spec. Users define
+the rules of how to parse node/proxy/cluster from the full hostname in the tsh config
+file `$TELEPORT_HOME/config/config.yaml` (or global `/etc/tsh.yaml`). Group captures
 are supported:
 
 ```yaml
@@ -92,9 +124,16 @@ proxy_templates:
 # Example template where nodes have FQDN names like node-1.leaf2.eu.acme.com.
 - template: '^(\w+)\.(leaf2.eu.acme.com):(.*)$'
   proxy: "$2:443"
+# Example template where we want to connect through the root proxy.
+- template: '^(\w+)\.(leaf3).us.acme.com:(.*)$'
+  cluster: "$2"
 ```
 
-Templates are evaluated in order and the first one matching will take effect.
+
+Templates are evaluated in order and the first one matching will take effect. For each
+replace rule set (`cluster`, `proxy`, and `host`), the corresponding cli value will be
+overridden (`--cluster`, `-J`, and `%h:%p`). If `template` and all replace rules are empty,
+the template is invalid.
 
 Note that the proxy address must point to the web proxy address (not SSH proxy):
 `tsh proxy ssh` will issue a ping request to the proxy to retrieve additional
@@ -126,11 +165,18 @@ is equivalent to the following when connecting to `node-1.leaf1.us.acme.com`:
 tsh proxy ssh -J leaf1.us.acme.com:3080 %r@node-1:3022
 ```
 
-### Auto-login
+#### Auto-login
 
 To further improve the UX, we should make sure that `tsh proxy ssh` command
 does not require users to log into each individual leaf cluster beforehand.
 
 Users should be able to login once to their root cluster and use that
 certificate to set up proxy with the extracted leaf's proxy (similar to proxy
 jump scenario).
+
+### Proxy Templates for `tsh ssh`
+
+Proxy Templates can also be used with `tsh ssh` and have feature parity with `ssh`.
+This means that users can go from using a proxy template to `ssh node-1.leaf1.us.acme.com`
+to `tsh ssh node-1.leaf1.us.acme.com` without making any additional change to the template.
+`tsh ssh` will support the same features as `tsh proxy ssh`, including [auto login](#auto-login).

tool/tsh/proxy.go

@@ -72,7 +72,7 @@ func onProxyCommandSSH(cf *CLIConf) error {
 		}
 
 		if len(tc.JumpHosts) > 0 {
-			err := setupJumpHost(cf, tc, *proxyParams)
+			err := setupJumpHost(cf, tc, proxyParams.clusterName)
 			if err != nil {
 				return trace.Wrap(err)
 			}
@@ -89,10 +89,6 @@ type sshProxyParams struct {
 	proxyHost string
 	// proxyPort is the Teleport proxy port.
 	proxyPort string
-	// targetHost is the target SSH node host name.
-	targetHost string
-	// targetPort is the target SSH node port.
-	targetPort string
 	// clusterName is the cluster where the SSH node resides.
 	clusterName string
 	// tlsRouting is true if the Teleport proxy has TLS routing enabled.
@@ -101,11 +97,6 @@ type sshProxyParams struct {
 
 // getSSHProxyParams prepares parameters for establishing an SSH proxy connection.
 func getSSHProxyParams(cf *CLIConf, tc *libclient.TeleportClient) (*sshProxyParams, error) {
-	targetHost, targetPort, err := net.SplitHostPort(tc.Host)
-	if err != nil {
-		return nil, trace.Wrap(err)
-	}
-
 	// Without jump hosts, we will be connecting to the current Teleport client
 	// proxy the user is logged into.
 	if len(tc.JumpHosts) == 0 {
@@ -116,8 +107,6 @@ func getSSHProxyParams(cf *CLIConf, tc *libclient.TeleportClient) (*sshProxyPara
 		return &sshProxyParams{
 			proxyHost:   proxyHost,
 			proxyPort:   strconv.Itoa(proxyPort),
-			targetHost:  cleanTargetHost(targetHost, tc.WebProxyHost(), tc.SiteName),
-			targetPort:  targetPort,
 			clusterName: tc.SiteName,
 			tlsRouting:  tc.TLSRoutingEnabled,
 		}, nil
@@ -143,8 +132,6 @@ func getSSHProxyParams(cf *CLIConf, tc *libclient.TeleportClient) (*sshProxyPara
 	return &sshProxyParams{
 		proxyHost:   sshProxyHost,
 		proxyPort:   sshProxyPort,
-		targetHost:  targetHost,
-		targetPort:  targetPort,
 		clusterName: ping.ClusterName,
 		tlsRouting:  ping.Proxy.TLSRoutingEnabled,
 	}, nil
@@ -161,19 +148,19 @@ func cleanTargetHost(targetHost, proxyHost, siteName string) string {
 }
 
 // setupJumpHost configures the client for connecting to the jump host's proxy.
-func setupJumpHost(cf *CLIConf, tc *libclient.TeleportClient, sp sshProxyParams) error {
+func setupJumpHost(cf *CLIConf, tc *libclient.TeleportClient, clusterName string) error {
 	return tc.WithoutJumpHosts(func(tc *libclient.TeleportClient) error {
 		// Fetch certificate for the leaf cluster. This allows users to log
 		// in once into the root cluster and let the proxy handle fetching
 		// certificates for leaf clusters automatically.
-		err := tc.LoadKeyForClusterWithReissue(cf.Context, sp.clusterName)
+		err := tc.LoadKeyForClusterWithReissue(cf.Context, clusterName)
 		if err != nil {
 			return trace.Wrap(err)
 		}
 
 		// We'll be connecting directly to the leaf cluster so make sure agent
 		// loads correct host CA.
-		tc.LocalAgent().UpdateCluster(sp.clusterName)
+		tc.LocalAgent().UpdateCluster(clusterName)
 		return nil
 	})
 }
@@ -229,7 +216,15 @@ func sshProxy(ctx context.Context, tc *libclient.TeleportClient, sp sshProxyPara
 		return trace.Wrap(err)
 	}
 
-	sshUserHost := fmt.Sprintf("%s:%s", sp.targetHost, sp.targetPort)
+	targetHost, targetPort, err := net.SplitHostPort(tc.Host)
+	if err != nil {
+		targetHost = tc.Host
+		targetPort = strconv.Itoa(tc.HostPort)
+	}
+
+	targetHost = cleanTargetHost(targetHost, tc.WebProxyHost(), tc.SiteName)
+
+	sshUserHost := fmt.Sprintf("%s:%s", targetHost, targetPort)
 	if err = sess.RequestSubsystem(ctx, proxySubsystemName(sshUserHost, sp.clusterName)); err != nil {
 		return trace.Wrap(err)
 	}

tool/tsh/tsh.go

@@ -3317,7 +3317,6 @@ func makeClientForProxy(cf *CLIConf, proxy string, useProfileLogin bool) (*clien
 
 	// 1: start with the defaults
 	c := client.MakeDefaultConfig()
-	c.Host = hostUser
 	if cf.TracingProvider == nil {
 		cf.TracingProvider = tracing.NoopProvider()
 	}
@@ -3326,19 +3325,57 @@ func makeClientForProxy(cf *CLIConf, proxy string, useProfileLogin bool) (*clien
 	ctx, span := c.Tracer.Start(cf.Context, "makeClientForProxy/init")
 	defer span.End()
 
-	// ProxyJump is an alias of Proxy flag
-	if cf.ProxyJump != "" {
-		proxyJump := cf.ProxyJump
-		if strings.Contains(cf.ProxyJump, "{{proxy}}") {
-			proxy, host, matched := cf.TshConfig.ProxyTemplates.Apply(c.Host)
-			if !matched {
-				return nil, trace.BadParameter("proxy jump contains {{proxy}} variable but did not match any of the templates in tsh config")
+	// Force the use of proxy template below.
+	useProxyTemplate := strings.Contains(cf.ProxyJump, "{{proxy}}")
+	if useProxyTemplate {
+		// clear proxy jump so it can be overwritten below
+		cf.ProxyJump = ""
+	}
+
+	c.Host = hostUser
+	c.HostPort = int(cf.NodePort)
+
+	// Host may be either %h or %h:%p depending on the command. Proxy
+	// templates match on %h:%p, so we get the full host name here.
+	fullHostName := c.Host
+	if _, _, err := net.SplitHostPort(fullHostName); err != nil {
+		fullHostName = net.JoinHostPort(c.Host, strconv.Itoa(c.HostPort))
+	}
+
+	// Check if this host has a matching proxy template.
+	tProxy, tHost, tCluster, tMatched := cf.TshConfig.ProxyTemplates.Apply(fullHostName)
+	if !tMatched && useProxyTemplate {
+		return nil, trace.BadParameter("proxy jump contains {{proxy}} variable but did not match any of the templates in tsh config")
+	} else if tMatched {
+		if tHost != "" {
+			c.Host = tHost
+			log.Debugf("Will connect to host %q according to proxy template.", tHost)
+
+			if host, port, err := net.SplitHostPort(c.Host); err == nil {
+				c.Host = host
+				c.HostPort, err = strconv.Atoi(port)
+				if err != nil {
+					return nil, trace.Wrap(err)
+				}
 			}
-			proxyJump = strings.ReplaceAll(proxyJump, "{{proxy}}", proxy)
-			c.Host = host
-			log.Debugf("Will connect to proxy %q and host %q according to proxy templates.", proxyJump, host)
 		}
-		hosts, err := utils.ParseProxyJump(proxyJump)
+
+		// Don't overwrite proxy jump if explicitly provided
+		if cf.ProxyJump == "" && tProxy != "" {
+			cf.ProxyJump = tProxy
+			log.Debugf("Will connect to proxy %q according to proxy template.", tProxy)
+		}
+
+		// Don't overwrite cluster if explicitly provided
+		if cf.SiteName == "" && tCluster != "" {
+			cf.SiteName = tCluster
+			log.Debugf("Will connect to cluster %q according to proxy template.", tCluster)
+		}
+	}
+
+	// ProxyJump is an alias of Proxy flag
+	if cf.ProxyJump != "" {
+		hosts, err := utils.ParseProxyJump(cf.ProxyJump)
 		if err != nil {
 			return nil, trace.Wrap(err)
 		}
@@ -3425,7 +3462,6 @@ func makeClientForProxy(cf *CLIConf, proxy string, useProfileLogin bool) (*clien
 	if hostLogin != "" {
 		c.HostLogin = hostLogin
 	}
-	c.HostPort = int(cf.NodePort)
 	c.Labels = labels
 	c.KeyTTL = time.Minute * time.Duration(cf.MinsToLive)
 	c.InsecureSkipVerify = cf.InsecureSkipVerify

tool/tsh/tshconfig.go

@@ -100,14 +100,14 @@ type ProxyTemplates []*ProxyTemplate
 // Apply attempts to match the provided full hostname against all the templates
 // in the list. Returns extracted proxy and host upon encountering the first
 // matching template.
-func (t ProxyTemplates) Apply(fullHostname string) (proxy, host string, matched bool) {
+func (t ProxyTemplates) Apply(fullHostname string) (proxy, host, cluster string, matched bool) {
 	for _, template := range t {
-		proxy, host, matched := template.Apply(fullHostname)
+		proxy, host, cluster, matched := template.Apply(fullHostname)
 		if matched {
-			return proxy, host, true
+			return proxy, host, cluster, true
 		}
 	}
-	return "", "", false
+	return "", "", "", false
 }
 
 // ProxyTemplate describes a single rule for parsing out proxy address from
@@ -119,18 +119,20 @@ type ProxyTemplate struct {
 	Proxy string `yaml:"proxy"`
 	// Host is optional hostname. Can refer to regex groups from the template.
 	Host string `yaml:"host"`
+	// Cluster is optional cluster name. Can refer to regex groups from the template.
+	Cluster string `yaml:"cluster"`
 	// re is the compiled template regexp.
 	re *regexp.Regexp
 }
 
 // Check validates the proxy template.
 func (t *ProxyTemplate) Check() (err error) {
-	if strings.TrimSpace(t.Proxy) == "" {
-		return trace.BadParameter("empty proxy expression")
-	}
 	if strings.TrimSpace(t.Template) == "" {
 		return trace.BadParameter("empty proxy template")
 	}
+	if strings.TrimSpace(t.Proxy) == "" && strings.TrimSpace(t.Cluster) == "" && strings.TrimSpace(t.Host) == "" {
+		return trace.BadParameter("empty proxy, cluster, and host fields in proxy template, but at least one is required")
+	}
 	t.re, err = regexp.Compile(t.Template)
 	if err != nil {
 		return trace.Wrap(err)
@@ -140,19 +142,20 @@ func (t *ProxyTemplate) Check() (err error) {
 
 // Apply applies the proxy template to the provided hostname and returns
 // expanded proxy address and hostname.
-func (t ProxyTemplate) Apply(fullHostname string) (proxy, host string, matched bool) {
+func (t ProxyTemplate) Apply(fullHostname string) (proxy, host, cluster string, matched bool) {
 	match := t.re.FindAllStringSubmatchIndex(fullHostname, -1)
 	if match == nil {
-		return "", "", false
+		return "", "", "", false
 	}
 
-	expandedProxy := []byte{}
-	for _, m := range match {
-		expandedProxy = t.re.ExpandString(expandedProxy, t.Proxy, fullHostname, m)
+	if t.Proxy != "" {
+		expandedProxy := []byte{}
+		for _, m := range match {
+			expandedProxy = t.re.ExpandString(expandedProxy, t.Proxy, fullHostname, m)
+		}
+		proxy = string(expandedProxy)
 	}
-	proxy = string(expandedProxy)
 
-	host = fullHostname
 	if t.Host != "" {
 		expandedHost := []byte{}
 		for _, m := range match {
@@ -161,7 +164,15 @@ func (t ProxyTemplate) Apply(fullHostname string) (proxy, host string, matched b
 		host = string(expandedHost)
 	}
 
-	return proxy, host, true
+	if t.Cluster != "" {
+		expandedCluster := []byte{}
+		for _, m := range match {
+			expandedCluster = t.re.ExpandString(expandedCluster, t.Cluster, fullHostname, m)
+		}
+		cluster = string(expandedCluster)
+	}
+
+	return proxy, host, cluster, true
 }
 
 // loadConfig load a single config file from given path. If the path does not exist, an empty config is returned instead.
@@ -193,7 +204,7 @@ func loadAllConfigs(cf CLIConf) (*TshConfig, error) {
 
 	globalConf, err := loadConfig(globalConfigPath)
 	if err != nil {
-		return nil, trace.Wrap(err, "failed to load global tsh config from %q", cf.GlobalTshConfigPath)
+		return nil, trace.Wrap(err, "failed to load global tsh config from %q", globalConfigPath)
 	}
 
 	fullConfigPath := filepath.Join(profile.FullProfilePath(cf.HomePath), tshConfigPath)