Recording Proxy Proposal

[kontsevoy]

Introduction

Teleport clusters implement session recording by having the end nodes submit session data to the audit/auth server. Some Teleport users have expressed interest in having the proxy act as a session recorder. There are pros and cons for both designs.

Node-based recording:

• Offers end to end encryption between the client and the server, with proxy acting as a dumb switch. Even if an attacker has physical access to the proxy, the session data and secrets will remain secure. This is why Teleport team had chosen this design to begin with.
• SSH sessions to sshd (OpenSSH servers) on legacy fleet behind a Teleport proxy will not be recorded.

Proposal

This proposal introduces a new mode of operation, AKA +"Recording Proxy Mode"_ in which:

• The nodes can run OpenSSH and other SSH-2.0 compatible servers.
• The SSH proxy terminates every connection, sends session data to an audit server and then proxies it using another SSH connection to the end node.

The original node-based recording is called "Pass-Through Proxy Mode". Lets review the difference in more detail:

Pass-Through Proxy Mode

To better illustrate the current default cluster mode, take a look at this diagram:

In the current design, the SSH proxy only performs authentication on the initial connection, after that, it simply acts as a bridge for SSH connections without ability to decrypt the traffic. The advantage of this design are:

1. A proxy is not a privileged member of the cluster. Highjacking a proxy does not give attacker any visibility into SSH sessions because the encryption is end-to-end.
2. System requirements for proxy servers are extremely lightweight.
3. All cluster secrets are only stored in the auth server, which also serves as an audit server.

Disadvantages of this “Pass-Through mode” include:

1. Session recording does not work with sshd, because sshd does not ship session data to Teleport auth servers.
2. Users with root privileges on a host can attempt to disrupt the session audit connection (known attack surface area).

Recording Proxy Mode

This document proposes a new method for proxying SSH connections, it’s called “terminating proxy” mode. In this mode the Teleport proxy performs a much bigger role. Consider the following diagram:

In the “terminating proxy mode” the connection between Alice and the sshd server is not simply proxied, but it gets terminated on the proxy level.

These are the steps of establishing an SSH session between the client (Alice) and the SSH server running sshd:

1. Alice issues ssh host command. Her SSH client is configured to use [work.example.com] as a proxy server for such connection. The configuration of an SSH client will be covered below.
2. The proxy server accepts her connection and validates her SSH credentials (certificate) against the cluster CA (auth server). This step is similar to the traditional “pass-through” mode.
3. Alice’s SSH client uses the opened SSH data channel to initiate the connection to the target node. Proxy hijacks the connection and presents itself as the target node. Proxy presents host certificate with the right principal to properly pass the authentication for the second connection, de-facto impersonating the target SSH node.
4. Proxy checks users certificate for the second time, and If the user certificate is valid, the proxy requests the auth server to generate a new ephemeral session credentials for Alice and serve them via an SSH agent mechanism. The SSH agent connection can be optionally forwarded to the SSH node running OpenSSH. Teleport proxies already use the SSH agent forwarding mechanism to implement web-based SSH terminals.
5. The SSH node receives the connection request. It also validates the presented SSH certificate against the cluster key it’s configured to trust.
6. Teleport supports SSH agent forwarding between a node and the proxy. This can be enabled for any user role. When turned on, it allows Alice to ssh from host A to host B, also located behind a proxy/proxy.

The primary advantage of the “terminating proxy” mode is that the OpenSSH sshd daemon can be used without the loss of the session recording. The primary disadvantage is that a proxy becomes a privileged node within the cluster, increasing the potential attack surface.

Client Configuration

A user has a choice of two SSH clients to use: the Teleport client tsh and OpenSSH client ssh. Regardless of the chosen method, a user must request the session credentials from the corporate proxy [work.example.com]. Typically this happens once a day, to receive SSH certificate valid for 8-12 hours by executing:

$ tsh login --proxy=work.example.com

This command will print a login URL which leads to a SAML-powered single sign-on (SSO) page where a user will have to enter their corporate identity credentials.

When successful, tsh login will place the ephemeral SSH certificate (again, the TTL can be configured by a user role) into ~/.tsh/work.example.com/keys and also the certificate will be loaded into the active SSH agent, if present, which can be verified by executing:

$ ssh-add -L

If storing SSH certificates in ~/.tsh directory or in the ssh agent is not desired, a user can tell tsh login to save the certificate into an identity file by adding --out=alice.pem CLI flag. The identity file alice.pem can be used later via -i flag of ssh.
Using Teleport SSH client

After tsh login a user may connect to any SSH server behind [work.example.com] SSH proxy simply typing:

$ tsh ssh addr

In this case the name resolution of ‘addr’ will be happening on the proxy. This means that internal, behind-firewall IP addresses may be used.

If typing tsh ssh becomes tiresome, or if easy compatibility with existing scripts is required, tsh binary can be simply renamed to ssh or a symlink can be created tsh -> ssh which will make all standard ssh commands work, like:

$ ssh addr

Using OpenSSH Client

To start using OpenSSH client with a Teleport proxy [work.example.com], Alice needs to configure ssh to connect via work.example.com for certain hostnames. This can be accomplished by modifying her ~/.ssh/config:

Host *.work.example.com
    HostName %h
    Port 3022
    ProxyCommand ssh -p 3023 %r@work.example.com -s proxy:%h:%p

Now she can connect to any server behind work.example.com by typing something like this:

$ ssh -i alice.pem jenkins.work.example.com

Note: the identity flag -i is not necessary if tsh login was allowed to store Alice’s cluster certificate in the ssh-agent.

Server Configuration

Every OpenSSH node running sshd would need to be configured to accept SSH certificates issued by the Teleport CA. This can be done by exporting a cluster cert into a file and copying it to every SSH node as shown in the Teleport documentation.

On Teleport auth node:

$ tctl auth --type=user export > ca.pub

On every sshd node you can do one of the following:

Append the .pub file to ~/.ssh/authorized_keys
Remove “cert-authority” substring from ca.pub and update /etc/ssh/sshd_config with the following line: TrustedUserCAKeys /path/to/ca.pub

[kontsevoy]

This mode is now available for early access users in Teleport 2.4.1 RC1