Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require SSH prefix in router.DialHost connections #33000

Merged
merged 12 commits into from Oct 18, 2023

Conversation

AntonAM
Copy link
Contributor

@AntonAM AntonAM commented Oct 4, 2023

This PR makes sure that first bytes we see on connections that were created in router.DialHost() is the SSH protocol prefix, so malicious users can't send their own PROXY headers.

Fixes https://github.com/gravitational/teleport-private/issues/737

Copy link
Contributor

@rosstimothy rosstimothy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that all connections established via tsh ssh no longer use the proxy subsystem to connect to a host. Do we need this for those as well?

@rosstimothy rosstimothy self-requested a review October 5, 2023 12:24
@@ -260,12 +263,46 @@ func (t *proxySubsys) proxyToHost(ctx context.Context, ch ssh.Channel, clientSrc
}

go func() {
t.close(utils.ProxyConn(ctx, ch, conn))
t.close(utils.ProxyConn(ctx, &checkedPrefixReader{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The correct place for this is (*Router).DialHost in lib/proxy/router.go, which ends up being used by proxy subsys and TransportService/ProxySSH (and even then, maybe only in places where we end up actually dialing a connection to the outside world?).

Copy link
Contributor

@jentfoo jentfoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a couple comments to consider, it would be good to add test cases for them as well.

return 0, trace.AccessDenied("required prefix %q was not found", string(c.requiredPrefix))
}

c.upstreamReader = reader
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By not setting this till the end, we are introducing a weird behavior pattern. If there is an error we end up discarding a chunk of the data, but we technically leave this function open to then verify later.

I would suggest setting upstreamReader to a reader which always returns an error around line 289, then let it be reset to a valid reader after verification

lib/srv/regular/proxy.go Outdated Show resolved Hide resolved
return c.Conn.Write(p)
}

// It is assumed that required prefix is small enough to always fit into data provided on the first write.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kind of a bad assumption and it will lead to really strange issues precisely in situations where connectivity is already bad enough to be annoying in other ways.

I recommend changing requiredPrefix and checked into requiredData, and then you can check if the beginning of p matches the beginning of requiredData (depending on which one is the shorter), then writing, then advancing requiredData by the returned amount by Write.

Right now you can bypass this (accidentally or not):

w := &checkedPrefixWriter{requiredPrefix: p}

w.SetWriteDeadline(time.UnixNano(1))
w.Write(p) // guaranteed to fail, but sets "checked" to true
w.SetWriteDeadline(time.Time{})
w.Write(notP)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, but it still will return an error. Our goal is to not let attacker write data starting with anything but required prefix without an error, and in our use case that will be enough and connection will be rejected, unless there's a bug somewhere else 😅
But I changed it to the scheme you suggested and it can now accommodate multiple writes to satisfy the prefix requirement.

lib/proxy/router.go Outdated Show resolved Hide resolved
@AntonAM
Copy link
Contributor Author

AntonAM commented Oct 6, 2023

@rosstimothy yes, it's not tsh ssh itself, but the fact that we have ProxySSH() on server, so we need to make sure malicious attacker can't use it to self-call Proxy and then send fake PROXY header. For some reason I thought new transport is only available starting from v14 🤦‍♂️ , and in v14 this attack already doesn't work, so we mainly need this fix for v12 and v13 (although it's safer to have it everywhere)

I've moved the check to the write side in the DialHost as @espadolini suggested.

@AntonAM
Copy link
Contributor Author

AntonAM commented Oct 6, 2023

@jentfoo changed the implementation.

@AntonAM AntonAM requested a review from jentfoo October 6, 2023 15:27
@AntonAM AntonAM changed the title Require SSH prefix in proxySubsys connections Require SSH prefix in router.DialHost connections Oct 6, 2023
Copy link
Contributor

@espadolini espadolini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels simpler to just advance the slice, but if you want to keep the entire original prefix (maybe to output as an error) feel free to ignore that.

The net.Conn contract prescribes that all functions must be callable concurrently, which this implementation fails to provide - that being said, neither does lib/multiplexer.Conn, so supporting concurrent calls to Write doesn't seem to be particularly important, anyway. It might be worth noting in a comment, tho.

lib/proxy/router.go Outdated Show resolved Hide resolved
lib/proxy/router.go Show resolved Hide resolved
lib/proxy/router.go Outdated Show resolved Hide resolved
@AntonAM
Copy link
Contributor Author

AntonAM commented Oct 6, 2023

@espadolini yeah, I started with advancing the slice, but then error message is kinda incorrect.

Yep, I know it's not concurrent-safe, but as you said, for our code in general that train has left the station (and I think that's fine, our networking is complicated as it is without concurrent read/writes to connections 😅 )
I'll add comment.

lib/proxy/router.go Outdated Show resolved Hide resolved
lib/proxy/router_test.go Outdated Show resolved Hide resolved
lib/proxy/router_test.go Outdated Show resolved Hide resolved
lib/proxy/router_test.go Outdated Show resolved Hide resolved
lib/proxy/router.go Outdated Show resolved Hide resolved
lib/proxy/router.go Outdated Show resolved Hide resolved
@AntonAM AntonAM added this pull request to the merge queue Oct 18, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 18, 2023
@AntonAM AntonAM added this pull request to the merge queue Oct 18, 2023
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 18, 2023
AntonAM and others added 12 commits October 18, 2023 16:24
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
@AntonAM AntonAM force-pushed the anton/require-ssh-prefix-on-proxy branch from 1e67950 to 32e20fc Compare October 18, 2023 20:24
@AntonAM AntonAM added this pull request to the merge queue Oct 18, 2023
Merged via the queue into master with commit fd73320 Oct 18, 2023
28 checks passed
@AntonAM AntonAM deleted the anton/require-ssh-prefix-on-proxy branch October 18, 2023 21:09
@public-teleport-github-review-bot

@AntonAM See the table below for backport results.

Branch Result
branch/v12 Failed
branch/v13 Failed
branch/v14 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants