Trusted Cluster goroutine leak

[rosstimothy]

Description

It looks like watchCertAuthorities is not returning when trusted clusters are removed. Which subsequently prevents the underlying CertAuthorityWatcher to never be closed and causes a memory leak.

Both leaks can be seen in the screenshot below:

The goroutine profile here was taken after deleting the clusters:

What happened:

While running the 500 trusted clusters tests I noticed that after the trusted clusters were deleted there was a leak of both memory and goroutines.

What you expected to happen:

When the trusted clusters are removed, all resources should return to levels they were at prior to adding the clusters.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

1. Add a bunch of trusted clusters
2. Delete them all
3. Monitor goroutine and heap metrics

Server Details

• Teleport version (run teleport version): 9.0.0-beta.1

Client Details

• Tsh version (tsh version): 9.0.0-beta.1

[rosstimothy]

Did a bit more digging on this and it ended up being caused by two things

Blocking indefinitely writing to c.CertAuthorityC

CertAuthorityWatcher tries to write to a channel that is no longer being read from when the remoteSite is closed.

teleport/lib/services/watcher.go

Line 1026 in 5023235

c.CertAuthorityC <- casToSlice(c.host, c.user)

teleport/lib/services/watcher.go

Line 1011 in 5023235

c.CertAuthorityC <- casToSlice(c.host, c.user)

The correct behavior would be to select on writing to c.CertAuthorityC and reading from ctx.Done. Note the only other watchers that currently due this are the ProxyWatcher and LockWatcher. Both the DatabaseWatcher and AppWatcher also have the same blocking bug.

select {
case c.CertAuthorityC <- casToSlice(c.host, c.user):
case <- ctx.Done():

HTTPClient hanging indefinitely waiting for a response

CertAuthorityWatcher hangs waiting for a response from GetCertAuthorities and RotateExternalCertAuthority. This is caused by two things. First we don't set a Timeout on the http.Client used by HTTPClient.

teleport/lib/auth/clt.go

Line 187 in 5023235

roundtrip.HTTPClient(&http.Client{Transport: transport}),

While we do set some timeouts on the http.Transport used by said http.Client, that doesn't appear to be enough to prevent hanging forever in some situations.

Second, we don't properly propagate the callers context all the way down the the outgoing requests. So when the caller's context is cancelled there is nothing to stop the request.

teleport/lib/auth/clt.go

Line 273 in bac0ccd

return httplib.ConvertResponse(c.Client.PostJSON(context.TODO(), endpoint, val))

teleport/lib/auth/clt.go

Line 288 in bac0ccd

return httplib.ConvertResponse(c.Client.Get(context.TODO(), u, params))

[rosstimothy]

Here is the goroutine dump difference from prior to running the 500 Trusted Cluster scaling test and after. It captures bot the waiting on the channel and being stuck waiting for the HTTP response