Skip to content
This repository has been archived by the owner on Apr 19, 2024. It is now read-only.

Fix goroutine leak in Instance.SetPeers #28

Merged
merged 2 commits into from
Dec 3, 2019

Conversation

bohde
Copy link
Contributor

@bohde bohde commented Nov 20, 2019

This fixes #26 through the following changes:

  1. Add a Shutdown method to PeerClient that will close the request queue, and drain in-flight requests with a timeout.
  2. Change PeerClient.run to send any enqueued requests when the request queue is closed
  3. Change PeerClient.run to call interval.Stop on return
  4. Reuse existing PeerClients inside Instance.SetPeers
  5. Shutdown any removed PeerClients inside Instance.SetPeers

I've been running it in a test environment with both traffic and a volatile peer set, and can confirm that this fixes leaking goroutines issue we were seeing earlier.

1. Reuse any existing connections to a peer before opening a new one
2. Add ability to gracefully shutdown any PeerClients
3. Shutdown PeerClients that are no longer needed
Use a `sync.WaitGroup` to keep track of in-flight requests, and wait
until either the `WaitGroup` has finished or the `Context` has been
cancelled before returning.
@bohde bohde requested a review from thrawn01 as a code owner November 20, 2019 19:38
@mailgun-ci
Copy link

Can one of the admins verify this patch?

Copy link
Contributor

@thrawn01 thrawn01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic code and a much needed improvement. Thanks for working on this!

The only thing missing is gubernator.go calls to GetPerrRateLimits() and UpdatePeerGlobals() should look for ErrClosing and attempt to get a new peer. (with retry limits). I don't wanna hold up this PR so I'm gonna merge and make those changes in a separate PR I'll @ mention you on.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Goroutine leak in Instance.SetPeers
3 participants