New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rate limiting to stream requests #726

Merged
merged 8 commits into from Oct 25, 2018

Conversation

Projects
None yet
2 participants
@lechengfan
Contributor

lechengfan commented Oct 24, 2018

  • fixes #715 @bartekn
  • Add an extra rate limit call for every call to stream.SSE in base.go so that clients are rate limited based on their DB queries
  • Upgrade throttled package and refactor all the deprecated methods

lechengfan added some commits Oct 24, 2018

@bartekn bartekn added this to the Horizon v0.15.0 milestone Oct 25, 2018

@bartekn bartekn added the horizon label Oct 25, 2018

@bartekn

This comment has been minimized.

Member

bartekn commented Oct 25, 2018

Regarding our chat yesterday and import cycles. I think we shouldn't merge actions package into internal. I think the idea behind actions package is to give you a nice, isolated implementation of actions, to serve both JSON and SSE responses. so I think it should stay in a separate package. I think we should do something like this:

  1. Create a new package context that will contain context keys used in Horizon, like:
type ContextKey string

var AppContextKey = ContextKey("app")
// httpx package
var RequestContextKey = ContextKey("request")
var ClientContextKey = ContextKey("client")
  1. Add a new interface in actions package:
type RateLimiterProvider interface {
	GetRateLimiter() *throttled.HTTPRateLimiter
}

This interface should be implemented by App (and it is in your PR).

  1. Finally you can get the object that implements RateLimiterProvider (in base.go) like:
// horizonContext is the new package from point 1.
app := base.R.Context().Value(&horizonContext.AppContextKey)
app.(RateLimiterProvider).RateLimiter.RateLimit(horizon.RemoteAddrIP(base.R), 1)
  1. Replace all the context keys in the app, moved to the new context package with the new variables (search for ctx.Value occurrences).
// https://github.com/stellar/go/issues/715 for more details.
app := horizon.AppFromContext(base.R.Context())
r := app.GetRateLimiter()
r.RateLimiter.RateLimit(horizon.RemoteAddrIP(base.R), 1)

This comment has been minimized.

@bartekn

bartekn Oct 25, 2018

Member

You should also send:

event: err
data: rate limited

and disconnect client if the limit has been reached.

@lechengfan

This comment has been minimized.

Contributor

lechengfan commented Oct 25, 2018

Thanks for the suggestion. Please take a look at the updated PR.

lechengfan added some commits Oct 25, 2018

@bartekn

Other than comments 👍

// https://github.com/stellar/go/issues/715 for more details.
app := base.R.Context().Value(&horizonContext.AppContextKey)
limited, _, err := app.(RateLimiterProvider).GetRateLimiter().RateLimiter.RateLimit(horizon.RemoteAddrIP(base.R), 1)
if limited || err != nil {

This comment has been minimized.

@bartekn

bartekn Oct 25, 2018

Member

err may also indicate storage issues, etc. In such case we should log the error and return error like below:

log.Ctx(ctx).Error(errors.Wrap(err, "RateLimiter error")
stream.Err(errors.New("Unexpected stream error"))
@@ -1,11 +1,11 @@
package main
import (
"github.com/throttled/throttled"

This comment has been minimized.

@bartekn

bartekn Oct 25, 2018

Member

Wrong import section should be below (https://github.com/golang/go/wiki/CodeReviewComments#imports).

if app.redis != nil {
rateLimitStore = store.NewRedisStore(app.redis, "throttle:", 0)
rateLimitStore, _ = redigostore.New(app.redis, "throttle:", 0)

This comment has been minimized.

@bartekn

bartekn Oct 25, 2018

Member

Please do not ignore errors (here and above in memstore). It can simply panic here.

// Rate limit the request if it's a call to stream since it queries the DB every second. See
// https://github.com/stellar/go/issues/715 for more details.
app := base.R.Context().Value(&horizonContext.AppContextKey)
limited, _, err := app.(RateLimiterProvider).GetRateLimiter().RateLimiter.RateLimit(horizon.RemoteAddrIP(base.R), 1)

This comment has been minimized.

@bartekn

bartekn Oct 25, 2018

Member

This line is too long. Also, instead of horizon.RemoteAddrIP let's use RateLimiter.VaryBy.Key. If RateLimiter.VaryBy.Key ever changes the change here won't be required. It will also remove the import cycle.

lechengfan added some commits Oct 25, 2018

@bartekn bartekn merged commit bfc6657 into stellar:horizon-0.15.0 Oct 25, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

bartekn added a commit that referenced this pull request Oct 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment