Skip to content

Commit

Permalink
Add Global Rate Limiting support
Browse files Browse the repository at this point in the history
  • Loading branch information
ElvinEfendi committed Jan 4, 2021
1 parent 14345eb commit e0dece4
Show file tree
Hide file tree
Showing 21 changed files with 1,179 additions and 38 deletions.
1 change: 1 addition & 0 deletions build/test-lua.sh
Expand Up @@ -34,4 +34,5 @@ resty \
--shdict "balancer_ewma 1M" \
--shdict "balancer_ewma_last_touched_at 1M" \
--shdict "balancer_ewma_locks 512k" \
--shdict "global_throttle_cache 5M" \
./rootfs/etc/nginx/lua/test/run.lua ${BUSTED_ARGS} ./rootfs/etc/nginx/lua/test/ ./rootfs/etc/nginx/lua/plugins/**/test
46 changes: 45 additions & 1 deletion docs/user-guide/nginx-configuration/annotations.md
Expand Up @@ -56,6 +56,10 @@ You can add these Kubernetes annotations to specific Ingress objects to customiz
|[nginx.ingress.kubernetes.io/http2-push-preload](#http2-push-preload)|"true" or "false"|
|[nginx.ingress.kubernetes.io/limit-connections](#rate-limiting)|number|
|[nginx.ingress.kubernetes.io/limit-rps](#rate-limiting)|number|
|[nginx.ingress.kubernetes.io/global-rate-limit](#global-rate-limiting)|number|
|[nginx.ingress.kubernetes.io/global-rate-limit-window](#global-rate-limiting)|duration|
|[nginx.ingress.kubernetes.io/global-rate-limit-key](#global-rate-limiting)|string|
|[nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs](#global-rate-limiting)|string|
|[nginx.ingress.kubernetes.io/permanent-redirect](#permanent-redirect)|string|
|[nginx.ingress.kubernetes.io/permanent-redirect-code](#permanent-redirect-code)|number|
|[nginx.ingress.kubernetes.io/temporal-redirect](#temporal-redirect)|string|
Expand Down Expand Up @@ -474,7 +478,7 @@ By default the controller redirects all requests to an existing service that pro
!!! note
For more information please see [global-auth-url](./configmap.md#global-auth-url).

### Rate limiting
### Rate Limiting

These annotations define limits on connections and transmission rates. These can be used to mitigate [DDoS Attacks](https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus).

Expand All @@ -492,6 +496,46 @@ To configure settings globally for all Ingress rules, the `limit-rate-after` and

The client IP address will be set based on the use of [PROXY protocol](./configmap.md#use-proxy-protocol) or from the `X-Forwarded-For` header value when [use-forwarded-headers](./configmap.md#use-forwarded-headers) is enabled.

### Global Rate Limiting

**Note:** Be careful when configuring both (Local) Rate Limiting and Global Rate Limiting at the same time.
They are two completely different rate limiting implementations. Whichever limit exceeds first will reject the
requests. It might be a good idea to configure both of them to ease load on Global Rate Limiting backend
in cases of spike in traffic.

The stock NGINX rate limiting does not share its counters among different NGINX instances.
Given that most ingress-nginx deployments are elastic and number of replicas can change any day
it is impossible to configure a proper rate limit using stock NGINX functionalities.
Global Rate Limiting overcome this by using [lua-resty-global-throttle](https://github.com/ElvinEfendi/lua-resty-global-throttle). `lua-resty-global-throttle` shares its counters via a central store such as `memcached`.
The obvious shortcoming of this is users have to deploy and operate a `memcached` instance
in order to benefit from this functionality. Configure the `memcached`
using [these configmap settings](./configmap.md#memcached).

**Here are a few remarks for ingress-nginx integration of `lua-resty-global-throttle`:**

1. We minimize `memcached` access by caching exceeding limit decisions. The expiry of
cache entry is the desired delay `lua-resty-global-throttle` calculates for us.
The Lua Shared Dictionary used for that is `global_throttle_cache`. Currently its size defaults to 10M.
Customize it as per your needs using [lua-shared-dicts](./configmap.md#lua-shared-dicts).
When we fail to cache the exceeding limit decision then we log an NGINX error. You can monitor
for that error to decide if you need to bump the cache size. Without cache the cost of processing a
request is two memcached commands: `GET`, and `INCR`. With the cache it is only `INCR`.
1. Log NGINX variable `$global_rate_limit_exceeding`'s value to have some visibility into
what portion of requests are rejected (value `y`), whether they are rejected using cached decision (value `c`),
or if they are not rejeced (default value `n`). You can use [log-format-upstream](./configmap.md#log-format-upstream)
to include that in access logs.
1. In case of an error it will log the error message and **fail open**.
1. The annotations below creates Global Rate Limiting instance per ingress.
That means if there are multuple paths configured under the same ingress,
the Global Rate Limiting will count requests to all the paths under the same counter.
Extract a path out into its own ingres if you need to isolate a certain path.


* `nginx.ingress.kubernetes.io/global-rate-limit`: Configures maximum allowed number of requests per window. Required.
* `nginx.ingress.kubernetes.io/global-rate-limit-window`: Configures a time window (i.e `1m`) that the limit is applied. Required.
* `nginx.ingress.kubernetes.io/global-rate-limit-key`: Configures a key for counting the samples. Defaults to `$remote_addr`. You can also combine multiple NGINX variables here, like `${remote_addr}-${http_x_api_client}` which would mean the limit will be applied to requests coming from the same API client (indicated by `X-API-Client` HTTP request header) with the same source IP address.
* `nginx.ingress.kubernetes.io/global-rate-limit-ignored-cidrs`: comma separated list of IPs and CIDRs to match client IP against. When there's a match request is not considered for rate limiting.

### Permanent Redirect

This annotation allows to return a permanent redirect (Return Code 301) instead of sending data to the upstream. For example `nginx.ingress.kubernetes.io/permanent-redirect: https://www.google.com` would redirect everything to Google.
Expand Down
22 changes: 22 additions & 0 deletions docs/user-guide/nginx-configuration/configmap.md
Expand Up @@ -192,6 +192,12 @@ The following table shows a configuration option's name, type, and the default v
|[block-referers](#block-referers)|[]string|""|
|[proxy-ssl-location-only](#proxy-ssl-location-only)|bool|"false"|
|[default-type](#default-type)|string|"text/html"|
|[global-rate-limit-memcached-host](#global-rate-limit)|string|""|
|[global-rate-limit-memcached-port](#global-rate-limit)|int|11211|
|[global-rate-limit-memcached-connect-timeout](#global-rate-limit)|int|50|
|[global-rate-limit-memcached-max-idle-timeout](#global-rate-limit)|int|10000|
|[global-rate-limit-memcached-pool-size](#global-rate-limit)|int|50|
|[global-rate-limit-status-code](#global-rate-limit)|int|429|

## add-headers

Expand Down Expand Up @@ -1152,3 +1158,19 @@ _**default:**_ text/html

_References:_
[http://nginx.org/en/docs/http/ngx_http_core_module.html#default_type](http://nginx.org/en/docs/http/ngx_http_core_module.html#default_type)

## global-rate-limit

* `global-rate-limit-status-code`: configure HTTP status code to return when rejecting requests. Defaults to 429.

Configure `memcached` client for [Global Rate Limiting](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#global-rate-limiting).

* `global-rate-limit-memcached-host`: IP/FQDN of memcached server to use. Required to enable Global Rate Limiting.
* `global-rate-limit-memcached-port`: port of memcached server to use. Defaults default memcached port of `11211`.
* `global-rate-limit-memcached-connect-timeout`: configure timeout for connect, send and receive operations. Unit is millisecond. Defaults to 50ms.
* `global-rate-limit-memcached-max-idle-timeout`: configure timeout for cleaning idle connections. Unit is millisecond. Defaults to 50ms.
* `global-rate-limit-memcached-pool-size`: configure number of max connections to keep alive. Make sure your `memcached` server can handle
`global-rate-limit-memcached-pool-size * worker-processes * <number of ingress-nginx replicas>` simultaneous connections.

These settings get used by [lua-resty-global-throttle](https://github.com/ElvinEfendi/lua-resty-global-throttle)
that ingress-nginx includes. Refer to the link to learn more about `lua-resty-global-throttle`.
3 changes: 3 additions & 0 deletions internal/ingress/annotations/annotations.go
Expand Up @@ -40,6 +40,7 @@ import (
"k8s.io/ingress-nginx/internal/ingress/annotations/customhttperrors"
"k8s.io/ingress-nginx/internal/ingress/annotations/defaultbackend"
"k8s.io/ingress-nginx/internal/ingress/annotations/fastcgi"
"k8s.io/ingress-nginx/internal/ingress/annotations/globalratelimit"
"k8s.io/ingress-nginx/internal/ingress/annotations/http2pushpreload"
"k8s.io/ingress-nginx/internal/ingress/annotations/influxdb"
"k8s.io/ingress-nginx/internal/ingress/annotations/ipwhitelist"
Expand Down Expand Up @@ -94,6 +95,7 @@ type Ingress struct {
Proxy proxy.Config
ProxySSL proxyssl.Config
RateLimit ratelimit.Config
GlobalRateLimit globalratelimit.Config
Redirect redirect.Config
Rewrite rewrite.Config
Satisfy string
Expand Down Expand Up @@ -142,6 +144,7 @@ func NewAnnotationExtractor(cfg resolver.Resolver) Extractor {
"Proxy": proxy.NewParser(cfg),
"ProxySSL": proxyssl.NewParser(cfg),
"RateLimit": ratelimit.NewParser(cfg),
"GlobalRateLimit": globalratelimit.NewParser(cfg),
"Redirect": redirect.NewParser(cfg),
"Rewrite": rewrite.NewParser(cfg),
"Satisfy": satisfy.NewParser(cfg),
Expand Down
111 changes: 111 additions & 0 deletions internal/ingress/annotations/globalratelimit/main.go
@@ -0,0 +1,111 @@
/*
Copyright 2020 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package globalratelimit

import (
"strings"
"time"

"github.com/pkg/errors"
networking "k8s.io/api/networking/v1beta1"

"k8s.io/ingress-nginx/internal/ingress/annotations/parser"
ing_errors "k8s.io/ingress-nginx/internal/ingress/errors"
"k8s.io/ingress-nginx/internal/ingress/resolver"
"k8s.io/ingress-nginx/internal/net"
"k8s.io/ingress-nginx/internal/sets"
)

const defaultKey = "$remote_addr"

// Config encapsulates all global rate limit attributes
type Config struct {
Namespace string `json:"namespace"`
Limit int `json:"limit"`
WindowSize int `json:"window-size"`
Key string `json:"key"`
IgnoredCIDRs []string `json:"ignored-cidrs"`
}

// Equal tests for equality between two Config types
func (l *Config) Equal(r *Config) bool {
if l.Namespace != r.Namespace {
return false
}
if l.Limit != r.Limit {
return false
}
if l.WindowSize != r.WindowSize {
return false
}
if l.Key != r.Key {
return false
}
if len(l.IgnoredCIDRs) != len(r.IgnoredCIDRs) || !sets.StringElementsMatch(l.IgnoredCIDRs, r.IgnoredCIDRs) {
return false
}

return true
}

type globalratelimit struct {
r resolver.Resolver
}

// NewParser creates a new globalratelimit annotation parser
func NewParser(r resolver.Resolver) parser.IngressAnnotation {
return globalratelimit{r}
}

// Parse extracts globalratelimit annotations from the given ingress
// and returns them structured as Config type
func (a globalratelimit) Parse(ing *networking.Ingress) (interface{}, error) {
config := &Config{}

limit, _ := parser.GetIntAnnotation("global-rate-limit", ing)
rawWindowSize, _ := parser.GetStringAnnotation("global-rate-limit-window", ing)

if limit == 0 || len(rawWindowSize) == 0 {
return config, nil
}

windowSize, err := time.ParseDuration(rawWindowSize)
if err != nil {
return config, ing_errors.LocationDenied{
Reason: errors.Wrap(err, "failed to parse 'global-rate-limit-window' value"),
}
}

key, _ := parser.GetStringAnnotation("global-rate-limit-key", ing)
if len(key) == 0 {
key = defaultKey
}

rawIgnoredCIDRs, _ := parser.GetStringAnnotation("global-rate-limit-ignored-cidrs", ing)
ignoredCIDRs, err := net.ParseCIDRs(rawIgnoredCIDRs)
if err != nil {
return nil, err
}

config.Namespace = strings.Replace(string(ing.UID), "-", "", -1)
config.Limit = limit
config.WindowSize = int(windowSize.Seconds())
config.Key = key
config.IgnoredCIDRs = ignoredCIDRs

return config, nil
}

0 comments on commit e0dece4

Please sign in to comment.