Skip to content

DDoS mitigation

Jump to bottom Edit New page
EvgeniiMekhanik edited this page May 6, 2024 · 33 revisions

Tempesta FW is a hybrid of web accelerator and firewall specially designed to be resistant to application layer DDoS attacks. High performance architecture allows to process usual HTTP floods without any special measures. Meantime, we constantly develop new filtering mechanisms helping to filter our complex DDoS attacks.

Tempesta FW automatically generates filtering rules on IP layer, which drop malicious traffic early:

  1. The IP packet is received by the network adapter and quickly verified against filtering tables.
  2. The HTTP request is immediately parsed in OS's deferred interrupt while the data is still hot in CPU caches.
  3. The request is analyzed by Frang module responsible for detection of HTTP DDoS and web attacks. If the request is classified as malicious, then the attacker is blocked at IP layer and all subsequent requests from them are blocked at step 1.
  4. Otherwise the request is serviced from the cache or forwarded to an upstream server according to the established load balancing policy. The web cache and filtering database are built on top of TempestaDB.

TCP/IP layer DDoS attacks

Tempesta FW extends standard Linux TCP/IP stack, so all well known Linux solutions to mitigate DDoS attacks such as Netfilter, Synproxy and eBPF can be used with Tempesta FW.

Sockstress

This type of attack uses vulnerabilities in TCP protocol implementation to utilize network resources of targeted server in attempt to prevent some or all legitimate clients from being serviced.

There are several methods of sockstress attack. One of the most well-known method is exploiting of TCP persist timer: in the last packet of the three-way handshake - attacker sets 0 size of receiver window which means that the client is unable to receive data, the persist timer on targeted server is triggered, forcing it to keep the connection alive and periodically probe the client to check if the window size updated and client can receive data. In such a way the server will try to send the remaining data, taking up a limited queue of connections. If the entire queue is occupied by such malicious connections, the server will start denying service to legitimated clients.

This kind of attacks can be mitigated using concurrent_tcp_connections directive in Tempesta FW (see Frang limit), which limit maximum number of concurrent connections per one client, e.g.

concurrent_tcp_connections 10;

HTTP DDoS attacks

Tempesta FW introduces application layer (HTTP) DDoS protection to web services. Following sections describe Tempesta FW use cases for several types for DDoS attacks.

HTTP floods

First of all Tempesta FW servers from the cache up to 1.8M HTTP requests per second on cheapest Xeon CPU, so in most cases you don't need to do any effort to protect your backend servers against HTTP floods.

HTTP Cookie challenge

However, you can use Sticky session cookies to quickly block DDoS bots at IP layer. To do so, just use following configuration line to vhost definition:

vhost protected {
  sticky {
    cookie name=__tfw_user_id__ enforce max_misses=10;
  }
}

With the configuration Tempesta FW generated cryptographically strong Cookie value for each client and sends HTTP redirect to the client. Tempesta FW doesn't forward a HTTP request to a backend without proper Cookie value (or without Cookie value at all) if enforce option is specified. If the amount of HTTP requests without Cookie value from separate client will exceed specified number for max_misses parameter (10 in our example), such client will be blocked in order not to spend system resources on malicious traffic. Note that several bot nets, e.g. Mirai, are able to process the Cookie challenge.

JavaScript challenge

With JavaScript challenge Tempesta FW sends a JavaScript code to a client. The JavaScript script sets cryptographic Cookie, just like the Cookie challenge, and pauses a client for a timeout encoded in the Cookie. If a client sends next HTTP request before expiration of the timeout, max_misses counter will be increased, if this counter exceeded the limit, then the client will be blocked, otherwise JS challenge will be restarted.

In comparison with traditional JS Challenge the Tempesta FW module have significant improvements:

  • If bot violates timeouts defined in JS Challenge, it's blocked immediately (max_misses counter is equal to one by default, so any violation leads to block). Following timeouts enforce bot to make significant delays between requests. Thus, Tempesta FW enforces a bot to significantly decrease requests rate lowering the attack efficiency.
  • Sticky Cookie includes timestamp in open and encrypted part of Cookie value, so it's not possible either to predict and hardcode Sticky Cookie value in bot code nor reuse the same cookie in long term attacks.
  • The module blocks bots, not just give decisions, no additional utilities like fail2ban are required. Upcoming integration with Frang module (planned for 0.6 release) will extend blocking feature by adding rate limits for JS challenge failures.

The module with the Sticky Cookie rate limiting (under development and planned for upcoming 0.6 release) can block requests from full web stack botnets.

iFrames on a busy site

In this type of DDoS attacks an attacker places an iFrame referring to a victim site on some quite busy resource (the resource can be even from the Top 100). As the result the victim receives huge traffic from the iFrames. However, an HTTP request for iFrame always has Referer HTTP header referring the malicious web site.

You can filter HTTP requests with particular value of Referer header using following rule for HTTP Tables:

srv_group backend {
    # put all your servers here
}

http_chain {
    hdr "Referer" == "http://badhost.com*" -> block;
    -> backend;
}

Such configuration drops all malicious requests in black_hole while all other requests are processed by servers from backend group.

XSS'ed 3rd-party site

An attacker can use an XSS injection against some 3rd-party web site (or many sites) like

<img src="http://victim.com/1.jpg">
<img src="http://victim.com/2.jpg">
<img src="http://victim.com/3.jpg">
    ....

Single injection can contain many resources requested from the victim web site. Thus, every visitor of an XSS'ed web site generates many requests to the victim. This attack is similar to iFrame attack and can be also mitigated by blocking particular Referer HTTP headers.

Such requests can generate many 404 error responses from backend server. Tempesta FW caches such error responses as negative cache entries, so only first client access to the XSS'ed page reaches backend server and all next requests are processed on Tempesta FW side, without access to backend server.

The attack can be efficiently mitigated using http_resp_code_block as described in the next section.

Web cache bypass

Random URLs is a popular method to exhaust Web cache by negative cache entries, so valid client requests can not be serviced from the cache and cause load onto backed servers. The attack is very similar to password crackers in producing tons of negative responses (however, the responses in this case are usually for the same URI and do not waste Web cache). You can emulate the attack with a simple script:

$ while :; do wget https://tempesta-tech.com/`echo $RANDOM|md5sum|cut -d' ' -f1` ; done
--2022-01-19 19:10:40--  https://tempesta-tech.com/539e03f8562fd7a8caadd279773d98d1
Resolving tempesta-tech.com (tempesta-tech.com)... 93.115.28.125
Connecting to tempesta-tech.com (tempesta-tech.com)|93.115.28.125|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-01-19 19:10:41 ERROR 404: Not Found.

--2022-01-19 19:10:41--  https://tempesta-tech.com/53f0b30938a84afa5da4f6022ec8dc20
Resolving tempesta-tech.com (tempesta-tech.com)... 93.115.28.125
Connecting to tempesta-tech.com (tempesta-tech.com)|93.115.28.125|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2022-01-19 19:10:41 ERROR 404: Not Found.

....

It's recommended to use the same Frang limit to mitigate such kind of attacks, e.g.

frang_limits {
    ip_block on;

    http_resp_code_block 401 403 100 10;
}

block_action attack reply;
block_action error reply;

Note that we use ip_block to block the attacker IP address and not to allow their requests to reach the upstream server. Also we used relatively high limit of 100 bad requests per 10 seconds to not to false positive on a valid users.

With this configuration the script above will firstly show 403 generated by Tempesta FW and next the script hangs on TCP connection, when the IP address is blocked by Tempesta FW:

--2022-01-19 19:10:43--  https://tempesta-tech.com/91ae5ed6072b4bdaaeb5160534730cb3
Resolving tempesta-tech.com (tempesta-tech.com)... 93.115.28.125
Connecting to tempesta-tech.com (tempesta-tech.com)|93.115.28.125|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2022-01-19 19:10:43 ERROR 403: Forbidden.

--2022-01-19 19:10:43--  https://tempesta-tech.com/4bc56b881af7d4d0cbdc367e16daac8d
Resolving tempesta-tech.com (tempesta-tech.com)... 93.115.28.125
Connecting to tempesta-tech.com (tempesta-tech.com)|93.115.28.125|:443...

^C

Slow HTTP

In this kind of attack the attacker tries to keep many connections to the target web server, sending through these connections partial HTTP requests - requests in small pieces with large timeouts. In this case server is forced to keep such connections open, spending its resources and waiting for the rest of the data. If the transfer rate is very low, affected server will fill up its concurrent connections pool and begin to deny the legitimate clients.

To mitigate slow HTTP attacks Tempesta FW provides several possibilities via special configuration directives (see Frang limit and Keep-alive timeout):

  • Limit the chunks count of header and/or body of HTTP message to minimal reasonable value: http_header_chunk_cnt and http_body_chunk_cnt directives respectively are intended for this purpose.
  • Limit maximum time for receiving header and/or body of HTTP message: client_header_timeout and client_body_timeout directives respectively. When we receive the first block of the request header or body, we remember the time when it was received. When the next block arrives, we check the time of its arrival and if the waiting timeout exceeds the specified one, we drop the request. We have no special timers to drop the request when timeout is exceeded because it is very expensive to have timer for each request especially in kernel space.
  • Limit maximum time between sending of server's HTTP responses: keepalive_timeout directive; also this limit will effectively drop idle TCP connections, which is not used for sending/receiving HTTP traffic at all.

Example might look as follows:

frang_limits {
    http_header_chunk_cnt 10;
    http_body_chunk_cnt 30;
    client_header_timeout 10;
    client_body_timeout 25;
}
keepalive_timeout 50;

HTTP/2 Rapid Reset

The HTTP/2 protocol allows clients to close streams unilaterally by sending a RST_STREAM frame. In the HTTP/2 Rapid Reset attack (see also CVE-2023-44487), an attacker opens a large number of streams at once as in the standard HTTP/2 flood attack, but instead of waiting for a response to each request stream from a server or proxy, an attacking client cancels each request immediately.

Tempesta FW provides options to protect against such an attack via special configuration directives: request_rate and request_burst (see a corresponding Frang limit) - the rate limit accounts even reset streams (requests), so it's efficient against the attack (see an appropriate functional test).

Configuration example:

frang_limits {
    ip_block on;
    request_rate 100;
    request_burst 20;
}
block_action attack drop;

in this case, all the attacker's client connections will be blocked via RST TCP (see block_action) and new connections cannot be opened for this IP (see ip_block)

Targeted DDoS attacks

A targeted DDoS attack is designed specially for a particular victim, so some protection techniques such as Cookie or JavaScript challenges can be efficiently evaded by DDoS bots. DDoS agents can implement ad-hoc solution to bypass the protection, e.g. if an agent doesn't have a JavaScript engine it still can parse Cookies and/or JavaScript/HTML code in some very simple way, just to find expected tokens, and generate required output.

To mitigate such kind of DDoS attacks we use application performance monitoring (APM) accounting ratio of HTTP requests to responses from backend servers. The classification logic is planned for next 1.0 release.

Whitelist traffic from Web Search Engines

Frang and especially Sticky Cookie modules may significantly impact user experience and the site availability for search engines, so IP addresses which should not be impacted by DDoS protection modules can be whitelisted.

Add a custom footer

Contents

Clone this wiki locally