Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Merge branch 'master' of github.com:baudehlo/Haraka

  • Loading branch information...
commit 508d6fe2ca1baa55205cb92bcc690818365d1038 2 parents 932b597 + c89f638
Matt Sergeant authored
47 config/rate_limit.ini
... ... @@ -0,0 +1,47 @@
  1 +; Example configuration file for the rate_limit plugin
  2 +
  3 +; redis_server = 1.2.3.4
  4 +; tarpit_delay = 30
  5 +
  6 +[concurrency]
  7 +; NOTE: this limit is per server child and does not use Redis
  8 +; Limit an IP or host to a maximum number of connections
  9 +
  10 +; Don't limit connections from localhost
  11 +127 = 0
  12 +
  13 +; Freemail
  14 +; hotmail.com = 20
  15 +; yahoo.com = 20
  16 +; google.com = 20
  17 +
  18 +; default = 5
  19 +
  20 +[rate_conn]
  21 +; Maximum number of connections from an IP or host over an interval
  22 +
  23 +127 = 0
  24 +; default = 5 ; no interval defaults to 60s
  25 +
  26 +[rate_rcpt_host]
  27 +; Maximum number of recipients from an IP or host over an interval
  28 +
  29 +127 = 0
  30 +; default = 50/5m ; 50 RCPT To: maximum in 5 minutes
  31 +
  32 +[rate_rcpt_sender]
  33 +; Maximum number of recipients from a sender over an interval
  34 +
  35 +127 = 0
  36 +; default = 50/5m
  37 +
  38 +[rate_rcpt]
  39 +; Limit the rate of message attempts over a interval to a recipient
  40 +
  41 +127 = 0
  42 +; default = 50/5m
  43 +
  44 +[rate_rcpt_null]
  45 +; Limit the number of DSN/MDN messages by recipient
  46 +
  47 +; default = 1
1  config/tarpit.timeout
... ... @@ -0,0 +1 @@
  1 +0
153 docs/plugins/rate_limit.md
Source Rendered
... ... @@ -0,0 +1,153 @@
  1 +rate_limit
  2 +==========
  3 +
  4 +This pluign enforces limits on connection concurrency, connection rate and
  5 +recipient rate.
  6 +
  7 +By default DENYSOFT will be returned when the limits are exceeded, but for
  8 +concurrency, connection rate and recipient rate by host you can optionally
  9 +tarpit the connection by adding a delay before every response sent back to the
  10 +client instead of sending a DENYSOFT. To do this requires the 'tarpit' plugin
  11 +to run immediately after this plugin.
  12 +
  13 +To use this plugin you will need a Redis server and will need the redis,
  14 +hiredis and ipaddr.js packages installed via:
  15 +
  16 + cd /path/to/haraka/home
  17 + npm install redis hiredis ipaddr.js
  18 +
  19 +Configuration
  20 +-------------
  21 +
  22 +This plugin uses the configuration file rate_limit.ini which is checked for
  23 +updates before each hook, so changes to this file will never require a restart
  24 +and will take effect immediately after the changes are saved.
  25 +
  26 +The configuration options for each heading are detailed below:
  27 +
  28 +### [main]
  29 +
  30 +- redis_server = \<ip | host\>[:port] *(optional)*
  31 +
  32 + If port is missing then it defaults to 6379.
  33 + If this setting is missing entirely then it defaults to 127.0.0.1:6379.
  34 +
  35 + Note that Redis does not currently support IPv6.
  36 +
  37 +- tarpit_delay = seconds *(optional)*
  38 +
  39 + Set this to the length in seconds that you want to delay every SMTP
  40 + response to a remote client that has exceeded the rate limits. For this
  41 + to work the 'tarpit' plugin must be loaded **after** this plugin in
  42 + config/plugins.
  43 +
  44 + If 'tarpit' is not loaded or is loaded before this plugin, then no
  45 + rate throttling will occur.
  46 +
  47 +* * *
  48 +
  49 +All of the following sections are optional. Any missing section disables
  50 +that particular test.
  51 +
  52 +They all use a common configuration format:
  53 +
  54 +- \<lookup\> = \<limit\>[/time[unit]] *(optional)*
  55 +
  56 + 'lookup' is based upon the limit being enforced and is either an IP
  57 + address, rDNS name, sender address or recipient address either in full
  58 + or part.
  59 + The lookup order is as follows and the first match in this order is
  60 + returned and is used as the record key in Redis (except for 'default'
  61 + which always uses the full lookup for that test as the record key):
  62 +
  63 + **IPv4/IPv6 address or rDNS hostname:**
  64 +
  65 + <pre>
  66 + fe80:0:0:0:202:b3ff:fe1e:8329
  67 + fe80:0:0:0:202:b3ff:fe1e
  68 + fe80:0:0:0:202:b3ff
  69 + fe80:0:0:0:202
  70 + fe80:0:0:0
  71 + fe80:0:0
  72 + fe80:0
  73 + fe80
  74 + 1.2.3.4
  75 + 1.2.3
  76 + 1.2
  77 + 1
  78 + host.part.domain.com
  79 + part.domain.com
  80 + domain.com
  81 + com
  82 + default
  83 + </pre>
  84 +
  85 + **Sender or Recipient address:**
  86 +
  87 + <pre>
  88 + user@host.sub.part.domain.com
  89 + host.sub.part.domain.com
  90 + sub.part.domain.com
  91 + part.domain.com
  92 + domain.com
  93 + com
  94 + default
  95 + </pre>
  96 +
  97 + In all tests 'default' is used to specify a default limit if nothing else has
  98 + matched.
  99 +
  100 + 'limit' specifies the limit for this lookup. Specify 0 (zero) to disable
  101 + limits on a matching lookup.
  102 +
  103 + 'time' is optional and if missing defaults to 60 seconds. You can optionally
  104 + specify the following time units (case-insensitive):
  105 +
  106 + - s (seconds)
  107 + - m (minutes)
  108 + - h (hours)
  109 + - d (days)
  110 +
  111 +### [concurrency]
  112 +
  113 +**IMPORTANT NOTE:** connection concurrency is recorded in-memory (in
  114 +connection.server.notes) and not in Redis, so the limits are per-server and
  115 +per-child if you use the cluster module.
  116 +
  117 +IP and rDNS names are looked up by this test. This section does *not* accept an
  118 +interval. It's a hard limit on the number of connections and not based on time.
  119 +
  120 +### [rate_conn]
  121 +
  122 +This section limits the number of connections per interval from a given host
  123 +or set of hosts.
  124 +
  125 +IP and rDNS names are looked up by this test.
  126 +
  127 +### [rate_rcpt_host]
  128 +
  129 +This section limits the number of recipients per interval from a given host or
  130 +set of hosts.
  131 +
  132 +IP and rDNS names are looked up by this test.
  133 +
  134 +### [rate_rcpt_sender]
  135 +
  136 +This section limits the number of recipients per interval from a sender or
  137 +sender domain.
  138 +
  139 +The sender is looked up by this test.
  140 +
  141 +### [rate_rcpt]
  142 +
  143 +This section limits the rate which a recipient or recipient domain can
  144 +receive messages over an interval.
  145 +
  146 +Each recipient is looked up by this test.
  147 +
  148 +### [rate_rcpt_null]
  149 +
  150 +This section limits the rate at which a recipient can receive messages from
  151 +a null sender (e.g. DSN, MDN etc.) over an interval.
  152 +
  153 +Each recipient is looked up by this test.
21 docs/plugins/tarpit.md
Source Rendered
... ... @@ -0,0 +1,21 @@
  1 +tarpit
  2 +======
  3 +
  4 +This plugin is designed to introduce deliberate delays on the response
  5 +of every hook in order to slow down a connection. It has no
  6 +configuration and is designed to be used only by other plugins.
  7 +
  8 +It must be loaded early in config/plugins (e.g. before any plugins
  9 +that accept recipients or that return OK) but must be loaded *after*
  10 +any plugins that wish to use it.
  11 +
  12 +To use this plugin in another plugin set:
  13 +
  14 + connection.notes.tarpit = <seconds to delay>;
  15 +
  16 +or
  17 +
  18 + connection.transaction.notes.tarpit = <seconds to delay>;
  19 +
  20 +When tarpitting a command it will log 'tarpitting response for Ns' to
  21 +the INFO facility where N is the number of seconds.
370 plugins/rate_limit.js
... ... @@ -0,0 +1,370 @@
  1 +// rate_limit
  2 +var ipaddr = require('ipaddr.js');
  3 +var redis = require('redis');
  4 +var client;
  5 +
  6 +exports.register = function () {
  7 + var config = this.config.get('rate_limit.ini');
  8 + if (config.main.redis_server) {
  9 + // No support for IPv6 in Redis yet...
  10 + // TODO: make this regex support IPv6 when it does.
  11 + var match = /^([^: ]+)(?::(\d+))?$/.exec(config.main.redis_server);
  12 + if (match) {
  13 + var host = match[1];
  14 + var port = match[2] || '6379';
  15 + this.logdebug('using redis on ' + host + ':' + port);
  16 + client = redis.createClient(port, host);
  17 + }
  18 + else {
  19 + // Syntax error
  20 + throw new Error('syntax error');
  21 + }
  22 + }
  23 + else {
  24 + // Client default is 127.0.0.1:6379
  25 + client = redis.createClient();
  26 + }
  27 + this.register_hook('connect', 'incr_concurrency');
  28 + this.register_hook('disconnect', 'decr_concurrency');
  29 +}
  30 +
  31 +exports.lookup_host_key = function (type, args, cb) {
  32 + var remote_ip = args[0];
  33 + var remote_host = args[1];
  34 + var config = this.config.get('rate_limit.ini');
  35 + if (!config[type]) {
  36 + return cb(new Error(type + ': not configured'));
  37 + }
  38 + var ip;
  39 + var ip_type;
  40 + try {
  41 + ip = ipaddr.parse(remote_ip);
  42 + ip_type = ip.kind();
  43 + if (ip_type === 'ipv6') {
  44 + ip = ipaddr.toNormalizedString();
  45 + }
  46 + else {
  47 + ip = ip.toString();
  48 + }
  49 + }
  50 + catch (err) {
  51 + return cb(err);
  52 + }
  53 +
  54 + var ip_array = ((ip_type === 'ipv6') ? ip.split(':') : ip.split('.'));
  55 + while (ip_array.length) {
  56 + var part = ((ip_type === 'ipv6') ? ip_array.join(':') : ip_array.join('.'));
  57 + if (config[type][part] || config[type][part] === 0) {
  58 + return cb(null, part, config[type][part]);
  59 + }
  60 + ip_array.pop();
  61 + }
  62 +
  63 + // rDNS
  64 + if (remote_host) {
  65 + var rdns_array = remote_host.toLowerCase().split('.');
  66 + while (rdns_array.length) {
  67 + var part = rdns_array.join('.');
  68 + if (config[type][part] || config[type][part] === 0) {
  69 + return cb(null, part, config[type][part]);
  70 + }
  71 + rdns_array.pop();
  72 + }
  73 + }
  74 +
  75 + // Default
  76 + if (config[type].default) {
  77 + return cb(null, ip, config[type].default);
  78 + }
  79 +}
  80 +
  81 +exports.lookup_mail_key = function (type, args, cb) {
  82 + var mail = args[0];
  83 + var config = this.config.get('rate_limit.ini');
  84 + if (!config[type] || !mail) {
  85 + return cb();
  86 + }
  87 +
  88 + // Full e-mail address (e.g. smf@fsl.com)
  89 + var email = mail.address();
  90 + if (config[type][email] || config[type][email] === 0) {
  91 + return cb(null, email, config[type][email]);
  92 + }
  93 +
  94 + // RHS parts e.g. host.sub.sub.domain.com
  95 + if (mail.host) {
  96 + var rhs_split = mail.host.toLowerCase().split('.');
  97 + while (rhs_split.length) {
  98 + var part = rhs_split.join('.');
  99 + if (config[type][part] || config[type][part] === 0) {
  100 + return cb(null, part, config[type][part]);
  101 + }
  102 + rhs_split.pop();
  103 + }
  104 + }
  105 +
  106 + // Default
  107 + if (config[type].default) {
  108 + return cb(null, email, config[type].default);
  109 + }
  110 +}
  111 +
  112 +exports.rate_limit = function (connection, key, value, cb) {
  113 + var self = this;
  114 + var limit, ttl;
  115 + if (!key || !value) return cb();
  116 + if (value === 0) {
  117 + // Limit disabled for this host
  118 + connection.loginfo(this, 'rate limit disabled for: ' + key);
  119 + return cb(null, false);
  120 + }
  121 + var match = /^(\d+)(?:\/(\d+)(\S)?)?$/.exec(value);
  122 + if (match) {
  123 + limit = match[1];
  124 + ttl = ((match[2]) ? match[2] : 60); // Default 60s
  125 + if (match[3]) {
  126 + // Unit
  127 + switch (match[3].toLowerCase()) {
  128 + case 's':
  129 + // Default is seconds
  130 + break;
  131 + case 'm':
  132 + ttl *= 60;
  133 + break;
  134 + case 'h':
  135 + ttl *= (60*60);
  136 + break;
  137 + case 'd':
  138 + ttl *= (60*60*24);
  139 + break;
  140 + default:
  141 + // Unknown time unit
  142 + return cb(new Error('unknown time unit \'' + match[3] + '\' key=' + key));
  143 + }
  144 + }
  145 + }
  146 + else {
  147 + // Syntax error
  148 + return cb(new Error('syntax error: key=' + key + ' value=' + value));
  149 + }
  150 +
  151 + connection.logdebug(self, 'key=' + key + ' limit=' + limit + ' ttl=' + ttl);
  152 +
  153 + client.incr(key, function(err, val) {
  154 + if (err) return cb(err);
  155 + connection.logdebug(self, 'key=' + key + ' value=' + val);
  156 + if (parseInt(val) === 1) {
  157 + // New key; set ttl
  158 + client.expire(key, ttl, function (err, result) {
  159 + if (err) {
  160 + connection.logerror(self, err);
  161 + }
  162 + });
  163 + }
  164 + if (parseInt(val) > parseInt(limit)) {
  165 + // Limit breached
  166 + connection.lognotice(self, key + ' rate ' + val + ' exceeds ' + limit + '/' + ttl + 's');
  167 + return cb(null, true);
  168 + }
  169 + else {
  170 + // OK
  171 + return cb(null, false);
  172 + }
  173 + });
  174 +}
  175 +
  176 +// TODO: support this in Redis somehow
  177 +exports.incr_concurrency = function (next, connection) {
  178 + var self = this;
  179 + var config = this.config.get('rate_limit.ini');
  180 + var snotes = connection.server.notes;
  181 +
  182 + // Concurrency
  183 + this.lookup_host_key('concurrency', [connection.remote_ip, connection.remote_host], function (err, key, value) {
  184 + if (err) {
  185 + connection.logerror(self, err);
  186 + return next();
  187 + }
  188 + if (value === 0) {
  189 + connection.logdebug(self, 'concurrency limit disabled for ' + key);
  190 + return next();
  191 + }
  192 + if (!snotes.concurrency) snotes.concurrency = {};
  193 + if (!snotes.concurrency[key]) snotes.concurrency[key] = 0;
  194 + snotes.concurrency[key]++;
  195 + connection.logdebug(self, '[concurrency] key=' + key + ' value=' + snotes.concurrency[key] + ' limit=' + value);
  196 + var count = 0;
  197 + var keys = Object.keys(snotes.concurrency);
  198 + for (var i=0; i<keys.length; i++) {
  199 + count += snotes.concurrency[keys[i]];
  200 + }
  201 + if (snotes.concurrency[key] > value) {
  202 + if (config.main.tarpit_delay) {
  203 + connection.notes.tarpit = config.main.tarpit_delay;
  204 + }
  205 + else {
  206 + return next(DENYSOFT, 'connection concurrency limit exceeded');
  207 + }
  208 + }
  209 + return next();
  210 + });
  211 +}
  212 +
  213 +exports.decr_concurrency = function (next, connection) {
  214 + var self = this;
  215 + var snotes = connection.server.notes;
  216 +
  217 + // Concurrency
  218 + this.lookup_host_key('concurrency', [connection.remote_ip, connection.remote_host], function (err, key, value) {
  219 + if (err) {
  220 + connection.logerror(self, err);
  221 + return next();
  222 + }
  223 + if (!snotes.concurrency) snotes.concurrency = {};
  224 + if (!snotes.concurrency[key]) snotes.concurrency[key] = 0;
  225 + if (snotes.concurrency[key] !== 0) snotes.concurrency[key]--;
  226 + if (snotes.concurrency[key] === 0) delete snotes.concurrency[key];
  227 + var count = 0;
  228 + var keys = Object.keys(snotes.concurrency);
  229 + for (var i=0; i<keys.length; i++) {
  230 + count += snotes.concurrency[keys[i]];
  231 + }
  232 + connection.loginfo(self, count + ' active connections to this child');
  233 + return next();
  234 + });
  235 +}
  236 +
  237 +exports.hook_connect = function (next, connection) {
  238 + var self = this;
  239 + var config = this.config.get('rate_limit.ini');
  240 +
  241 + this.lookup_host_key('rate_conn', [connection.remote_ip, connection.remote_host], function (err, key, value) {
  242 + if (err) {
  243 + connection.logerror(self, err);
  244 + return next();
  245 + }
  246 + // Check rate limit
  247 + self.rate_limit(connection, 'rate_conn:' + key, value, function (err, over) {
  248 + if (err) {
  249 + connection.logerror(self, err);
  250 + return next();
  251 + }
  252 + if (over) {
  253 + if (config.main.tarpit_delay) {
  254 + connection.notes.tarpit = config.main.tarpit_delay;
  255 + }
  256 + else {
  257 + return next(DENYSOFT, 'connection rate limit exceeded');
  258 + }
  259 + }
  260 + // See if we need to tarpit rate_rcpt_host
  261 + if (config.main.tarpit_delay) {
  262 + self.lookup_host_key('rate_rcpt_host', [connection.remote_ip, connection.remote_host], function (err, key, value) {
  263 + if (!err && key && value) {
  264 + var match = /^(\d+)/.exec(value);
  265 + var limit = match[0];
  266 + client.get('rate_rcpt_host:' + key, function (err, result) {
  267 + if (!err && result && limit) {
  268 + connection.logdebug(self, 'rate_rcpt_host:' + key + ' value ' + result + ' exceeds limit ' + limit);
  269 + if (result > limit) {
  270 + connection.notes.tarpit = config.main.tarpit_delay;
  271 + }
  272 + }
  273 + return next();
  274 + });
  275 + }
  276 + else {
  277 + return next();
  278 + }
  279 + });
  280 + }
  281 + else {
  282 + return next();
  283 + }
  284 + });
  285 + });
  286 +}
  287 +
  288 +exports.hook_rcpt = function (next, connection, params) {
  289 + var self = this;
  290 + var config = this.config.get('rate_limit.ini');
  291 + var transaction = connection.transaction;
  292 +
  293 + var chain = [
  294 + {
  295 + name: 'rate_rcpt_host',
  296 + lookup_func: 'lookup_host_key',
  297 + lookup_args: [connection.remote_ip, connection.remote_host],
  298 + },
  299 + {
  300 + name: 'rate_rcpt_sender',
  301 + lookup_func: 'lookup_mail_key',
  302 + lookup_args: [connection.transaction.mail_from],
  303 + },
  304 + {
  305 + name: 'rate_rcpt_null',
  306 + lookup_func: 'lookup_mail_key',
  307 + lookup_args: [params[0]],
  308 + check_func: function () {
  309 + if (transaction && !transaction.mail_from.user) {
  310 + // Message from the null sender
  311 + return true;
  312 + }
  313 + return false;
  314 + },
  315 + },
  316 + {
  317 + name: 'rate_rcpt',
  318 + lookup_func: 'lookup_mail_key',
  319 + lookup_args: [params[0]],
  320 + },
  321 + ];
  322 +
  323 + var chain_caller = function (code, msg) {
  324 + if (code) {
  325 + return next(code, msg);
  326 + }
  327 + if (!chain.length) {
  328 + return next();
  329 + }
  330 + var next_in_chain = chain.shift();
  331 + // Run any check functions
  332 + if (next_in_chain.check_func && typeof next_in_chain.check_func === 'function') {
  333 + if (!next_in_chain.check_func()) {
  334 + return chain_caller();
  335 + }
  336 + }
  337 + self[next_in_chain.lookup_func](next_in_chain.name, next_in_chain.lookup_args, function (err, key, value) {
  338 + if (err) {
  339 + connection.logerror(self, err);
  340 + return chain_caller();
  341 + }
  342 + self.rate_limit(connection, next_in_chain.name + ':' + key, value, function (err, over) {
  343 + if (err) {
  344 + connection.logerror(self, err);
  345 + return chain_caller();
  346 + }
  347 + if (over) {
  348 + // Delay this response if we are not already tarpitting
  349 + if (config.main.tarpit_delay &&
  350 + !(connection.notes.tarpit || (transaction && transaction.notes.tarpit)))
  351 + {
  352 + connection.loginfo(self, 'tarpitting response for ' + config.main.tarpit + 's');
  353 + setTimeout(function () {
  354 + if (connection) {
  355 + return chain_caller(DENYSOFT, 'rate limit exceeded');
  356 + }
  357 + }, config.main.tarpit_delay*1000);
  358 + }
  359 + else {
  360 + return chain_caller(DENYSOFT, 'rate limit exceeded')
  361 + }
  362 + }
  363 + else {
  364 + return chain_caller();
  365 + }
  366 + });
  367 + });
  368 + }
  369 + chain_caller();
  370 +}
33 plugins/tarpit.js
... ... @@ -0,0 +1,33 @@
  1 +// tarpit
  2 +
  3 +exports.register = function () {
  4 + // Register tarpit function last
  5 + var self = this;
  6 + ['connect', 'helo', 'ehlo', 'mail', 'rcpt', 'rcpt_ok', 'data',
  7 + 'data_post', 'queue', 'unrecognized_command', 'vrfy', 'noop',
  8 + 'rset', 'quit'].forEach(function (hook) {
  9 + self.register_hook(hook, 'tarpit');
  10 + });
  11 +}
  12 +
  13 +exports.tarpit = function (next, connection) {
  14 + var transaction = connection.transaction;
  15 + var conn_delay, trans_delay;
  16 + if (transaction && transaction.notes) {
  17 + trans_delay = transaction.notes.tarpit;
  18 + }
  19 + if (connection && connection.notes) {
  20 + conn_delay = connection.notes.tarpit;
  21 + }
  22 + var delay = trans_delay || conn_delay;
  23 + if (delay) {
  24 + connection.loginfo(this, 'tarpitting response for ' + delay + 's');
  25 + setTimeout(function () {
  26 + // Only return if we still have a connection...
  27 + if (connection) return next();
  28 + }, (delay*1000));
  29 + }
  30 + else {
  31 + return next();
  32 + }
  33 +}

0 comments on commit 508d6fe

Please sign in to comment.
Something went wrong with that request. Please try again.