Quotas

The quotas module allows to set arbitrary limits on various factors, where the limits can be different per (predefined) factor value.

A sliding window with an accuracy of 1 second is used. Common use cases are

• rate limiting (X messages per 30 seconds)
• or e.g. daily quotas for customers per 24 hours (scenario outlined below)

If a message has multiple recipient, the number of recipients will be accounted for, rather than just the number of messages.

Currently supported factors:

• Client Address: The IP of the connecting client.
• Sender: The email address of the party sending the email.
• Sender Domain: The domain part of the sending email address
• Sender SLD: The Second-Level Domain of the sender (e.g. example.org for foo.bar.example.org)
• Recipient: The email address of the recipient.
• Recipient Domain: The domain part of the recipient.
• Recipient SLD: The Second-Level Domain of the recipient.
• Sasl Username: The SASL Username that was used to connect with.

Each combination of predefined factor and factor value (stored in the quota table) is assigned a quota profile. Next, a quota profile has one or more profile periods. These periods determine the maximum amount of messages accepted over that period.

For example, say you're an ESP that has two offerings (packages large and small) and you're using SASL authentication. Your user john@doe.com has the small package, the jane@doe.com SASL user pays for the large package. With the large package you can send 500 emails per 5 minutes, and a total of 10.000 per 24 hours. The small package allows for a total of 150 messages per 24 hours.

To implement this scenario you'd make sure your database contains the following entries.

quota:
+-----+----------------+----------------+----------+---------+
| id  | selector       | value          | is_regex | profile |
+-----+----------------+----------------+----------+---------+
|   1 | sasl_username  | john@doe.com   |        0 |       1 |
|   2 | sasl_username  | jane@doe.com   |        0 |       2 |
+-----+----------------+----------------+----------+---------+

quota_profile:
+----+----------------------------+
| id | class | name               |
+----+----------------------------+
|  1 |     1 | small-sasl         |
|  2 |     1 | large-sasl         |
+----+----------------------------+

quota_class:
+----+-------------------------------+
| id | instance | name               |
+----+-------------------------------+
|  1 |        1 | Paying Customers   |
+----+-------------------------------+

quota_profile_period:
+----+---------+--------+-------+
| id | profile | period | curb  |
+----+---------+--------+-------+
|  1 |       1 |  86400 |   150 |
|  2 |       2 |    300 |   500 |
|  3 |       2 |  86400 | 10000 |
+----+---------+--------+-------+

The quota_class table allows to group multiple quota profiles together. This can be used to automatically promote a user to another quota profile if the volume of legit email increases, as long as the other quota profile is in the same quota class.

Regexes

Some times it's not possible to know all the factor values that you need a quota for in advance. For example, when you want to do rate limiting based on IP addresses. For this reason, you can use a regex in the quota table,

That could look like this:

+-----+----------------+----------------+----------+---------+
| id  | selector       | value          | is_regex | profile |
+-----+----------------+----------------+----------+---------+
|   1 | client_address | 127.0.0.1      |        0 |       3 |
|   2 | client_address | ^.*$           |        1 |       4 |
+-----+----------------+----------------+----------+---------+

ClueGetter will first use the regular (non-is_regex) quotas if they're available. If they're not available, it will scan the regexes to see if any of these match. You can have millions of regular quotas, but it is generally not expected to have more than a few regex based quotas. Performance will be O(N).

Redis

In early versions of ClueGetter all sliding windows and current usages were accounted for using the RDBMS. It turned out though that the ACID capabilities the RDBMS provides were both unnecessary and somewhat of a bottleneck once a particular IP started spamming hundreds of messages per second. As such, you're now strongly advised to enable Redis when you want to use the quota functionality. Using the RDBMS without Redis will soon not be supported any more with the Quotas module.

Currently, ClueGetter will automatically import all quotas every few minutes from the RDBMS into Redis. Just insert it into the RDBMS and ClueGetter will pick it up from there. The quotas module was designed in such a way that it will continue functioning, even if the RDBMS becomes unavailable.

Configuration

[quotas]
# Enable the Quota module
enabled = false

# Factors to account for. 
account-client-address = true
account-sender = false
account-sender-domain = false
account-sender-sld = false
account-recipient = false
account-recipient-domain = false
account-recipient-sld = false
account-sasl-username = false