Working with traps

Costa Tsaousis edited this page Feb 21, 2015 · 23 revisions

FireHOL installation


FireQOS


Link Balancer - routing tables with inheritance, multiple balancing gateways, routing rules


FireHOL & iptables marks


FireHOL & ipsets


FireHOL & SYNPROXY (DDoS mitigation)


FireHOL with basic IDS - just with plain iptables and ipsets

Clone this wiki locally

Working with traps


For this feature set you need the latest version of firehol, currently in github. You can get it with this:

# git clone https://github.com/ktsaou/firehol.git firehol.git

Once you do this, you will have firehol in firehol.git/sbin/firehol.in. You can run it from there.


There is a feature of recent iptables and kernels that allows us to build some sort of an IDS (intrusion detection system).

It works like this:

  1. Linux kernel supports ipsets. Please check Working With IPSETs if you don't know what ipsets are.

  2. iptables supports the SET action. This action allows adding and removing IPs to ipsets, dynamically, from within the firewall itself. To see its options check: iptables -j SET --help

  3. We could therefore use these features in several ways:

    • Detect network-wide port scans There are hundreds of people scanning the internet for unupdated and unsecured applications. These people usually focus on one port on all possible IPs. What we want, is to find them and block them from our network for some time. To do it, we will create an empty ipset named trap and when we detect that someone is trying to access forbidden services (i.e. we don't provide a public mysql server, but we see someone trying to connect to a mysql on our host or network), we could add his IP on the trap ipset for a day, or a week (this timeout is configurable per IP and is handled by the kernel, which will automatically remove the IP from the ipset after the timeout). So, after a few days we will have a large blacklist of all the attackers found. Another way of achieving the same (even a little better) is if you have a few spare public IPs. If you have unused public IPs, then you could trap all those trying to connect anywhere on these IPs. This is better because you will be able update your blacklist with all those scanning any port.

    • Detect single host port scans The second way of using it is for detecting port scans on one host. To do it, we will change the policy of the internet interface. Normally you have either REJECT or DROP on the internet side. We will replace this with a custom action that when it detects that someone is trying to reach something that is not allowed, will add his IP on a new ipset named policytrap. This ipset does not block him. It just counts how many times he has tried to connect to something we didn't allow. Once we have the policytrap ipset that counts the illegal accesses, we will add a rule to the firewall so that all IPs that have a count of more than 50, in less than an hour, will be automatically added to the trap ipset, that blocks them.

    • Knock-knock Another way of using ipsets and traps, is to provide sequences of ports that once "knocked" in a short time and in proper sequence by an IP, this IP will be allowed to access a service, or just be removed from the blacklist. This will work like this: You attempt to access port A and the firewall adds you to an ipset called step1 for 30 seconds. Now there is another rule that says if anyone on step1 accesses port B, then his IP is added on the ipset step2. We can continue adding steps as needed. These ipsets (step1, step2, etc) can be used by services to provide different levels of access (for example, the intranet web server requires 3 steps, but the ssh server requires 6).

The examples bellow refer to firehol.conf.


Decide which is your public interface

It is very important to apply all these rules on your internet interface. If, even by error, you apply it on your VPN or intranet link you may end up locked out.

world="dsl0"

We will use the ${world} variable from now on.

Keep in mind that if you have multiple internet interfaces, you can just add them to the same variable. Just separate them with spaces, but keep then inside the quotes (like this: world="dsl0 ppp12 eth4").


Whitelist

Before we proceed you should create a whitelist. The whitelist should include all IPs you know that should never be blacklisted for any reason.

world="dsl0"

# create the whitelist
ipset4 create whitelist hash:net
ipset4 add whitelist "IPs IN WHITELIST"

To avoid any accidental blacklisting of private IP, I always add ${PRIVATE_IPS} in this list, like this: ipset4 add whitelist "1.2.3.4 5.6.7.8 ${PRIVATE_IPS}".


Traps

To setup the traps we will use the iptrap helper. This works like this:

iptrap ipset type seconds [ timeout | counters ] [ rule-params ] [ except rule-params ]

  • ipset is the name of the ipset to use. If the ipset does not exist, the iptrap helper will create it.
  • type is either src or dst and defines which IP of the matching packets will be used.
  • seconds is the duration in seconds each IP address will remain in the ipset.
  • timeout and counters are mutualy exclusive. They control how IPs that are already in the ipset will be handled. timeout is the default and means that for every packet matching the IP address already in the ipset, its seconds will be reset and restart counting down, while counters means that for every packet matching the IP address already in the ipset, its packets and bytes counters will be incremented. Unfortunately, the kernel cannot do both.

The iptrap helper will compare packets against its first rule-params. If they match, and except rules have been given it will also check that the packets do NOT match against the excepted rule-params and then, it will add the IP of the packet (src or dst depending on type) to the ipset given, for timeout seconds.

Thats it. It does nothing more. The packet will continue to the next rule. It will not be dropped, rejected or accepted by the iptrap helper.

Now lets create our first iptrap. Remember this will not block anything. It will just add a few IPs to an, otherwise, unused ipset.

world="dsl0"

# create the whitelist
ipset4 create whitelist hash:net
ipset4 add whitelist "IPs IN WHITELIST"

# trap the source IP of packets, going to tcp/3306
iptrap4 trap src 86400 log "TRAP MYSQL" \
    inface "${world}" proto tcp dport 3306 \
    src not ipset:whitelist state NEW


# ... rest of firehol.conf ...

Lets see what this statement says:

  • create an ip address trap (iptrap)
  • using the ipset trap (the helper will create it)
  • using the source (src) IP address of packets
  • coming in from the internet (inface "${world}")
  • going to mysql (proto tcp dport 3306)
  • which are not in our whitelist (src not ipset:whitelist)
  • and are not part of an existing connection (state NEW) [this is important, because a valid client you run may use port 3306 for making an outgoing call - we don't want to trap the reply]
  • for each packet matched and trapped, "TRAP MYSQL" will appear in the system logs (log "TRAP MYSQL").
  • each IP address will remain in the ipset trap for 86400 seconds (a day).

You can apply this. It will not harm. Keep in mind that you need the latest firehol from github.

Once applied, on a remote console (not on your whitelist) run:

telnet YOUR_PUBLIC_IP 3306

It doesn't matter that you don't have mysql. The firewall will work.

Now on your server, run on a console:

# ipset list trap

You should see the IP of the computer your run telnet from.

Once you have this working you can add more ports. I usually trap:

  • tcp/3306 (mysql)
  • tcp/21 (ftp)
  • tcp/22 (ssh)
  • tcp/23 (telnet)
  • tcp/3128 and tcp/8080 (proxy)
  • tcp/5038 (asterisk admin)
  • tcp/111 and udp/111 (portmap)
  • tcp/5060 and udp/5060 (SIP)
  • tcp/5900 (vnc)
  • tcp/1433 (mssql)
  • tcp/3389 and udp/3389 (ms rdp)

You can write a small loop if you dont want to write too much:

world="dsl0"

# create the whitelist
ipset4 create whitelist
ipset4 add whitelist "IPs IN WHITELIST"

for x in tcp/3306 tcp/21 tcp/22 tcp/23 tcp/3128 tcp/8080 tcp/5038 tcp/111 udp/111 tcp/5060 udp/5060 tcp/5900 tcp/1433 tcp/3389 udp/3389
do
    iptrap4 trap src 86400 \
        log "TRAP ${x}" \
        inface "${world}" proto ${x/\/*/} dport ${x/*\//} \
        src not ipset:whitelist state NEW
done

Now start your firewall and leave it running. After a few minutes you will start seeing IPs in the trap ipset and your logs.

This is mine, after just 5 minutes:

# ipset list trap
Name: trap
Type: hash:ip
Revision: 2
Header: family inet hashsize 1024 maxelem 65536 timeout 3600
Size in memory: 17624
References: 20
Members:
92.101.92.27 timeout 863029
141.212.121.141 timeout 863946
94.102.63.55 timeout 862687
190.146.92.97 timeout 863625
189.121.206.224 timeout 862676
110.153.0.208 timeout 863361
61.156.202.137 timeout 862740
117.21.173.31 timeout 863914
112.228.22.176 timeout 862643
50.77.102.77 timeout 862963
118.122.95.137 timeout 862804
175.101.66.73 timeout 862795
212.7.192.136 timeout 862691

If you whois a few of the IPs you will get the idea: China (most of them), Russia, Colombia, a few Universities and virtual machine hosting companies. All these scanned my udp/5060 (SIP), tcp/23 (TELNET), tcp/3306 (mysql) and the other trapped ports.

Personally, I also have a few spare IPs. Public internet IPs that I never used, that do not have a dns name and are not routed to any real machine. I have setup on them a trap (on the router) for all possible ports. That ipset is 3 times longer!

Before blocking IPs we should prepare a way of unblocking us, so that in case we get locked out, we will have a way to get back in:


knock-knock

Using a very similar procedure we can create a method for removing ourselves from the trap ipset.

Lets see what this does:

iptrap4 knock1 src 30 \
    inface "${world}" proto tcp dport 12345 \
    log "KNOCK 1"

This adds to ipset knock1, for 30 seconds, the source IP addresses of packets, which are, coming in from the internet and going to port tcp/12345. It is very similar to what we have done before.

We don't want match state NEW with this, so that every packet to this port will be examined.

Lets add another one:

iptrap4 knock2 src 30 \
    src ipset:knock1 proto tcp dport 23456 \
    log "KNOCK 2"

This one is tricky. Instead of saying inface ${world} we said src ipset:knock1, so only the IPs already in knock1 are eligible for knock 2.

And another step:

iptrap4 knock3 src 30 \
    src ipset:knock2 proto tcp dport 34567 \
    log "KNOCK 3"

This one allows all IPs already in knock2 to enter knock3.

You get the idea. We just copy IPs from ipset to ipset. Each knock needs a different dport and a different ipset. The timeout we give is very small. 30 seconds to get from knock to knock.

Of course we can add as many steps as we like.

Without adding more rules, we could say somewhere in the firewall:

...
    server ssh accept src ipset:knock3
...

Using just this, our ssh server will allow connections only from the IPs in knock3. And to get in knock3 to have to be in knock2... and in knock1... and move from knock1 to knock3 in 60 seconds (30 seconds per step).

So, we can have different servers requiring different levels of protection like this:

...
    server ssh accept src ipset:knock3
    server pptpd accept src ipset:knock1
...

Using the above, our pptpd requires just one knock, while ssh requires 3.

Now lets go back to "untrapping us".

There is another helper called untrap. It is the same with trap but it does not need a timeout. It can be used like this:

ipuntrap4 trap src \
    src ipset:knock3 proto tcp dport 45678 \
    log "UNTRAP"

This is the 4th step (its a new dport). If you knock it, and you are in knock3 you will be removed from the trap ipset, the one we will use for blacklisting IPs.

Lets see all the config together:

world="dsl0"

# create the whitelist
ipset4 create whitelist hash:net
ipset4 add whitelist "IPs IN WHITELIST"


# auto-blacklisting
for x in tcp/3306 tcp/21 tcp/22 tcp/23 tcp/3128 tcp/8080 tcp/5038 tcp/111 udp/111 tcp/5060 udp/5060 tcp/5900 tcp/1433 tcp/3389 udp/3389
do
    iptrap4 trap src 86400 \
        log "TRAP ${x}" \
        inface "${world}" proto ${x/\/*/} dport ${x/*\//} \
        src not ipset:whitelist state NEW
done

# knock knock
iptrap4 knock1 src 30 \
    inface "${world}" proto tcp dport 12345 \
    log "KNOCK 1"

iptrap4 knock2 src 30 \
    src ipset:knock1 proto tcp dport 23456 \
    log "KNOCK 2"

iptrap4 knock3 src 30 \
    src ipset:knock2 proto tcp dport 34567 \
    log "KNOCK 3"

ipuntrap4 trap src \
    src ipset:knock3 proto tcp dport 45678 \
    log "UNTRAP"

Apply it and try it. Attempt to get into the trap ipset (you will not be locked out - we have not added the blacklist yet) and knock the ports in sequence to get out. Verify it works by running ipset list on your console.

Once you are confident that it works, lets drop some packets...


Blacklisting

Just one command will blacklist all the IPs in the trap ipset.

Before adding this command, please make sure that you have tried to enter and leave the trap ipset successfully. The place you will add this command in firehol.conf is important. You have to put it below the untrap command. If you put it above, you will not be able to be untrapped.

blacklist4 full log "BLACKLIST TRAP" \
        inface "${world}" src ipset:trap \
        except src ipset:whitelist

It is easy to understand what this does. except src ipset:whitelist is redundant, I know. The ipset trap cannot have any of the IPs of the ipset whitelist since all the commands that add IPs to trap have already done this check. I add it because it is better to be safe than sorry.

The first argument to blacklist is full. full means you will not have any communication with the blacklisted hosts. If you replace full with input, then you will be able to connect to them, but they will not be able to connect to you.

Remember, add the blacklist command below ipuntrap.

This is it. Network scanning blacklisting and knock functionality.

Lets now move to trapping single host port scans.

Keep in mind this is the only blacklist command in this document. Only this command drops packets. All the other commands are just adding or removing IPs from ipsets. Without the blacklist command active, you can test everything without any risk for getting locked out of your systems.


Single host port scans

In order to detect single host port scans we need to examine all the packets not accepted by the firewall. What happens to these packets is controlled by the policy command. You can add one such command for each interface and router you have in your firehol.conf.

The default for interfaces is policy DROP. The default for routers is policy RETURN (routers do not drop or reject traffic by default - traffic in routers traverses all the routers you have and is dropped at the end of the firewall).

What can we use for a policy that will allow us to examine all packets rejected/dropped and possibly add them to an ipset?

Well, another new feature of FireHOL, is an extended action helper. The action helper allows us to define new actions like this:

action name [ table table_name ] type type_params [ next [ type type_params [ next ... ] ] ]

It seems complicated, it is not.

  • name is a name for this action. Lets use TRAP_AND_DROP.
  • table is optional, it defines a list of netfilter tables to add the action to. The default is filter which is ok for our needs.
  • type is the magic. It can be chain (or action which is an alias for chain), rule or iptrap. Each of these has its own type_params. Check the manual page in github repo (doc/firehol/firehol-action.5.md).
  • next is used to link multiple action types together.

This is what we need:

action4 TRAP_AND_DROP \
        iptrap policytrap src 3600 counters log "POLICYTRAP" \
            inface "${world}" state NEW \
            except src ipset:whitelist \
        next action DROP

The above defines a new action called TRAP_AND_DROP.

When packets are given to this action, they are first examined by an iptrap that tries to find if they are coming from dsl0, have a state NEW (they are not part of existing connections) and their source IP is not in the ipset whitelist. If all these are true, then it adds the source IP of the packet to a new ipset called policytrap for an hour (3600 seconds).

The counters keyword is very important. It makes iptrap increase the packets and bytes counters of the IPs that are already in the ipset. Without it, the default is to reset the timeout of the IPs already in the ipset.

Since ipset does not take any filtering action on the traffic, the next action that will take the packets is DROP.

So, what we have done here is that we created an action, that drops traffic, but before doing this, conditionally it adds the IPs we are interested in the ipset policytrap.

Try it. It does not blacklist anything yet. Change the policy of your internet interface to TRAP_AND_DROP.

(If you now use REJECT for your internet interface policy, just change the last DROP to REJECT and rename it to TRAP_AND_REJECT).

When you do this, you will be able to see what gets into the policytrap ipset and decide what to do next. This is mine:

# ipset list policytrap
Name: policytrap
Type: hash:ip
Revision: 2
Header: family inet hashsize 1024 maxelem 65536 timeout 3600 counters
Size in memory: 50776
References: 2
Members:
92.98.88.218 timeout 2616 packets 1 bytes 90
188.138.17.205 timeout 2375 packets 1 bytes 40
37.6.206.145 timeout 2739 packets 9 bytes 540
79.65.73.164 timeout 2306 packets 11 bytes 639
41.160.86.241 timeout 2646 packets 3 bytes 1044
210.101.73.45 timeout 2486 packets 15 bytes 1380
79.196.34.242 timeout 2438 packets 9 bytes 534
213.231.139.106 timeout 2644 packets 1 bytes 90
93.184.211.2 timeout 2408 packets 1 bytes 75
68.184.31.14 timeout 3273 packets 7 bytes 420
78.151.154.134 timeout 3239 packets 11 bytes 630
174.35.32.136 timeout 2485 packets 9 bytes 828

I have quite a few IPs in it. Check the details. Each IP has a timeout, a packets counter and total bytes.

The packet counter is what we are interested in. It actually says how many times this IP has reached the policy of the interface.

Before doing anything more, use an online service and do a port scan on your IP. Once the port scan completes, check the packet counter of the IP that scanned you, on this ipset. It should be very high.

I suggest to temporarily comment out the blacklist line we added above. If the port scanner hits a port we have trapped, it will enter the trap ipset and be blocked before completing the test. So, for the test, just comment out the blacklist helper we added in the previous section.

Once you have tested it and feel comfortable with it, it is time to blacklist all the IPs in this set we consider a threat.

Personally, I consider a threat all IPs that have more than 50 packets dropped in an hour. Our blacklist will not wait for an hour. We will keep the IP in the policytrap ipset for an hour, but the blacklist will block the IP the moment it receives the 51st packet from it.

To do it, we need this:

iptrap4 trap src 86400 log "MOVED FROM POLICYTRAP TO TRAP" \
        inface "${world}" state NEW \
        ipset policytrap src packets-above 50 \
        except src ipset:trap,ipset:whitelist

Another tricky command. This one is an iptrap like all the ones we have seen so far. It will add an IP to the trap ipset for a day (86400 seconds), much like all the previous iptraps did.

The trick here is the ipset match. The line ipset policytrap src packets-above 50 says to match the packet's source IP with all the IPs in the policytrap ipset, only if the packets counter for this IP in the ipset is above 50.

So, this iptrap command copies once more an IP from one ipset to another, but it is smart enough to do it only when the packets counter on the source ipset is above 50.

Now, with the blacklist command disabled, do the port scan again. The IP of the port scanner should now be in trap ipset, which means that the blacklist helper, once activated, will block it.

Important

In such a huge network such as the Internet, it is probable that you will encounter all kinds of problems with network traffic. Trapping and blacklisting IPs on the policy of an interface, means that even legitimate clients that face some network related issue may be trapped.

We should make the most to avoid such scenarios. Hopefully, FireHOL provides a few such ways:

  1. Add protection strong or if your don't want this (strong applies a rate-limit on the connections), at least add protection new-tcp-w/o-syn to your internet facing interface and router. This will drop TCP packets that lost their way to you, buffered at some router and came reordered and too late. They are already useless, since the connection tracker has already decided they are not part of an existing connection.

  2. The connection tracker sometimes closes sockets too early and when the other party sends TCP ACK+FIN packets, they get dropped. Make sure you have FIREHOL_DROP_ORPHAN_TCP_ACK_FIN=1 (this is already the default) so that TCP ACK+FIN packets do not reach our TRAP_AND_DROP action.

You can monitor which IPs get trapped due to the TRAP_AND_DROP action, by checking your iptables logs for lines with the text: "MOVED FROM POLICYTRAP TO TRAP".


All together

The whole configuration this this:

world="dsl0"

# ----------------------------------------------------
# create the whitelist
ipset4 create whitelist hash:net
ipset4 add whitelist "IPs IN WHITELIST"


# ----------------------------------------------------
# auto-blacklisting
for x in tcp/3306 tcp/21 tcp/22 tcp/23 tcp/3128 tcp/8080 tcp/5038 tcp/111 udp/111 tcp/5060 udp/5060 tcp/5900 tcp/1433 tcp/3389 udp/3389
do
    iptrap4 trap src 86400 \
        log "TRAP ${x}" \
        inface "${world}" proto ${x/\/*/} dport ${x/*\//} \
        src not ipset:whitelist state NEW
done


# ----------------------------------------------------
# knock knock
iptrap4 knock1 src 30 \
    inface "${world}" proto tcp dport 12345 \
    log "KNOCK 1"

iptrap4 knock2 src 30 \
    src ipset:knock1 proto tcp dport 23456 \
    log "KNOCK 2"

iptrap4 knock3 src 30 \
    src ipset:knock2 proto tcp dport 34567 \
    log "KNOCK 3"

ipuntrap4 trap src \
    src ipset:knock3 proto tcp dport 45678 \
    log "UNTRAP"


# ----------------------------------------------------
# move IPs from policytrap ipset to trap ipset
iptrap4 trap src 86400 log "MOVED FROM POLICYTRAP TO TRAP" \
        inface "${world}" state NEW \
        ipset policytrap src packets-above 50 \
        except src ipset:trap,ipset:whitelist


# ----------------------------------------------------
# the custom policy action
action4 TRAP_AND_DROP \
        iptrap policytrap src 3600 counters log "POLICYTRAP" \
            inface "${world}" state NEW \
            except src ipset:whitelist \
        next action DROP


# ----------------------------------------------------
# the blacklist
# this is the only command that drops traffic
blacklist4 full log "BLACKLIST TRAP" \
        inface "${world}" src ipset:trap \
        except src ipset:whitelist


# ----------------------------------------------------
# our internet interface
interface "${world}" world
    policy TRAP_AND_DROP
    protection new-tcp-w/o-syn

    ...

I keep the blacklist just before the filtering rules and below of iptraps. This is because we need it to be the last action performed, after all knocks, traps and untraps.