IPSet module

[k9982874]

Created a module to support the ipset that could add the domain's ip to a list easily.

Purposes:

• In my case, I can't access the facebook, twitter, youtube and thousands web site for some reason. VPN is a solution. But the internet too slow whether all traffics pass through the vpn.
  So, I set up a transparent proxy to proxy the traffic which has been blocked only.
  At the final step, I need to install a dns service which would work with ipset well to launch the system.
  I did some research for this. Unfortunately, Unbound, My favorite dns service doesn't support ipset yet. So, I decided to implement it by my self and contribute the patch. It's good for me and the community.

# unbound.conf
server:
  ...
  local-zone: "facebook.com" ipset
  local-zone: "twitter.com" ipset
  local-zone: "instagram.com" ipset
  more social website

ipset:
  name-v4: "gfwlist"

# iptables
iptables -A PREROUTING -p tcp -m set --match-set gfwlist dst -j REDIRECT --to-ports 10800
iptables -A OUTPUT -p tcp -m set --match-set gfwlist dst -j REDIRECT --to-ports 10800

• This patch could work with iptables rules to batch block the IPs.

# unbound.conf
server:
  ...
  local-zone: "facebook.com" ipset
  local-zone: "twitter.com" ipset
  local-zone: "instagram.com" ipset
  more social website

ipset:
  name-v4: "blacklist"
  name-v6: "blacklist6"

# iptables
iptables -A INPUT -m set --set blacklist src -j DROP
ip6tables -A INPUT -m set --set blacklist6 src -j DROP

Notes:

• To enable this module the root privileges is required.
• Please create a set with ipset command first. eg. ipset -N blacklist iphash

How to use:

./configure --enable-ipset
make && make install

Configuration:

# unbound.conf
server:
  ...
  local-zone: "example.com" ipset

ipset:
  name-v4: "blacklist"

[wcawijngaards]

Hi,
This pull request looks real good! Could you spend some words to describe a way that it can be used. In what situation would users use this feature? Or how can the IPset get used (for example)?
From what I read it adds the IP addresses from the *.example.com A and AAAA lookups into the IPset. And that IPset can be used by the Firewall? Is that correct?
Best regards, Wouter

[k9982874]

Hi,
This pull request looks real good! Could you spend some words to describe a way that it can be used. In what situation would users use this feature? Or how can the IPset get used (for example)?
From what I read it adds the IP addresses from the *.example.com A and AAAA lookups into the IPset. And that IPset can be used by the Firewall? Is that correct?
Best regards, Wouter

Thank you for your reply.
Yes, This patch could work with firewall to batch block the IPs.
And, I have already updated the description for the PR. Please check and have fun.

[rexy74]

Hi,
This patch can also be used in the Whitelist of a NAC (Network Access Controler). For example, in our project ALCASAR, we use this behavior with DNSMask. For the next version of ALCASAR, we've decided to replace 'Dnsmasq' with 'Unbound' (due to the better perfs). However, we must keep one instance of 'dnsmasq' (just behind 'unbound') in order to deal with the firewall and its ipset. With this module, we won't need dnsmasq anymore.

[wcawijngaards]

Thank you for the PR! The documentation you write, I have put in doc/README.ipset.md, so that other people can reference it.

[dimqua]

Many thanks for implementing this feature! Do you know the how many local-zone rules can Unbound handle?

[wcawijngaards]

There is no limit on it. It uses memory to store them. You can have as many as you like. If you have a local-zone without local-data in it, it uses less memory. It then also uses more time to read them in. Consider using the include: "file" directive to organize the lists.

[schm0]

Thank you for this patch but I can't get this working.
I want to use this on OpenWRT.
So I modified the make file to enable ipset support and installed the libmnl headers.
Compiled fine.
Edited my unbound conf:

server:
  ...
  module-config: "ipset validator iterator"
  ...
  local-zone: "example.com" ipset
  ...
ipset:
  name-v4: "test4"
  name-v6: "test6"

Created the ipsets:

ipset create test4 iphash
ipset create test6 iphash

But nothing is added to the sets.
The log shows:

[1567384109] unbound[25802:0] notice: init module 0: ipset
[1567384109] unbound[25802:0] notice: init module 1: validator
[1567384109] unbound[25802:0] notice: init module 2: iterator
...
[1567384958] unbound[27613:1] debug: ipset[module 0] operate: extstate:module_wait_module event:module_event_moddone
[1567384958] unbound[27613:1] info: ipset operate: query example.com. A IN
[1567384958] unbound[27613:1] debug: mesh_run: ipset module exit state is module_finished

And is it possible to have multiple ipsets?

[wcawijngaards]

The previous commit should allow you to see in the logs (set verbosity high up for it) when an ipaddress is added to the set.

[wcawijngaards]

The output should look like this (for me in a test), with verbosity 3 or higher:
debug: ipset: add 93.184.216.34 to test4 for www.example.com.

[mercora]

i probably have the same issue as @schm0 although i get the log message as described nothing is appended to the ipset at all. I also found some issues in addition to that.

1. The confgparser.y does not allow local-zone statements of type ipset in views. It appears as it was just forgotten as i can see code added in that block too. I have a patch for this but its trivial. However. neither using this module globally nor in views works for me.

2. Second and in my opinion more serve is that it wont work if i have a local-zone set for "." in my view which is of type "refuse", "static" or similar. While this might seem obvious as there is no zone-data i don't know how to use this module to whitelist IP addresses in my firewall.

What i would like to do is having a special view for clients with restricted internet access. i.e. my TV set should not be able to access the internet by default but there are some cases where i want to allow it based on the response of unbound for i.e. "pool.ntp.org" as far as i can see the way this module is implemented does not allow such configuration at all. Its not a problem if you don't want to avoid responding with addresses that are not routable but doing so makes my TV set a lot less noisy with regards to connection attempts (that will fail repeatedly).

3. The documentation is missing that we need to add this module to the module-config line. If you already have setup another module before you probably know already about this but i was left confused at first. Moreover i recognized something strange. If i omit that line i can see in the log that 3 modules are loaded "subnet", "validator" and "iterator". However, adding the module-config line does not allow me to load the subnet module (whatever it does) and denies startup with this message:
   fatal error: module_conf lists module 'subnet' but that module is not available.
   But this is probably completely unrelated to this module and should be addressed in a separate ticket.

[mercora]

Additionally i would like to add that i would have a use-case for multiple ipset lists too which currently is not possible either.

[k9982874]

@mercora Thank you for your suggestion. Basically, I think to support the multiple ipset is the answer to solve the problem that you facing.
Unfortunately, This patch does not aim for that at the beginning.
To support the multiple ipset need to do a lot of works on the structure level that the reason why I don't implement it.
For me, I just need unbound could add ip address into the ipset that's all.

[mercora]

ok,sounds reasonable. Then lets find out why it is not working as is for now :)

[wcawijngaards]

Hi,
@k9982874 I hope you can see this comment here.

We would like to include the ipset module under the BSD license. It was contributed by you (Kevin Chou) but you did not mention the license. Is it okay to include the ipset module under the 3-clause BSD license that we use for Unbound?

Best regards, Wouter

[k9982874]

@wcawijngaards No worries. It's ok.

[wcawijngaards]

Thanks for the quick reply! We try to keep our licensing as clear as possible.
Best regards, Wouter

[paulgear]

Hi all, are there plans for the ipset support in this module to be extended to allow multiple ipsets, for example to maintain a block list for some domains and an allow list for others?

[hatemosphere]

@wcawijngaards question from a noob - how does the update loop look like here? i mean what triggers queries for the domains specified in local-zone after initial startup query is done? i was trying to answer this question by looking at the code base, but failed miserably.

[wcawijngaards]

@paulgear No plans at the present for multiple ipsets. We have contributed code.

After the lookup to the internet is complete for the query, the modules are done. At that point the ipset.c:ipset_update() function is called with the just newly looked up query answer. This scans the message for the names listed in the local-zone statements, and then adds them to the ipset in ipset.c:add_to_ipset().