Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPSet module #28

Merged
merged 6 commits into from Jun 18, 2019

Conversation

@k9982874
Copy link
Contributor

commented May 3, 2019

Created a module to support the ipset that could add the domain's ip to a list easily.

Purposes:

  • In my case, I can't access the facebook, twitter, youtube and thousands web site for some reason. VPN is a solution. But the internet too slow whether all traffics pass through the vpn.
    So, I set up a transparent proxy to proxy the traffic which has been blocked only.
    At the final step, I need to install a dns service which would work with ipset well to launch the system.
    I did some research for this. Unfortunately, Unbound, My favorite dns service doesn't support ipset yet. So, I decided to implement it by my self and contribute the patch. It's good for me and the community.
# unbound.conf
server:
  ...
  local-zone: "facebook.com" ipset
  local-zone: "twitter.com" ipset
  local-zone: "instagram.com" ipset
  more social website

ipset:
  name-v4: "gfwlist"
# iptables
iptables -A PREROUTING -p tcp -m set --match-set gfwlist dst -j REDIRECT --to-ports 10800
iptables -A OUTPUT -p tcp -m set --match-set gfwlist dst -j REDIRECT --to-ports 10800
  • This patch could work with iptables rules to batch block the IPs.
# unbound.conf
server:
  ...
  local-zone: "facebook.com" ipset
  local-zone: "twitter.com" ipset
  local-zone: "instagram.com" ipset
  more social website

ipset:
  name-v4: "blacklist"
  name-v6: "blacklist6"
# iptables
iptables -A INPUT -m set --set blacklist src -j DROP
ip6tables -A INPUT -m set --set blacklist6 src -j DROP

Notes:

  • To enable this module the root privileges is required.
  • Please create a set with ipset command first. eg. ipset -N blacklist iphash

How to use:

./configure --enable-ipset
make && make install

Configuration:

# unbound.conf
server:
  ...
  local-zone: "example.com" ipset

ipset:
  name-v4: "blacklist"
k9982874 added 5 commits May 2, 2019
@wcawijngaards

This comment has been minimized.

Copy link
Member

commented May 6, 2019

Hi,
This pull request looks real good! Could you spend some words to describe a way that it can be used. In what situation would users use this feature? Or how can the IPset get used (for example)?
From what I read it adds the IP addresses from the *.example.com A and AAAA lookups into the IPset. And that IPset can be used by the Firewall? Is that correct?
Best regards, Wouter

@k9982874

This comment has been minimized.

Copy link
Contributor Author

commented May 7, 2019

Hi,
This pull request looks real good! Could you spend some words to describe a way that it can be used. In what situation would users use this feature? Or how can the IPset get used (for example)?
From what I read it adds the IP addresses from the *.example.com A and AAAA lookups into the IPset. And that IPset can be used by the Firewall? Is that correct?
Best regards, Wouter

Thank you for your reply.
Yes, This patch could work with firewall to batch block the IPs.
And, I have already updated the description for the PR. Please check and have fun.

@rexy74

This comment has been minimized.

Copy link

commented May 12, 2019

Hi,
This patch can also be used in the Whitelist of a NAC (Network Access Controler). For example, in our project ALCASAR, we use this behavior with DNSMask. For the next version of ALCASAR, we've decided to replace 'Dnsmasq' with 'Unbound' (due to the better perfs). However, we must keep one instance of 'dnsmasq' (just behind 'unbound') in order to deal with the firewall and its ipset. With this module, we won't need dnsmasq anymore.

@wcawijngaards wcawijngaards merged commit 19c2ece into NLnetLabs:master Jun 18, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
wcawijngaards added a commit that referenced this pull request Jun 18, 2019
- PR #28: IPSet module, by Kevin Chou. Created a module to support
  the ipset that could add the domain's ip to a list easily.
  Needs libmnl, and --enable-ipset and config it, doc/README.ipset.md.
- Fix to omit RRSIGs from addition to the ipset.
@wcawijngaards

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

Thank you for the PR! The documentation you write, I have put in doc/README.ipset.md, so that other people can reference it.

jedisct1 added a commit to jedisct1/unbound that referenced this pull request Jun 29, 2019
Merge remote-tracking branch 'nlnet/master'
* nlnet/master: (22 commits)
  Nicer spelling and layout.
  - For NLnetLabs#45, check that 127.0.0.1 and ::1 are not used in unbound.conf   when do-not-query-localhost is turned on, or at default on,   unbound-checkconf prints a warning if it is found in forward-addr or   stub-addr statements.
  - Fix memleak in unit test, reported from the clang 8.0 static analyzer.
  - Fix python dict reference and double free in config.
  - Merge PR NLnetLabs#6: Python module: support multiple instances - Merge PR NLnetLabs#5: Python module: define constant MODULE_RESTART_NEXT - Merge PR NLnetLabs#4: Python module: assign something useful to the   per-query data store 'qdata' Noted in Changelog.
  - Added documentation to the ipset files (for doxygen output).
  - make depend
  - Fix to make unbound-control with ipset, remove unused variable,   use unsigned type because of comparison, and assign null instead   of compare with it.  Remade lex and yacc output.
  - PR NLnetLabs#28: IPSet module, by Kevin Chou.  Created a module to support   the ipset that could add the domain's ip to a list easily.   Needs libmnl, and --enable-ipset and config it, doc/README.ipset.md. - Fix to omit RRSIGs from addition to the ipset.
  - Fix for NLnetLabs#24: Fix abort due to scan of auth zone masters using old   address from previous scan.
  - Fix NLnetLabs#39: In libunbound, leftover logfile is close()d unpredictably.
  - Master contains version 1.9.3 in development.
  fix segmentation fault
  rollback the code
  bugfix
  performance improvement
  edit config parser to support ipset
  Add support for ipset
  Document how to configure multiple python modules
  Support multiple python module instances
  ...
@dimqua

This comment has been minimized.

Copy link

commented Aug 29, 2019

Many thanks for implementing this feature! Do you know the how many local-zone rules can Unbound handle?

@wcawijngaards

This comment has been minimized.

Copy link
Member

commented Aug 29, 2019

There is no limit on it. It uses memory to store them. You can have as many as you like. If you have a local-zone without local-data in it, it uses less memory. It then also uses more time to read them in. Consider using the include: "file" directive to organize the lists.

@schm0

This comment has been minimized.

Copy link

commented Sep 2, 2019

Thank you for this patch but I can't get this working.
I want to use this on OpenWRT.
So I modified the make file to enable ipset support and installed the libmnl headers.
Compiled fine.
Edited my unbound conf:

server:
  ...
  module-config: "ipset validator iterator"
  ...
  local-zone: "example.com" ipset
  ...
ipset:
  name-v4: "test4"
  name-v6: "test6"

Created the ipsets:

ipset create test4 iphash
ipset create test6 iphash

But nothing is added to the sets.
The log shows:

[1567384109] unbound[25802:0] notice: init module 0: ipset
[1567384109] unbound[25802:0] notice: init module 1: validator
[1567384109] unbound[25802:0] notice: init module 2: iterator
...
[1567384958] unbound[27613:1] debug: ipset[module 0] operate: extstate:module_wait_module event:module_event_moddone
[1567384958] unbound[27613:1] info: ipset operate: query example.com. A IN
[1567384958] unbound[27613:1] debug: mesh_run: ipset module exit state is module_finished

And is it possible to have multiple ipsets?

wcawijngaards added a commit that referenced this pull request Sep 2, 2019
@wcawijngaards

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

The previous commit should allow you to see in the logs (set verbosity high up for it) when an ipaddress is added to the set.

@wcawijngaards

This comment has been minimized.

Copy link
Member

commented Sep 2, 2019

The output should look like this (for me in a test), with verbosity 3 or higher:
debug: ipset: add 93.184.216.34 to test4 for www.example.com.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.