Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPSet module #28

Merged
merged 6 commits into from Jun 18, 2019
Merged

IPSet module #28

merged 6 commits into from Jun 18, 2019

Conversation

k9982874
Copy link
Contributor

@k9982874 k9982874 commented May 3, 2019

Created a module to support the ipset that could add the domain's ip to a list easily.

Purposes:

  • In my case, I can't access the facebook, twitter, youtube and thousands web site for some reason. VPN is a solution. But the internet too slow whether all traffics pass through the vpn.
    So, I set up a transparent proxy to proxy the traffic which has been blocked only.
    At the final step, I need to install a dns service which would work with ipset well to launch the system.
    I did some research for this. Unfortunately, Unbound, My favorite dns service doesn't support ipset yet. So, I decided to implement it by my self and contribute the patch. It's good for me and the community.
# unbound.conf
server:
  ...
  local-zone: "facebook.com" ipset
  local-zone: "twitter.com" ipset
  local-zone: "instagram.com" ipset
  more social website

ipset:
  name-v4: "gfwlist"
# iptables
iptables -A PREROUTING -p tcp -m set --match-set gfwlist dst -j REDIRECT --to-ports 10800
iptables -A OUTPUT -p tcp -m set --match-set gfwlist dst -j REDIRECT --to-ports 10800
  • This patch could work with iptables rules to batch block the IPs.
# unbound.conf
server:
  ...
  local-zone: "facebook.com" ipset
  local-zone: "twitter.com" ipset
  local-zone: "instagram.com" ipset
  more social website

ipset:
  name-v4: "blacklist"
  name-v6: "blacklist6"
# iptables
iptables -A INPUT -m set --set blacklist src -j DROP
ip6tables -A INPUT -m set --set blacklist6 src -j DROP

Notes:

  • To enable this module the root privileges is required.
  • Please create a set with ipset command first. eg. ipset -N blacklist iphash

How to use:

./configure --enable-ipset
make && make install

Configuration:

# unbound.conf
server:
  ...
  local-zone: "example.com" ipset

ipset:
  name-v4: "blacklist"

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented May 6, 2019

Hi,
This pull request looks real good! Could you spend some words to describe a way that it can be used. In what situation would users use this feature? Or how can the IPset get used (for example)?
From what I read it adds the IP addresses from the *.example.com A and AAAA lookups into the IPset. And that IPset can be used by the Firewall? Is that correct?
Best regards, Wouter

@k9982874
Copy link
Contributor Author

@k9982874 k9982874 commented May 7, 2019

Hi,
This pull request looks real good! Could you spend some words to describe a way that it can be used. In what situation would users use this feature? Or how can the IPset get used (for example)?
From what I read it adds the IP addresses from the *.example.com A and AAAA lookups into the IPset. And that IPset can be used by the Firewall? Is that correct?
Best regards, Wouter

Thank you for your reply.
Yes, This patch could work with firewall to batch block the IPs.
And, I have already updated the description for the PR. Please check and have fun.

@rexy74
Copy link

@rexy74 rexy74 commented May 12, 2019

Hi,
This patch can also be used in the Whitelist of a NAC (Network Access Controler). For example, in our project ALCASAR, we use this behavior with DNSMask. For the next version of ALCASAR, we've decided to replace 'Dnsmasq' with 'Unbound' (due to the better perfs). However, we must keep one instance of 'dnsmasq' (just behind 'unbound') in order to deal with the firewall and its ipset. With this module, we won't need dnsmasq anymore.

@wcawijngaards wcawijngaards merged commit 19c2ece into NLnetLabs:master Jun 18, 2019
1 check passed
wcawijngaards added a commit that referenced this issue Jun 18, 2019
  the ipset that could add the domain's ip to a list easily.
  Needs libmnl, and --enable-ipset and config it, doc/README.ipset.md.
- Fix to omit RRSIGs from addition to the ipset.
@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Jun 18, 2019

Thank you for the PR! The documentation you write, I have put in doc/README.ipset.md, so that other people can reference it.

jedisct1 added a commit to jedisct1/unbound that referenced this issue Jun 29, 2019
* nlnet/master: (22 commits)
  Nicer spelling and layout.
  - For NLnetLabs#45, check that 127.0.0.1 and ::1 are not used in unbound.conf   when do-not-query-localhost is turned on, or at default on,   unbound-checkconf prints a warning if it is found in forward-addr or   stub-addr statements.
  - Fix memleak in unit test, reported from the clang 8.0 static analyzer.
  - Fix python dict reference and double free in config.
  - Merge PR NLnetLabs#6: Python module: support multiple instances - Merge PR NLnetLabs#5: Python module: define constant MODULE_RESTART_NEXT - Merge PR NLnetLabs#4: Python module: assign something useful to the   per-query data store 'qdata' Noted in Changelog.
  - Added documentation to the ipset files (for doxygen output).
  - make depend
  - Fix to make unbound-control with ipset, remove unused variable,   use unsigned type because of comparison, and assign null instead   of compare with it.  Remade lex and yacc output.
  - PR NLnetLabs#28: IPSet module, by Kevin Chou.  Created a module to support   the ipset that could add the domain's ip to a list easily.   Needs libmnl, and --enable-ipset and config it, doc/README.ipset.md. - Fix to omit RRSIGs from addition to the ipset.
  - Fix for NLnetLabs#24: Fix abort due to scan of auth zone masters using old   address from previous scan.
  - Fix NLnetLabs#39: In libunbound, leftover logfile is close()d unpredictably.
  - Master contains version 1.9.3 in development.
  fix segmentation fault
  rollback the code
  bugfix
  performance improvement
  edit config parser to support ipset
  Add support for ipset
  Document how to configure multiple python modules
  Support multiple python module instances
  ...
@dimqua
Copy link

@dimqua dimqua commented Aug 29, 2019

Many thanks for implementing this feature! Do you know the how many local-zone rules can Unbound handle?

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Aug 29, 2019

There is no limit on it. It uses memory to store them. You can have as many as you like. If you have a local-zone without local-data in it, it uses less memory. It then also uses more time to read them in. Consider using the include: "file" directive to organize the lists.

@schm0
Copy link

@schm0 schm0 commented Sep 2, 2019

Thank you for this patch but I can't get this working.
I want to use this on OpenWRT.
So I modified the make file to enable ipset support and installed the libmnl headers.
Compiled fine.
Edited my unbound conf:

server:
  ...
  module-config: "ipset validator iterator"
  ...
  local-zone: "example.com" ipset
  ...
ipset:
  name-v4: "test4"
  name-v6: "test6"

Created the ipsets:

ipset create test4 iphash
ipset create test6 iphash

But nothing is added to the sets.
The log shows:

[1567384109] unbound[25802:0] notice: init module 0: ipset
[1567384109] unbound[25802:0] notice: init module 1: validator
[1567384109] unbound[25802:0] notice: init module 2: iterator
...
[1567384958] unbound[27613:1] debug: ipset[module 0] operate: extstate:module_wait_module event:module_event_moddone
[1567384958] unbound[27613:1] info: ipset operate: query example.com. A IN
[1567384958] unbound[27613:1] debug: mesh_run: ipset module exit state is module_finished

And is it possible to have multiple ipsets?

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Sep 2, 2019

The previous commit should allow you to see in the logs (set verbosity high up for it) when an ipaddress is added to the set.

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Sep 2, 2019

The output should look like this (for me in a test), with verbosity 3 or higher:
debug: ipset: add 93.184.216.34 to test4 for www.example.com.

@mercora
Copy link

@mercora mercora commented Sep 19, 2019

i probably have the same issue as @schm0 although i get the log message as described nothing is appended to the ipset at all. I also found some issues in addition to that.

  1. The confgparser.y does not allow local-zone statements of type ipset in views. It appears as it was just forgotten as i can see code added in that block too. I have a patch for this but its trivial. However. neither using this module globally nor in views works for me.

  2. Second and in my opinion more serve is that it wont work if i have a local-zone set for "." in my view which is of type "refuse", "static" or similar. While this might seem obvious as there is no zone-data i don't know how to use this module to whitelist IP addresses in my firewall.

What i would like to do is having a special view for clients with restricted internet access. i.e. my TV set should not be able to access the internet by default but there are some cases where i want to allow it based on the response of unbound for i.e. "pool.ntp.org" as far as i can see the way this module is implemented does not allow such configuration at all. Its not a problem if you don't want to avoid responding with addresses that are not routable but doing so makes my TV set a lot less noisy with regards to connection attempts (that will fail repeatedly).

  1. The documentation is missing that we need to add this module to the module-config line. If you already have setup another module before you probably know already about this but i was left confused at first. Moreover i recognized something strange. If i omit that line i can see in the log that 3 modules are loaded "subnet", "validator" and "iterator". However, adding the module-config line does not allow me to load the subnet module (whatever it does) and denies startup with this message:
    fatal error: module_conf lists module 'subnet' but that module is not available.
    But this is probably completely unrelated to this module and should be addressed in a separate ticket.

@mercora
Copy link

@mercora mercora commented Sep 20, 2019

Additionally i would like to add that i would have a use-case for multiple ipset lists too which currently is not possible either.

@k9982874
Copy link
Contributor Author

@k9982874 k9982874 commented Sep 20, 2019

@mercora Thank you for your suggestion. Basically, I think to support the multiple ipset is the answer to solve the problem that you facing.
Unfortunately, This patch does not aim for that at the beginning.
To support the multiple ipset need to do a lot of works on the structure level that the reason why I don't implement it.
For me, I just need unbound could add ip address into the ipset that's all.

@mercora
Copy link

@mercora mercora commented Sep 20, 2019

ok,sounds reasonable. Then lets find out why it is not working as is for now :)

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Dec 4, 2019

Hi,
@k9982874 I hope you can see this comment here.

We would like to include the ipset module under the BSD license. It was contributed by you (Kevin Chou) but you did not mention the license. Is it okay to include the ipset module under the 3-clause BSD license that we use for Unbound?

Best regards, Wouter

@k9982874
Copy link
Contributor Author

@k9982874 k9982874 commented Dec 5, 2019

@wcawijngaards No worries. It's ok.

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Dec 5, 2019

Thanks for the quick reply! We try to keep our licensing as clear as possible.
Best regards, Wouter

Borromini pushed a commit to Borromini/packages that referenced this issue Feb 23, 2020
This patch enables ipset support in the unbound-daemon-heavy variant. See [1] for
instructions on how to use it.

Also fix a minor typo in the libunbound-light description.

[1] NLnetLabs/unbound#28

Signed-off-by: Stijn Segers <foss@volatilesystems.org>
Borromini pushed a commit to Borromini/packages that referenced this issue Feb 23, 2020
This patch enables ipset support in the unbound-daemon-heavy variant. See [1] for
instructions on how to use it.

Also fix a minor typo in the libunbound-light description.

[1] NLnetLabs/unbound#28

Signed-off-by: Stijn Segers <foss@volatilesystems.org>
Borromini pushed a commit to Borromini/packages that referenced this issue Feb 23, 2020
This patch enables ipset support in the unbound-daemon-heavy variant. See [1] for
instructions on how to use it.

Also fix a minor typo in the libunbound-light description.

[1] NLnetLabs/unbound#28

Signed-off-by: Stijn Segers <foss@volatilesystems.org>
@paulgear
Copy link

@paulgear paulgear commented Apr 16, 2020

Hi all, are there plans for the ipset support in this module to be extended to allow multiple ipsets, for example to maintain a block list for some domains and an allow list for others?

pffmachado pushed a commit to pffmachado/packages that referenced this issue May 25, 2020
This patch enables ipset support in the unbound-daemon-heavy variant. See [1] for
instructions on how to use it.

Also fix a minor typo in the libunbound-light description.

[1] NLnetLabs/unbound#28

Signed-off-by: Stijn Segers <foss@volatilesystems.org>
farmergreg pushed a commit to farmergreg/packages that referenced this issue Sep 8, 2020
This patch enables ipset support in the unbound-daemon-heavy variant. See [1] for
instructions on how to use it.

Also fix a minor typo in the libunbound-light description.

[1] NLnetLabs/unbound#28

Signed-off-by: Stijn Segers <foss@volatilesystems.org>
farmergreg pushed a commit to farmergreg/packages that referenced this issue Sep 8, 2020
This patch enables ipset support in the unbound-daemon-heavy variant. See [1] for
instructions on how to use it.

Also fix a minor typo in the libunbound-light description.

[1] NLnetLabs/unbound#28

Signed-off-by: Stijn Segers <foss@volatilesystems.org>
@hatemosphere
Copy link

@hatemosphere hatemosphere commented Sep 10, 2020

@wcawijngaards question from a noob - how does the update loop look like here? i mean what triggers queries for the domains specified in local-zone after initial startup query is done? i was trying to answer this question by looking at the code base, but failed miserably.

@wcawijngaards
Copy link
Member

@wcawijngaards wcawijngaards commented Sep 11, 2020

@paulgear No plans at the present for multiple ipsets. We have contributed code.

After the lookup to the internet is complete for the query, the modules are done. At that point the ipset.c:ipset_update() function is called with the just newly looked up query answer. This scans the message for the names listed in the local-zone statements, and then adds them to the ipset in ipset.c:add_to_ipset().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants