Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dnsdist: Added XDP filter to perform bit TC and block #10498

Closed
wants to merge 3 commits into from
Closed

dnsdist: Added XDP filter to perform bit TC and block #10498

wants to merge 3 commits into from

Conversation

MiniPierre
Copy link

Short description

According to this issue #10451, this pull request adds two XDP filters, one for sending TC bit when an IP/QName is blocked and another one for blocking the IP/QName.

Checklist

I have:

  • read the CONTRIBUTING.md document
  • compiled this code
  • tested this code
  • included documentation (including possible behaviour changes)
  • documented the code
  • added or modified regression test(s)
  • added or modified unit test(s)

@rgacogne
Copy link
Member

Thanks a lot, this is awesome! I'll review the code as soon as possible and think about how to best integrate this in dnsdist.

@MiniPierre
Copy link
Author

Just to share some thoughts about XDP we had during filter development: using such filter would make you loose all information about dropped packets for tools using libpcap, as XDP run first.
As it is not dnsdist job to solve this, I am looking at a way of duplicating packets to a virtual interface, but as two XDP filters cannot run in parallel (first running, last served), things are getting quite touchy

@rgacogne
Copy link
Member

Right, I believe this is expected when using XDP indeed, and we don't need to fix it for the dnsdist case.

@rgacogne rgacogne self-requested a review June 21, 2021 12:22
Copy link
Member

@rgacogne rgacogne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is very promising, thanks again! How did you test this (load the XDP program, populate/update the maps, ...)?

pdns/xdp-filter-block.ebpf.src Outdated Show resolved Hide resolved
return 0;
}
c->pos += 1;
if (qname_byte == 0 || (length == i) * qname_byte > 63 ) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might get a null byte in the middle of a valid label, we need to use the label length to know where the label ends instead of looking for a null byte. In fact since bpf_probe_read_kernel can read several bytes at once we might want to read the whole label in one go then iterate over the bytes one by one to do the lower casing?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not sure if it could be possible to face such situation. Your idea seems to be promising, I will have a look at it as soon as possible

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading a dynamic size with bpf_probe_read_kernel was raising errors, so I changed the null byte check to occur only when size should be read

pdns/xdp-filter-block.ebpf.src Outdated Show resolved Hide resolved
pdns/xdp-filter-tc.ebpf.src Outdated Show resolved Hide resolved
@MiniPierre
Copy link
Author

I solved the different points, I m going to push modifications soon, I will also upload my Python test script so you have a look. Should be done before 6:00 PM UTC+2

@rgacogne
Copy link
Member

Nice, thanks!

@MiniPierre
Copy link
Author

Changes are pushed, you will find the xdp.py test script in the test directory, just copy the filter in this directory before

@rgacogne
Copy link
Member

That looks great, thanks a lot! Now I need to test this a bit, figure out what the right thing to do with regards to the copyright is, and how to integrate that nicely into dnsdist :-)

@rgacogne
Copy link
Member

I included that work in #10883 which added a bit of infrastructure to allow dnsdist to communicate with an external XDP program via pinned maps, so I will now close this PR. Thanks a lot!

@dqos
Copy link

dqos commented Jan 31, 2022

@MiniPierre @rgacogne nice job with this PR. This feature is essential to fight of DNS-attacks. I have been looking through the source code but I couldn't find anything about Truncate inside BPF DynBlocks. Is this correct?

Currently you can only drop traffic using XDP:
addBPFFilterDynBlocks(exceedQRate(10, 1), dbpf, 60)

Somewhere in the latest 1.7.0 code I have seen commits about custom XDP programs and the important Truncate. Are the docs not up to date or am I missing something? Can we do dynamic truncate? 😄

@MiniPierre
Copy link
Author

Hi, XDP is not currently added to dnsdist, as merging everything would take quite a long time. For now, you can only use external XDP program that will read pinned dnsdist BPF maps (a new feature of dnsdist 1.7.0) and perform actions based on this. What you showed does not actually use XDP if not used in coordination with external program, but dnsdist BPF socket filter, which is way slower.

Dnsdist will add IP and Qnames matching DynBlocks rules in BPF maps for the configured time. Then, the external XDP program will compare every packet source IP and queried QName with these maps to determine if any action should be performed.

You will find an example of XDP program in contrib/xdp.py, which you can use to perform TC or DROP actions using XDP. If needed, I can post a sample of my dnsdist configuration tomorrow to show you how to connect dnsdist and the external program

@dqos
Copy link

dqos commented Jan 31, 2022

@MiniPierre thanks that makes it clear. I would like to see how you have done it with dnsdist and the external XDP program.

I have my XDP program which can force DNS queries to truncate to TCP, but I only want to enable this as soon as a large attack is incoming.

Isn't dnsdist using XDP with their eBPF implementation?

@MiniPierre
Copy link
Author

MiniPierre commented Feb 1, 2022

Isn't dnsdist using XDP with their eBPF implementation?

As said, dnsdist is using a BPF_PROG_TYPE_SOCKET_FILTER BPF program, which perform actions quite high in the network stack (most of packet decoding has already been done), this kind of program cannot modify the incoming packet, thus it is unable to perform TC operation. BPF_PROG_TYPE_XDP programs are executed very early, as soon as the packet reach the network interface, it is not even seen by iptables and you have to do decode the different parts to make decisions, but it can perform way more actions on the packet, including modifying it and send it back.

I have my XDP program which can force DNS queries to truncate to TCP, but I only want to enable this as soon as a large attack is incoming.

Thanks to @rgacogne work, you now can read dnsdist BPF maps from your XDP program, which means you have access to the IP and QNames blocked by your DynBlocks. Here's what you can do to test this behaviour:

  • Create a BPF filesystem dedicated to dnsdist, where you will store your BPF maps:
    mkdir /var/lib/dnsdist && mount -t bpf bpffs /var/lib/dnsdist
    In fact, it could be possible to create them in the default /sys/fs/bpf filesystem but I couldn't manage to make it work
  • Configure dnsdist to create a BPF filter with pinned maps:
bpf = newBPFFilter({maxItems=1048576, pinnedPath="/var/lib/dnsdist/v4filter"}, {maxItems=1048576, pinnedPath="/var/lib/dnsdist/v6filter"}, {maxItems=1048576, pinnedPath="/var/lib/dnsdist/qnamefilter"}, true)
setDefaultBPFFilter(bpf)

The true means that you are using an external BPF program, thus the default dnsdist BPF filter will not be used

  • Create DynBlocks rules:
local dbr = dynBlockRulesGroup()
dbr:setQueryRate(10, 1, "Exceeded query rate", 60, DNSAction.Truncate)

function maintenance()
  dbr:apply()
end

The addBPFFilterDynBlocks method is quite outdated now, with dynBlockRulesGroup you can set which action to perform

  • Restart dnsdist and launch the XDP program. You should now be able to perform TC on UDP packets for IP flagged by the dynBlockRulesGroup

In a first time I'd recommend to use the contrib/xdp.py program to test the behaviour before moving to create your own program, as you need to use specific structures.

If you are using Debian bullseye distro and launch dnsdist with a non-root user, you should also do sysctl -w kernel.unprivileged_bpf_disabled=0 to allow dnsdist to write in the BPF maps. I known there are capabilities to avoid such thing but I couldn't make it work

@dqos
Copy link

dqos commented Feb 1, 2022

@MiniPierre thank you! I will test it out this week and report back.

@dqos
Copy link

dqos commented Feb 2, 2022

@MiniPierre Okay I got the following running. Note that the location to /var/lib/dnsdist/ must be updated in xdp-filter.ebpf.src otherwise it crashes. But as soon as xdp.py is running dnsdist is not responding to queries:

Blocking ('1.2.3.4', 1)
Blocking ('2001:db8::1', 2)
Blocking ('localhost', 'A', 1)
Filter is ready

When I disabled dnsdist it's responding again. Also I noticed in the web interface that dynBlocks rules are always ending up in 'Dyn blocked netmask' instead of 'Kernel-based dyn blocked netmask'.

I am also exploring a different solution; fetch amount of queries from dnsdist through the console using any language, then according to thresholds set my XDP truncate program (or worst case drop). Remember that most DNS-attacks are spoofed, coming from unique IPs and random hostnames. So filtering for DDoS based on qname/ip is not efficient.

PS: This all is not working on a VM with virtio as network driver, after changing it to E1000 it works.

@MiniPierre
Copy link
Author

I forgot to specify few things that you should modify in the XDP program to make it work, that might cause you some issues:

  • Remove the values inside blocked_ipv4, blocked_ipv6 and blocked_qnames variables in the xdp.py program. Now that you can use dnsdist bpf maps, they are of no use
  • Change the path value in the map declaration in xdp-filter.ebpf.src to point to dnsdist pinned BPF maps:
BPF_TABLE_PINNED("hash", uint32_t, struct map_value, v4filter, 1024, "/var/lib/dnsdist/v4filter");
BPF_TABLE_PINNED("hash", struct in6_addr, struct map_value, v6filter, 1024, "//var/lib/dnsdist/v6filter");
BPF_TABLE_PINNED("hash", struct dns_qname, struct map_value, qnamefilter, 1024, "/var/lib/dnsdist/qnamefilter");

For the web interface, I could not tell you if it's ok or not as I'm not using it. I guess the fact that it shows this is because the dynBlocks do not make the difference, they are just inserting the configured actions in the BPF map and do not know what you do with it

Your solution could be interesting, but it would also mean that every legitimate queries will be retried in TCP, which could cause massive issues

I've always had issues with virtio driver, dunno why it exists x)

@dqos
Copy link

dqos commented Feb 2, 2022

@MiniPierre

I got the XDP part working and I use the blocked_ipv4 etc for testing. I already adjusted the the paths to the correct files. I think there is something wrong with the maps:

[root@test-xdp-e1000 ~]# cat /var/lib/dnsdist/v4filter 
cat: /var/lib/dnsdist/v4filter: Input/output error

The files can't be read and are 0 bytes (tested with one v4 blocked). When something is blocked, I expected these files to be filled right?

Regarding virtio; default virtio devices do not have enough TX rings, so you either have to increase it or use E1000 driver. I got it working with my XDP program by setting it using XDPGENERIC: https://man7.org/linux/man-pages/man8/ip-link.8.html but I think it emulated XDP support by the kernel, so no real XDP benefits here.

@MiniPierre
Copy link
Author

That's because you can't read BPF pinned maps like this, they are not regular files.
To inspect them, you need to install bpftool. Then, you can list your maps by using bpftool map list. Once you have identified the correct map, you can show its content by using bpftool map dump id your_map_ip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants