Request: complete gettag example for lua scripting in recursor

[CpuID]

The documentation here is relatively complete, but the example script does not utilise gettag at all.

Struggling to find any other examples online also, have trawled through the source code and cannot pinpoint why my gettag calls are not being fired exactly.

Would love to see the example script above augmented to include gettag for completeness, thanks.

[Habbie]

gettag should be fired for every query. Can you show it not being fired for some?

[CpuID]

@Habbie I have it enabled in the following state right now, just to get an idea of the input data before adding logic to it:

function gettag (remote, ednssubnet, vlocal, qname, qtype)
  print("gettag -- remote: "..remote.." - ednssubnet: "..ednssubnet.." - local: "..vlocal.." - qname: "..qname.." - qtype: "..qtype)
  return 0
end

With the above I am not getting any output in the PDNS logs with a gettag -- prefix, while performing at least 12 different queries against different IPs (confirmed via the Protobuf exports being received by an external collector).

To confirm, does dq.variable impact how and where gettag() is fired at all? I have tried with it set as true and false just for completeness, with no difference.

Running PDNS Recursor 4.0.3

[CpuID]

Also to confirm I do get output in my logs for other Lua functions such as prerpz and preresolve, which have similar print() lines in them (not a log verbosity issue).

[Habbie]

Try using pdnslog instead of print. We haven't overridden print so it really just goes to stdout which you might not be capturing.

[CpuID]

@Habbie same issue, changed my gettag, prerpz and preresolve hooks to all use pdnslog(). I get prerpz and preresolve output but nothing from gettag.

Next test: a completely isolated setup, strip out most other lua logic, gettag only to see if it outputs. Getting nothing right now. Running the process in Docker, using pdns_recursor (no other args) as my CMD.

Config (noting this is an isolated development instance, insecure by default):

any-to-tcp=yes
api-logfile=/var/log/pdns.log
api-readonly=yes
dnssec=validate
dnssec-log-bogus=yes
local-address=0.0.0.0 ::1 
logging-facility=5
loglevel=5
lua-config-file=/usr/local/etc/recursor-conf.lua
lua-dns-script=/usr/local/etc/recursor-query.lua
security-poll-suffix=
version-string=test-recursor
webserver-allow-from=127.0.0.1/32 ::1 
webserver-password=nottheactualpassword
webserver-port=8082
webserver=yes
non-local-bind=yes

And my recursor-conf.lua:

-- RPZ Servers
rpzMaster("x.x.x.x", "zone1.rpz", {defpol=Policy.NXDOMAIN, refresh=120})
rpzMaster("x.x.x.x", "zone2.rpz", {defpol=Policy.NXDOMAIN, refresh=120})

-- Protobuf Exports
protobufServer("x.x.x.x:4242", 2, 100, 1, 32, 128)

And my recursor-query.lua:

function gettag (remote, ednssubnet, vlocal, qname, qtype)
  pdnslog("gettag -- remote: "..remote.." - ednssubnet: "..ednssubnet.." - local: "..vlocal.." - qname: "..qname.." - qtype: "..qtype.." - policytags: "..policytags)
  return 0
end

Startup logs:

Nov 20 21:19:48 PowerDNS Recursor 4.0.3 (C) 2001-2016 PowerDNS.COM BV
Nov 20 21:19:48 Using 64-bits mode. Built using gcc 5.3.0 on Oct 23 2016 18:45:46 by root@4c573a7cf4a3.
Nov 20 21:19:48 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Nov 20 21:19:48 Reading random entropy from '/dev/urandom'
Nov 20 21:19:48 If using IPv6, please raise sysctl net.ipv6.route.max_size, currently set to 4096 which is < 16384
Nov 20 21:19:48 NOT using IPv6 for outgoing queries - set 'query-local-address6=::' to enable
Nov 20 21:19:48 Only allowing queries from: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10
Nov 20 21:19:48 Will not send queries to: 127.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 192.168.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10, 0.0.0.0/8, 192.0.0.0/24, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, 240.0.0.0/4, ::/96, ::ffff:0:0/96, 100::/64, 2001:db8::/32, 0.0.0.0, ::
Nov 20 21:19:48 PowerDNS Recursor itself will distribute queries over threads
Nov 20 21:19:48 Inserting rfc 1918 private space zones
Nov 20 21:19:48 Listening for UDP queries on 0.0.0.0:53
Nov 20 21:19:48 Listening for UDP queries on [::1]:53
Nov 20 21:19:48 Enabled TCP data-ready filter for (slight) DoS protection
Nov 20 21:19:48 Listening for TCP queries on 0.0.0.0:53
Nov 20 21:19:48 Listening for TCP queries on [::1]:53
Nov 20 21:19:48 Launching 3 threads
Nov 20 21:19:48 Done priming cache with root hints
Nov 20 21:19:48 Loaded 'lua' script from '/usr/local/etc/recursor-query.lua'
Nov 20 21:19:48 Enabling web server
Nov 20 21:19:48 Listening for HTTP requests on 127.0.0.1:8082
Nov 20 21:19:48 Enabled 'epoll' multiplexer
Nov 20 21:19:48 Done priming cache with root hints
Nov 20 21:19:48 Loaded 'lua' script from '/usr/local/etc/recursor-query.lua'
Nov 20 21:19:48 Done priming cache with root hints
Nov 20 21:19:48 Loaded 'lua' script from '/usr/local/etc/recursor-query.lua'
Nov 20 21:19:49 Refreshed . records
Nov 20 21:19:49 Refreshed . records
Nov 20 21:19:49 Refreshed . records

[rgacogne]

Your Lua code triggers several issues with my 4.0.3:

• Nov 21 10:36:28 Error parsing a query packet qname='powerdns.com' for tag determination, setting tag=0: [string "chunk"]:2: attempt to concatenate a nil value (global 'policytags')
• After removing policytags: Nov 21 10:37:17 Error parsing a query packet qname='powerdns.com' for tag determination, setting tag=0: [string "chunk"]:2: attempt to concatenate a userdata value (local 'qname') (and so on for vlocal, ednssubnet and remote)

Replacing your pdnslog() line by this one works for me:
pdnslog("gettag -- remote: "..remote:toString().." - ednssubnet: "..ednssubnet:toString().." - local: "..vlocal:toString().." - qname: "..qname:toString().." - qtype: "..qtype)

Nov 21 10:38:22 gettag -- remote: 127.0.0.1 - ednssubnet: invalid/0 - local: 127.0.0.1 - qname: powerdns.com. - qtype: 1

[CpuID]

@rgacogne my bad - I had that in there for debugging.

I can confirm that your pdnslog example did work, but I wasn't getting any of the Lua error output that you were, weird. I normally get those types of errors (in the form of STL error for ...) for other Lua functions. The :toString() is helping here, thanks heaps :)

Note I am using LuaJIT here, when compiling PDNS Recursor using ./configure --with-protobuf --with-luajit.

Now that I have a working gettag, would it be possible to get the public examples augmented to add a real world use case covering the various response types etc? I am sure that will be useful to others also.

[Habbie]

but I wasn't getting any of the Lua error output that you were, weird

You need log-common-errors to see them.

[dr3n83]

I would like to renew the example order with gettag. I also miss examples. In my case, I use postresolve with packetcache disabled, to be able to change domain IPs after resolution. But the performance is horrible. I read reports that the way is to migrate to gettag + packetcache but I can't find a reference and examples of how to achieve it.

[omoerbeek]

I agree that a gettag() answer would be nice. But a more general comment:

You can use the packet cache with Lua hook that modify the answers if the answers are modified in the same way always (e.g. independent of time or of the client asking the question).
If that is not true, you can still use the packet cache, but use variable (https://docs.powerdns.com/recursor/lua-scripting/dq.html#DNSQuestion.variable) for answers that are dependent on specifics. That causes the question (and answer) not to be inserted into the packet cache.

[dr3n83]

@omoerbeek Thanks a lot for the quick response.

In a presentation by Ber Hubert entitled "Implementing safe browsing cost-effectively
with DNS", it mentions performance tips and says:

'Performance trick: implement gettag() which determines user/domain status and then relies on the packet cache'

It says this within an apparent context of per-user filtering. Could you help me understand this better? Any tips on which documentation I should follow?

I really wanted to figure out a way to even need to change the response based on rules per user, get the benefit of packetcache. gettag won't allow this?

I currently use dnsdist + recursor. My intercept and change in Lua is in the Recursor.