dnsdist: implemented query counting #4175

Merged
merged 1 commit into from Aug 23, 2016

Projects

None yet

3 participants

@skoef
Contributor
skoef commented Jul 14, 2016 edited

I like the carbonServer functionality, but want to know more about the queries (or in my specific case: domains) that are served by dnsdist. So I implemented query counting, a per-record counter of the amount of requests for a specific record. This then is included in the data sent to carbon and can be adjusted/filtered by setting a Lua function with setQueryCountFilter()

@rgacogne rgacogne commented on an outdated diff Jul 18, 2016
pdns/dnsdist-lua.cc
g_lua.executeCode(R"(function topQueries(top, labels) top = top or 10; for k,v in ipairs(getTopQueries(top,labels)) do show(string.format("%4d %-40s %4d %4.1f%%",k,v[1],v[2], v[3])) end end)");
+
+ g_lua.writeFunction("clearQueryLog", []() {
+ unsigned int size = g_qlog.records.size();
+ g_qlog.records.clear();
@rgacogne
rgacogne Jul 18, 2016 Member

It looks like this code need to acquire a write lock.

@rgacogne rgacogne commented on an outdated diff Jul 18, 2016
pdns/dnsdist-lua.cc
+ g_lua.writeFunction("getQueryLog", [](boost::optional<unsigned int> optMax) {
+ setLuaNoSideEffect();
+ boost::format fmt("query logging is currently: %s (%d records in buffer)\n");
+ g_outputBuffer = (fmt % (g_qlog.enabled ? "enabled" : "disabled") % g_qlog.records.size()).str();
@rgacogne
rgacogne Jul 18, 2016 Member

And probably a read lock there.

@skoef skoef changed the title from Implemented query logging to dnsdist: implemented query counting Aug 17, 2016
@cmouse cmouse commented on an outdated diff Aug 17, 2016
pdns/dnsdist-carbon.cc
@@ -104,6 +104,19 @@ try
str<<base<<"cache-ttl-too-shorts" << " " << cache->getTTLTooShorts() << " " << now << "\r\n";
}
}
+
+ {
+ WriteLock wl(&g_qcount.queryLock);
+ QueryCountRecords::iterator it;
+ std::string qname;
+ for(it=g_qcount.records.begin(); it!=g_qcount.records.end(); ++it) {
@cmouse
cmouse Aug 17, 2016 Contributor

maybe use for(auto qname: g_qcount.records) here?

@cmouse cmouse commented on an outdated diff Aug 17, 2016
pdns/dnsdist-lua.cc
+ boost::format fmt("%d records cleared from query counter buffer\n");
+ g_outputBuffer = (fmt % size).str();
+ });
+
+ g_lua.writeFunction("getQueryCounters", [](boost::optional<unsigned int> optMax) {
+ setLuaNoSideEffect();
+ ReadLock rl(&g_qcount.queryLock);
+ g_outputBuffer = "query counting is currently: ";
+ g_outputBuffer+= g_qcount.enabled ? "enabled" : "disabled";
+ g_outputBuffer+= (boost::format(" (%d records in buffer)\n") % g_qcount.records.size()).str();
+
+ boost::format fmt("%-3d %s: %d request(s)\n");
+ QueryCountRecords::iterator it;
+ unsigned int max = optMax ? *optMax : 10;
+ unsigned int index{1};
+ for(it = g_qcount.records.begin(); it != g_qcount.records.end(); ++it, ++index) {
@cmouse
cmouse Aug 17, 2016 Contributor

for(it = g_qcount.records.begin(); it != g_qcount.records.end() && index <= max; ++it, ++index);

@cmouse
cmouse Aug 17, 2016 Contributor

or just it = g_qcount.records.begin() + std::min(g_qcount.records.size(), max);

@rgacogne rgacogne commented on an outdated diff Aug 23, 2016
pdns/README-dnsdist.md
@@ -971,6 +971,47 @@ latest version of [PowerDNS
Metronome](https://github.com/ahupowerdns/metronome) comes with attractive
graphs for `dnsdist` by default.
+Query counters
+-------------
+When using `carbonServer`, it is also possible to send per-records statistics of
+the amount of queries by using `setQueryCount(true)`. With query counting enabled,
+`dnsdist` will increase a counter for every unique record or the behaviour you define
+in a custom Lua function by setting `setQueryCountFilter(func)`. This filter can decide
+whether to keep count on a query at all or rewrite the queried record.
@rgacogne
rgacogne Aug 23, 2016 Member

Perhaps clarify that only the counter key is rewritten, not the query. It should make sense of course, but I am worried about someone skimming over the README looking how to rewrite queries.

@rgacogne rgacogne commented on an outdated diff Aug 23, 2016
pdns/README-dnsdist.md
+the amount of queries by using `setQueryCount(true)`. With query counting enabled,
+`dnsdist` will increase a counter for every unique record or the behaviour you define
+in a custom Lua function by setting `setQueryCountFilter(func)`. This filter can decide
+whether to keep count on a query at all or rewrite the queried record.
+An example of a `QueryCountFilter` would be:
+
+```
+function filter(dq)
+ qname = dq.qname:toString()
+
+ -- don't count PTRs at all
+ if(qname:match('in%-addr.arpa$')) then
+ return false, ""
+ end
+
+ -- rewrite these queries
@rgacogne
rgacogne Aug 23, 2016 Member

Same thing here, it might be useful to clarify that the counting key is rewritten, not the query.

@rgacogne rgacogne commented on an outdated diff Aug 23, 2016
pdns/README-dnsdist.md
+ -- rewrite these queries
+ if(qname:match('^www.')) then
+ qname = qname:gsub('^www.', '')
+ end
+
+ -- count queries by default
+ return true, qname
+end
+
+setQueryCountFilter(filter)
+```
+
+Valid return values for `QueryCountFilter` functions are:
+
+* `true`: count the specified query
+* `false`: discard the query
@rgacogne
rgacogne Aug 23, 2016 Member

Perhaps replace discard the query by don't count the query?

@rgacogne rgacogne commented on an outdated diff Aug 23, 2016
pdns/dnsdist.cc
@@ -748,6 +749,22 @@ bool processQuery(LocalStateHolder<NetmaskTree<DynBlock> >& localDynNMGBlock,
g_rings.queryRing.push_back({now,*dq.remote,*dq.qname,dq.len,dq.qtype,*dq.dh});
}
+ if(g_qcount.enabled) {
+ WriteLock wl(&g_qcount.queryLock);
@rgacogne
rgacogne Aug 23, 2016 Member

I don't think you need to acquire a WriteLock until after if (countQuery) { evaluates to true below

@rgacogne rgacogne commented on an outdated diff Aug 23, 2016
pdns/dnsdist.cc
@@ -748,6 +749,22 @@ bool processQuery(LocalStateHolder<NetmaskTree<DynBlock> >& localDynNMGBlock,
g_rings.queryRing.push_back({now,*dq.remote,*dq.qname,dq.len,dq.qtype,*dq.dh});
}
+ if(g_qcount.enabled) {
+ WriteLock wl(&g_qcount.queryLock);
+ string qname = (*dq.qname).toString(".", false);
+ bool countQuery{true};
+ if(g_qcount.filter) {
+ std::tie (countQuery, qname) = g_qcount.filter(dq);
@rgacogne
rgacogne Aug 23, 2016 Member

I think you should acquire the Lua mutex here with something like std::lock_guard<std::mutex> lock(g_luamutex);

@rgacogne rgacogne commented on an outdated diff Aug 23, 2016
pdns/dnsdist.cc
@@ -748,6 +749,22 @@ bool processQuery(LocalStateHolder<NetmaskTree<DynBlock> >& localDynNMGBlock,
g_rings.queryRing.push_back({now,*dq.remote,*dq.qname,dq.len,dq.qtype,*dq.dh});
}
+ if(g_qcount.enabled) {
+ WriteLock wl(&g_qcount.queryLock);
+ string qname = (*dq.qname).toString(".", false);
@rgacogne
rgacogne Aug 23, 2016 Member

Just a nit, but for a better readability we have toStringNoDot() that does toString(".", false)

@rgacogne
Member

LGTM once travis is happy. Thank you!

@rgacogne rgacogne merged commit 55c7d8e into PowerDNS:master Aug 23, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment