Add arbitrary tag capability to protobuf output in dnsdist #5396

sethomisc · 2017-06-13T18:19:56Z

Short description

New Lua commands for storing text table information in DNSQuestion and reading the table in DNSResponse. Additionally storing table in DNSProtoBufMessage as tags along with a function to alter the protobuf message from a 'query' to that of a 'response'. These added functions are to allow reporting of Lua based RPZ detection and reporting. Notes, example scripts and config file are located in the directory pdns/zzz-gca-example Tests will be coming soon.

Checklist

I have:

[ X] read the CONTRIBUTING.md document
[ X] compiled and tested this code
[ X] included documentation (including possible behaviour changes)
[ X] documented the code
added or modified regression test(s)
added or modified unit test(s)
[X ]
dnsdist-Lua additions-June_7_2017.pdf

johnhtodd · 2017-06-20T19:22:09Z

A better title for this PR would be "Add arbitrary tag capability to protobuf output in dnsdist"

Habbie · 2017-06-20T19:22:54Z

A better title for this PR would be "Add arbitrary tag capability to protobuf output in dnsdist"

Title updated!

rgacogne

Hello!

Thank you for this pull request, I have written a few comments below. Other than that this pull request lacks documentation for the new methods in pdns/dnsdistdist/README.md and the whole zzz-gca-example directory should probably go (please don't commit .pdf or .abw files).

rgacogne · 2017-06-21T08:54:42Z

pdns/dnsdist-lua.cc

+
+  // --------------------------------------------------------------------------
+  // GCA - Seth Ornstein added lua callable functions - 6/2/2017
+  // DNSQuestion - setTag, setTagArray, getTagMatch, getTagArray


Please remove this kind of comments, it doesn't really help and even make the code less readable.

Comments removed.

rgacogne · 2017-06-21T08:56:08Z

pdns/dnsdist-lua.cc

+
+     g_lua.registerFunction<void(DNSQuestion::*)(vector<pair<string, string>>)>("setTagArray", [](DNSQuestion& dq, const vector<pair<string, string>>&tags) {
+
+      setLuaSideEffect();


setLuaSideEffect() and setLuaNoSideEffect() only make sense for functions called from the command line so they show up in delta().

calls to setLuaSideEffect() and setLuaNoSideEffect() removed.

rgacogne · 2017-06-21T08:57:51Z

pdns/dnsdist-lua2.cc

+
+        message.setQueryTime(0, 0);                             // seconds and microseconds
+
+        message.addRRs(strQueryName);                           // add a RR to the response


A method called setProtobufResponseType() should probably not add a RR, please move that to a separate one.

New function setProtobufResponseRR() added. setProtobufResponseType() does not add a RR.

rgacogne · 2017-06-21T08:58:14Z

pdns/dnsdist-lua2.cc

+
+        message.setType(DNSProtoBufMessage::Response);          // set protobuf type to be response - not query
+
+#ifdef TRASH


I have the feeling this should have been removed before opening this PR :-)

Removed #ifdef TRASH lines.

rgacogne · 2017-06-21T08:59:02Z

pdns/dnsdist-lua2.cc

+     });
+
+                                                                // setProtobufResponseTypeQT - with query time as function parameter
+     g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string&, time_t sec, uint uSec)>("setProtobufResponseTypeTS", [](DNSDistProtoBufMessage& message, const std::string& strQueryName, time_t sec, uint uSec) {


This method could be merge with the previous one using optional time parameters, same remark about adding a RR.

setProtobufResponseType() now can take optional parameters to set time.

rgacogne · 2017-06-21T09:03:31Z

pdns/dnsdist.hh

+// ----------------------------------------------------------------------------
+// count() - return number of tag entries
+// ----------------------------------------------------------------------------
+int count() const


size_t or unsigned int

Replaced with size_t

rgacogne · 2017-06-21T09:09:37Z

pdns/dnsdist.hh

  bool skipCache{false};
  bool ecsOverride;
  bool useECS{true};    
+  QTag qTag;                        // GCA - Seth Ornstein - extra class for tags 5/30/2017


Since we are trying to keep the DNSQuestion object as light as possible for performance reason, would you be willing to consider making that an std::unique_ptr<QTag> so that we don't pay the cost of constructing the object until we really use it? Please also move it up in the DNSQuestion object with the other pointers to make padding more optimal.
By the way, if I'm not mistaken, the QTag object is basically a std::unordered_map<std::string,std::string>, perhaps we could avoid defining a new object and just handle the needful in the Lua bindings?

rgacogne · 2017-06-21T09:12:07Z

pdns/protobuf.cc

+// ----------------------------------------------------------------------------
+
+void DNSProtoBufMessage::addTags(const std::string& strLabel, const std::string& strValue)
+{


I think we should move RecProtoBufMessage::setPolicyTags() and RecProtoBufMessage::getPolicyTags() here instead.

rgacogne · 2017-06-21T09:13:49Z

pdns/protobuf.cc

+        cTemp[2] = 0;
+        cTemp[3] = 1;
+        rr->set_rdata(cTemp, 4);
+      }


This method only adds one RR, it should probably called addRR() instead. should Please don't hardcode the type, class, TTL and content here, those should be parameters passed to this method.

rgacogne · 2017-06-21T09:18:30Z

pdns/dnsdist-lua.cc

+     g_lua.registerFunction<string(DNSQuestion::*)(std::string)>("getTagMatch", [](const DNSQuestion& dq, const std::string& strLabel) {
+
+        std::string strValue = dq.qTag.getMatch(strLabel);
+        return(strValue);


The parentheses are not needed for this kind of return statement.

Parentheses removed for the return.

sethomisc · 2017-06-30T20:27:31Z

Bert and Remi,
Thank you for your constructive comments. I have attempted to do all of them. In the structure DNSQuestion in dnsdist.hh I changed QTag to be a shared_ptr as I had a lot of difficulty in making it a unique_ptr. Please let me know if any of my changes are unclear or need further work.

Thanks,

Seth

rgacogne

Thank you for making these changes! In addition to my individual comments on the code, it might make more sense to merge test_ProtobufTag.py with the existing test_Protobuf.py to avoid code duplication.

Documentation is also lacking for the new methods:

setTag(), setTagArray(), getTagMatch() (getTag()?), getTagArray() for the DNSQuestion object
setTag(), setTagArray(), setProtobufResponseType(), setProtobufResponseRR() (addResponseRR()) for the DNSDistProtoBufMessage one

rgacogne · 2017-07-03T09:22:30Z

pdns/dnsdist-lua.cc


+    g_lua.registerFunction<void(DNSQuestion::*)(std::string, std::string)>("setTag", [](DNSQuestion& dq, const std::string& strLabel, const std::string& strValue) {
+
+      if(dq.qTag == NULL)


It would be a bit nicer to use nullptr instead of NULL, as the former can't be confused with an integer.

rgacogne · 2017-07-03T09:23:24Z

pdns/dnsdist-lua.cc

+    g_lua.registerFunction<void(DNSQuestion::*)(std::string, std::string)>("setTag", [](DNSQuestion& dq, const std::string& strLabel, const std::string& strValue) {
+
+      if(dq.qTag == NULL)
+        {


We usually place the { following an if or else on the same line.

rgacogne · 2017-07-03T09:24:11Z

pdns/dnsdist-lua.cc

+
+      if(dq.qTag == NULL)
+        {
+         dq.qTag = std::shared_ptr<QTag>(new QTag);


Replacing std::shared_ptr<QTag>(new QTag) by std::make_shared<QTag>() would save an allocation and be a bit more easy to read.

rgacogne · 2017-07-03T09:26:19Z

pdns/dnsdist-lua.cc

+      if(dq.qTag == NULL)
+        {
+         dq.qTag = std::shared_ptr<QTag>(new QTag);
+        }


We usually don't indent the closing }

rgacogne · 2017-07-03T09:30:00Z

pdns/dnsdist-lua2.cc

+
+     });
+
+     g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string&, uint uType, uint uClass, uint uTTL, vector<pair<int, int>> )>("setProtobufResponseRR", [](DNSDistProtoBufMessage& message,


Please use uint16_t for type and class, uint32_t for ttl. A simple std:string might be more suited for blobData, it would prevent to have to allocate an array and copy the content to it?

rgacogne · 2017-07-03T09:33:27Z

pdns/dnsdist.hh

+bool bStatus = true;
+
+   tagData.insert( {strLabel, strValue});
+  return(bStatus);


It looks like it always return true, so perhaps the return value might be omitted?

rgacogne · 2017-07-03T09:33:57Z

pdns/dnsdist.hh

+    std::unordered_map<std::string, std::string>::const_iterator got =tagData.find (strLabel);
+    if(got == tagData.end())
+      {
+       return("");


We usually don't use parenthesis for return statements.

rgacogne · 2017-07-03T09:34:42Z

pdns/protobuf.cc

+    return;
+
+  PBDNSMessage_DNSResponse_DNSRR* rr = response->add_rrs();
+      if (rr) {


Indentation looks off.

rgacogne · 2017-07-03T09:36:21Z

pdns/protobuf.cc

+#endif /* HAVE_PROTOBUF */
+}
+
+void DNSProtoBufMessage::addRR(const std::string& strName, uint32_t uType, uint32_t uClass, uint32_t uTTL, const uint8_t *ptrBlob, size_t uBlobLen)


uType and uClass should be uint16_t, ptrBlob and uBlobLen can be merged into a std::string.

rgacogne · 2017-07-03T09:38:52Z

pdns/dnsdist-lua.cc

+     });
+
+
+     g_lua.registerFunction<string(DNSQuestion::*)(std::string)>("getTagMatch", [](const DNSQuestion& dq, const std::string& strLabel) {


Perhaps getTag() would be more logical, to match setTag()?

sethomisc · 2017-07-06T15:55:34Z

Remi,
Thanks for your very constructive comments, I've gone through the list and corrected them.
I'm not clear on how I should document my changes, any direction on that would also be appreciated.
I've attached a very brief description of the new lua commands with examples below.
Seth

ReadMe-7-5-2017.txt

pieterlexis · 2017-07-07T08:44:39Z

@sethomisc I have created new documentation for the upcoming 4.1 release (#5481), you can PR documentation there. Or if you want, I can add your readme contents to them.

rgacogne · 2017-07-07T15:36:51Z

pdns/dnsdist-lua2.cc

+     });
+
+    g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(boost::optional <time_t> sec, boost::optional <uint> uSec)>("setProtobufResponseType",
+                                        [](DNSDistProtoBufMessage& message, boost::optional <time_t> sec, boost::optional <uint> uSec) {


uSec should be a uint32_t here to be consistent with DNSProtoBufMessage::setQueryTime(). Be careful that you need to change it twice due to the LuaWrapper syntax.

rgacogne

I realize I haven't said it before but I really like the features that this pull request implements, so thank you again. It's merge-able as it is, but if you don't mind fixing the few remaining nits it would be almost perfect. Regarding the documentation, you could just drop a few lines in pdns/README-dnsdist.md to document the member functions that you are adding. Look for DNSDistProtoBufMessage related and DNSQuestion related.
If you can't figure it out just let me know, I can fix the PR myself, or even merge it and fix the remaining issues afterward, but I think it would be more rewarding to you if you could do it yourself :-)

rgacogne · 2017-07-07T15:37:49Z

pdns/dnsdist-lua2.cc

+     });
+
+    g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string& strQueryName, uint uType, uint uClass, uint uTTL, const std::string& strBlob)>("addResponseRR", [](DNSDistProtoBufMessage& message,
+                                                            const std::string& strQueryName, uint16_t uType, uint uClass, uint32_t uTTL, const std::string& strBlob) {


uType should be uint16_t on the first line too. uClass should be uint16_t on both lines

rgacogne · 2017-07-07T15:38:20Z

pdns/dnsdist.hh

+std::string getEntry(size_t iEntry) const
+{
+std::string strEntry;
+size_t iCounter = 0;


small indentation issue

rgacogne · 2017-07-07T15:38:43Z

pdns/protobuf.cc

+    return;
+
+  PBDNSMessage_DNSResponse_DNSRR* rr = response->add_rrs();
+      if (rr) {


rgacogne · 2017-07-07T15:39:43Z

pdns/protobuf.cc

+#endif /* HAVE_PROTOBUF */
+}
+
+void DNSProtoBufMessage::addRR(const std::string& strName, uint32_t uType, uint32_t uClass, uint32_t uTTL, const std::string& strBlob)


uType and uClass should be uint16_t.

rgacogne · 2017-07-07T15:40:08Z

regression-tests.dnsdist/test_Protobuf.py

-        requestor = newCA(dq.remoteaddr:toString())
-        if requestor:isIPv4() then
-          requestor:truncate(24)
+    luasmn = newSuffixMatchNode()				


You have inserted a lot of trailing spaces, could you fix that?

rgacogne · 2017-07-07T15:43:56Z

regression-tests.dnsdist/test_Protobuf.py

+		self.assertEquals(msg.type, dnsmessage_pb2.PBDNSMessage.DNSQueryType)		# testProtobuf()
+	else:
+		self.assertEquals(msg.type, dnsmessage_pb2.PBDNSMessage.DNSResponseType)	# testLuaProtobuf()
+


That blocks looks very weird because it's effectively making checkProtobufQuery() accept responses. Perhaps we should call checkProtobufResponse() when the query has been turned into a response instead?

rgacogne

A few nits, but nothing that can't be fixed at a later time.

rgacogne · 2017-07-17T08:58:05Z

pdns/dnsdist.hh

+{
+}
+
+void add(std::string strLabel, std::string strValue)


strLabel and strValue could be const references instead

rgacogne · 2017-07-17T08:58:39Z

pdns/dnsdist.hh

+  std::unordered_map<std::string, std::string>tagData;
+
+private:
+  const char *strSep = "\t";


Could be static

rgacogne · 2017-07-17T08:59:14Z

pdns/protobuf.cc

+#endif /* HAVE_PROTOBUF */
+}
+
+void DNSProtoBufMessage::addRR(const std::string& strName, uint16_t uType, uint16_t uClass, uint32_t uTTL, const std::string& strBlob)


strName should probably be a const DNSName& instead of a string

rgacogne · 2017-07-17T08:59:47Z

pdns/protobuf.cc

+    rr->set_type(uType);
+    rr->set_class_(uClass);
+    rr->set_ttl(uTTL);
+    rr->set_rdata((const uint8_t *) strBlob.c_str(), strBlob.size());


If I'm not mistaken the C-style cast to const uint8_t * is not needed.

sethomisc · 2017-07-28T23:18:12Z

Remi,

changed the strLabel and strValue to be const references in Qtag::add() located in dnsdist.hh

changed const char *strSep =\t"; to static constexpr char const *strSep = "\t"; in dnsdist.hh

removed the (const uint8 t *) cast in DNSProBufMessage:addRR() in protobuf.cc

unsure if I should change strName from a string to a const DNSName& as I intended to call the function with just the DNS string like dr.qname:toString(), please advise me Remi.

Again thank you for all your input Remi, Seth

sethomisc · 2017-08-01T22:59:17Z

Remi,
I have rebased dnsdist-mod2.
Please let me know if there are any problems.
Seth

rgacogne · 2017-08-02T07:59:20Z

Hi Seth! Thanks, there is an issue with your last commit that completely removes the README-dnsdist.md file, causing the test to file. Could you just drop that last commit and amend e75a021 ("Fixed requests from Remi July 7") to remove any alteration to that file?

Commented line around new code segments has key words 'GCA' and 'Seth' for text searching. Commands don't have XXX suffix anymore. Timestamp for the setProtobufResponseType() function is supplied by lua now and not done in C++. Current commands are: setTag, getTagArray, setTagArray, setProtobufResponseType

…lanced'.

…ed example script in pdns/zzz-gca-example directory

…est_ProtobufTag.py script to execute it in pdns/zzz-gca-example/test-protobuf-tag.sh

It's OK for the Lua binding to accept a string, but the internal functions should really use a `DNSName`.

nesv · 2017-08-02T15:56:23Z

@rgacogne Along with your requested amendment to e75a021, I have also removed some other files that were introduced by that commit, that probably shouldn't have been there:

pdns/dnsdistdist/cdb.so
contrib/dnsmessage_pb2.py{,c}

rgacogne · 2017-08-03T08:53:31Z

Merged, thank you again!

Habbie changed the title ~~Dnsdist mod2~~ Add arbitrary tag capability to protobuf output in dnsdist Jun 20, 2017

johnhtodd mentioned this pull request Jun 20, 2017

WIP: Add dnstap-compatible protobuf support to dnsdist. #5201

Closed

6 tasks

rgacogne requested changes Jun 21, 2017

View reviewed changes

rgacogne requested changes Jul 3, 2017

View reviewed changes

rgacogne reviewed Jul 7, 2017

View reviewed changes

rgacogne approved these changes Jul 17, 2017

View reviewed changes

rgacogne added dnsdist enhancement labels Jul 29, 2017

rgacogne added this to the dnsdist-1.2.0 milestone Jul 29, 2017

sethomisc and others added 11 commits August 2, 2017 11:06

Added some more Lua commands to dnsdist to make the commands more 'ba…

99fce2d

…lanced'.

documentation of new lua commands, stripped down dnsdist.conf, improv…

273193c

…ed example script in pdns/zzz-gca-example directory

completed test for mods to dnsdist in pdns/regression-tests.dnsdist/t…

8a87971

…est_ProtobufTag.py script to execute it in pdns/zzz-gca-example/test-protobuf-tag.sh

Changes requested by Bert Hubert and Remi Gacogne of dnsdist

e6956c9

addressed fixes requested by Remi July 3rd

741ebe0

Fixed requests from Remi July 7

29cd61c

fixed 3 of the "nits" flagged by Remi

c959e01

dnsdist: Fix indentation, move tags check out of the generic ones

0096ada

dnsdist: Pass the qname as a DNSName to DNSProtoBufMessage::addRR()

ff09c37

It's OK for the Lua binding to accept a string, but the internal functions should really use a `DNSName`.

dnsdist: Fix indentation

5b8255b

rgacogne approved these changes Aug 3, 2017

View reviewed changes

rgacogne merged commit 5ab772f into PowerDNS:master Aug 3, 2017

rgacogne mentioned this pull request Aug 3, 2017

dnsdist: Document the tag features #5577

Merged

6 tasks


		g_lua.registerFunction<void(DNSQuestion::*)(vector<pair<string, string>>)>("setTagArray", [](DNSQuestion& dq, const vector<pair<string, string>>&tags) {

		setLuaSideEffect();


		message.setQueryTime(0, 0); // seconds and microseconds

		message.addRRs(strQueryName); // add a RR to the response


		message.setType(DNSProtoBufMessage::Response); // set protobuf type to be response - not query

		#ifdef TRASH


		g_lua.registerFunction<void(DNSQuestion::*)(std::string, std::string)>("setTag", [](DNSQuestion& dq, const std::string& strLabel, const std::string& strValue) {

		if(dq.qTag == NULL)


		});

		g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string&, uint uType, uint uClass, uint uTTL, vector<pair<int, int>> )>("setProtobufResponseRR", [](DNSDistProtoBufMessage& message,

		});


		g_lua.registerFunction<string(DNSQuestion::*)(std::string)>("getTagMatch", [](const DNSQuestion& dq, const std::string& strLabel) {

Add arbitrary tag capability to protobuf output in dnsdist #5396

Add arbitrary tag capability to protobuf output in dnsdist #5396

Uh oh!

Conversation

sethomisc commented Jun 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Short description

Checklist

Uh oh!

johnhtodd commented Jun 20, 2017

Uh oh!

Habbie commented Jun 20, 2017

Uh oh!

rgacogne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sethomisc commented Jun 30, 2017

Uh oh!

rgacogne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sethomisc commented Jul 6, 2017

sethomisc commented Jun 13, 2017 •

edited

Loading