Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arbitrary tag capability to protobuf output in dnsdist #5396

Merged
merged 11 commits into from Aug 3, 2017

Conversation

sethomisc
Copy link
Contributor

@sethomisc sethomisc commented Jun 13, 2017

Short description

New Lua commands for storing text table information in DNSQuestion and reading the table in DNSResponse. Additionally storing table in DNSProtoBufMessage as tags along with a function to alter the protobuf message from a 'query' to that of a 'response'. These added functions are to allow reporting of Lua based RPZ detection and reporting. Notes, example scripts and config file are located in the directory pdns/zzz-gca-example Tests will be coming soon.

Checklist

I have:

  • [ X] read the CONTRIBUTING.md document
  • [ X] compiled and tested this code
  • [ X] included documentation (including possible behaviour changes)
  • [ X] documented the code
  • added or modified regression test(s)
  • added or modified unit test(s)
  • [X ]
    dnsdist-Lua additions-June_7_2017.pdf

@johnhtodd
Copy link

A better title for this PR would be "Add arbitrary tag capability to protobuf output in dnsdist"

@Habbie Habbie changed the title Dnsdist mod2 Add arbitrary tag capability to protobuf output in dnsdist Jun 20, 2017
@Habbie
Copy link
Member

Habbie commented Jun 20, 2017

A better title for this PR would be "Add arbitrary tag capability to protobuf output in dnsdist"

Title updated!

Copy link
Member

@rgacogne rgacogne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello!

Thank you for this pull request, I have written a few comments below. Other than that this pull request lacks documentation for the new methods in pdns/dnsdistdist/README.md and the whole zzz-gca-example directory should probably go (please don't commit .pdf or .abw files).


// --------------------------------------------------------------------------
// GCA - Seth Ornstein added lua callable functions - 6/2/2017
// DNSQuestion - setTag, setTagArray, getTagMatch, getTagArray
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this kind of comments, it doesn't really help and even make the code less readable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments removed.


g_lua.registerFunction<void(DNSQuestion::*)(vector<pair<string, string>>)>("setTagArray", [](DNSQuestion& dq, const vector<pair<string, string>>&tags) {

setLuaSideEffect();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setLuaSideEffect() and setLuaNoSideEffect() only make sense for functions called from the command line so they show up in delta().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calls to setLuaSideEffect() and setLuaNoSideEffect() removed.


message.setQueryTime(0, 0); // seconds and microseconds

message.addRRs(strQueryName); // add a RR to the response
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A method called setProtobufResponseType() should probably not add a RR, please move that to a separate one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New function setProtobufResponseRR() added. setProtobufResponseType() does not add a RR.


message.setType(DNSProtoBufMessage::Response); // set protobuf type to be response - not query

#ifdef TRASH
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the feeling this should have been removed before opening this PR :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed #ifdef TRASH lines.

});

// setProtobufResponseTypeQT - with query time as function parameter
g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string&, time_t sec, uint uSec)>("setProtobufResponseTypeTS", [](DNSDistProtoBufMessage& message, const std::string& strQueryName, time_t sec, uint uSec) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method could be merge with the previous one using optional time parameters, same remark about adding a RR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setProtobufResponseType() now can take optional parameters to set time.

pdns/dnsdist.hh Outdated
// ----------------------------------------------------------------------------
// count() - return number of tag entries
// ----------------------------------------------------------------------------
int count() const
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

size_t or unsigned int

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced with size_t

pdns/dnsdist.hh Outdated
@@ -71,6 +196,7 @@ struct DNSQuestion
bool skipCache{false};
bool ecsOverride;
bool useECS{true};
QTag qTag; // GCA - Seth Ornstein - extra class for tags 5/30/2017
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are trying to keep the DNSQuestion object as light as possible for performance reason, would you be willing to consider making that an std::unique_ptr<QTag> so that we don't pay the cost of constructing the object until we really use it? Please also move it up in the DNSQuestion object with the other pointers to make padding more optimal.
By the way, if I'm not mistaken, the QTag object is basically a std::unordered_map<std::string,std::string>, perhaps we could avoid defining a new object and just handle the needful in the Lua bindings?

pdns/protobuf.cc Outdated
// ----------------------------------------------------------------------------

void DNSProtoBufMessage::addTags(const std::string& strLabel, const std::string& strValue)
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should move RecProtoBufMessage::setPolicyTags() and RecProtoBufMessage::getPolicyTags() here instead.

pdns/protobuf.cc Outdated
cTemp[2] = 0;
cTemp[3] = 1;
rr->set_rdata(cTemp, 4);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method only adds one RR, it should probably called addRR() instead. should Please don't hardcode the type, class, TTL and content here, those should be parameters passed to this method.

g_lua.registerFunction<string(DNSQuestion::*)(std::string)>("getTagMatch", [](const DNSQuestion& dq, const std::string& strLabel) {

std::string strValue = dq.qTag.getMatch(strLabel);
return(strValue);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parentheses are not needed for this kind of return statement.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parentheses removed for the return.

@sethomisc
Copy link
Contributor Author

Bert and Remi,
Thank you for your constructive comments. I have attempted to do all of them. In the structure DNSQuestion in dnsdist.hh I changed QTag to be a shared_ptr as I had a lot of difficulty in making it a unique_ptr. Please let me know if any of my changes are unclear or need further work.

Thanks,

Seth

Copy link
Member

@rgacogne rgacogne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for making these changes! In addition to my individual comments on the code, it might make more sense to merge test_ProtobufTag.py with the existing test_Protobuf.py to avoid code duplication.

Documentation is also lacking for the new methods:

  • setTag(), setTagArray(), getTagMatch() (getTag()?), getTagArray() for the DNSQuestion object
  • setTag(), setTagArray(), setProtobufResponseType(), setProtobufResponseRR() (addResponseRR()) for the DNSDistProtoBufMessage one

@@ -1572,6 +1572,57 @@ vector<std::function<void(void)>> setupLua(bool client, const std::string& confi
}
});

g_lua.registerFunction<void(DNSQuestion::*)(std::string, std::string)>("setTag", [](DNSQuestion& dq, const std::string& strLabel, const std::string& strValue) {

if(dq.qTag == NULL)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be a bit nicer to use nullptr instead of NULL, as the former can't be confused with an integer.

g_lua.registerFunction<void(DNSQuestion::*)(std::string, std::string)>("setTag", [](DNSQuestion& dq, const std::string& strLabel, const std::string& strValue) {

if(dq.qTag == NULL)
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually place the { following an if or else on the same line.


if(dq.qTag == NULL)
{
dq.qTag = std::shared_ptr<QTag>(new QTag);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replacing std::shared_ptr<QTag>(new QTag) by std::make_shared<QTag>() would save an allocation and be a bit more easy to read.

if(dq.qTag == NULL)
{
dq.qTag = std::shared_ptr<QTag>(new QTag);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually don't indent the closing }


});

g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string&, uint uType, uint uClass, uint uTTL, vector<pair<int, int>> )>("setProtobufResponseRR", [](DNSDistProtoBufMessage& message,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use uint16_t for type and class, uint32_t for ttl. A simple std:string might be more suited for blobData, it would prevent to have to allocate an array and copy the content to it?

pdns/dnsdist.hh Outdated
bool bStatus = true;

tagData.insert( {strLabel, strValue});
return(bStatus);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it always return true, so perhaps the return value might be omitted?

pdns/dnsdist.hh Outdated
std::unordered_map<std::string, std::string>::const_iterator got =tagData.find (strLabel);
if(got == tagData.end())
{
return("");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually don't use parenthesis for return statements.

pdns/protobuf.cc Outdated
return;

PBDNSMessage_DNSResponse_DNSRR* rr = response->add_rrs();
if (rr) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation looks off.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still true

pdns/protobuf.cc Outdated
#endif /* HAVE_PROTOBUF */
}

void DNSProtoBufMessage::addRR(const std::string& strName, uint32_t uType, uint32_t uClass, uint32_t uTTL, const uint8_t *ptrBlob, size_t uBlobLen)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uType and uClass should be uint16_t, ptrBlob and uBlobLen can be merged into a std::string.

});


g_lua.registerFunction<string(DNSQuestion::*)(std::string)>("getTagMatch", [](const DNSQuestion& dq, const std::string& strLabel) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps getTag() would be more logical, to match setTag()?

@sethomisc
Copy link
Contributor Author

Remi,
Thanks for your very constructive comments, I've gone through the list and corrected them.
I'm not clear on how I should document my changes, any direction on that would also be appreciated.
I've attached a very brief description of the new lua commands with examples below.
Seth

ReadMe-7-5-2017.txt

@pieterlexis
Copy link
Contributor

@sethomisc I have created new documentation for the upcoming 4.1 release (#5481), you can PR documentation there. Or if you want, I can add your readme contents to them.

});

g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(boost::optional <time_t> sec, boost::optional <uint> uSec)>("setProtobufResponseType",
[](DNSDistProtoBufMessage& message, boost::optional <time_t> sec, boost::optional <uint> uSec) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uSec should be a uint32_t here to be consistent with DNSProtoBufMessage::setQueryTime(). Be careful that you need to change it twice due to the LuaWrapper syntax.

Copy link
Member

@rgacogne rgacogne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize I haven't said it before but I really like the features that this pull request implements, so thank you again. It's merge-able as it is, but if you don't mind fixing the few remaining nits it would be almost perfect. Regarding the documentation, you could just drop a few lines in pdns/README-dnsdist.md to document the member functions that you are adding. Look for DNSDistProtoBufMessage related and DNSQuestion related.
If you can't figure it out just let me know, I can fix the PR myself, or even merge it and fix the remaining issues afterward, but I think it would be more rewarding to you if you could do it yourself :-)

});

g_lua.registerFunction<void(DNSDistProtoBufMessage::*)(const std::string& strQueryName, uint uType, uint uClass, uint uTTL, const std::string& strBlob)>("addResponseRR", [](DNSDistProtoBufMessage& message,
const std::string& strQueryName, uint16_t uType, uint uClass, uint32_t uTTL, const std::string& strBlob) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uType should be uint16_t on the first line too. uClass should be uint16_t on both lines

pdns/dnsdist.hh Outdated
std::string getEntry(size_t iEntry) const
{
std::string strEntry;
size_t iCounter = 0;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small indentation issue

pdns/protobuf.cc Outdated
return;

PBDNSMessage_DNSResponse_DNSRR* rr = response->add_rrs();
if (rr) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still true

pdns/protobuf.cc Outdated
#endif /* HAVE_PROTOBUF */
}

void DNSProtoBufMessage::addRR(const std::string& strName, uint32_t uType, uint32_t uClass, uint32_t uTTL, const std::string& strBlob)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uType and uClass should be uint16_t.

requestor = newCA(dq.remoteaddr:toString())
if requestor:isIPv4() then
requestor:truncate(24)
luasmn = newSuffixMatchNode()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have inserted a lot of trailing spaces, could you fix that?

self.assertEquals(msg.type, dnsmessage_pb2.PBDNSMessage.DNSQueryType) # testProtobuf()
else:
self.assertEquals(msg.type, dnsmessage_pb2.PBDNSMessage.DNSResponseType) # testLuaProtobuf()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That blocks looks very weird because it's effectively making checkProtobufQuery() accept responses. Perhaps we should call checkProtobufResponse() when the query has been turned into a response instead?

Copy link
Member

@rgacogne rgacogne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits, but nothing that can't be fixed at a later time.

pdns/dnsdist.hh Outdated
{
}

void add(std::string strLabel, std::string strValue)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strLabel and strValue could be const references instead

pdns/dnsdist.hh Outdated
std::unordered_map<std::string, std::string>tagData;

private:
const char *strSep = "\t";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be static

pdns/protobuf.cc Outdated
#endif /* HAVE_PROTOBUF */
}

void DNSProtoBufMessage::addRR(const std::string& strName, uint16_t uType, uint16_t uClass, uint32_t uTTL, const std::string& strBlob)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strName should probably be a const DNSName& instead of a string

pdns/protobuf.cc Outdated
rr->set_type(uType);
rr->set_class_(uClass);
rr->set_ttl(uTTL);
rr->set_rdata((const uint8_t *) strBlob.c_str(), strBlob.size());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm not mistaken the C-style cast to const uint8_t * is not needed.

@sethomisc
Copy link
Contributor Author

Remi,

changed the strLabel and strValue to be const references in Qtag::add() located in dnsdist.hh

changed const char *strSep =\t"; to static constexpr char const *strSep = "\t"; in dnsdist.hh

removed the (const uint8 t *) cast in DNSProBufMessage:addRR() in protobuf.cc

unsure if I should change strName from a string to a const DNSName& as I intended to call the function with just the DNS string like dr.qname:toString(), please advise me Remi.

Again thank you for all your input Remi, Seth

@sethomisc
Copy link
Contributor Author

Remi,
I have rebased dnsdist-mod2.
Please let me know if there are any problems.
Seth

@rgacogne
Copy link
Member

rgacogne commented Aug 2, 2017

Hi Seth! Thanks, there is an issue with your last commit that completely removes the README-dnsdist.md file, causing the test to file. Could you just drop that last commit and amend e75a021 ("Fixed requests from Remi July 7") to remove any alteration to that file?

Commented line around new code segments has key words 'GCA' and 'Seth' for text searching.
Commands don't have XXX suffix anymore.
Timestamp for the setProtobufResponseType() function is supplied by lua now and not done in C++.
Current commands are: setTag, getTagArray, setTagArray, setProtobufResponseType
…ed example script in pdns/zzz-gca-example directory
…est_ProtobufTag.py

script to execute it in pdns/zzz-gca-example/test-protobuf-tag.sh
It's OK for the Lua binding to accept a string, but the internal functions
should really use a `DNSName`.
@nesv
Copy link

nesv commented Aug 2, 2017

@rgacogne Along with your requested amendment to e75a021, I have also removed some other files that were introduced by that commit, that probably shouldn't have been there:

  • pdns/dnsdistdist/cdb.so
  • contrib/dnsmessage_pb2.py{,c}

@rgacogne rgacogne merged commit 5ab772f into PowerDNS:master Aug 3, 2017
@rgacogne
Copy link
Member

rgacogne commented Aug 3, 2017

Merged, thank you again!

@rgacogne rgacogne mentioned this pull request Aug 3, 2017
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants