LDAP analyzer

[mmguero]

The LDAP analyzer (and its underlying ASN.1 module) is to the point where it has the things I need from it. I've tested it on as much live traffic and PCAP traffic as I could and it seems to be working well.

Here's what it has:

• ASN.1 structure decoding: this is probably generally useful for more than just the LDAP parser, so it may be of interest for this to be included somehow as part of spicy's standard modules or whatever
  • everything is working except for the "constructed" forms of ASN1BitString and ASN1OctetString
• LDAP: the LDAP parsing is basically "done once" through a single call to ASN1Message (which parses itself recursively) and then the application-level data is also parsed via &parse-from a byte array belonging to the outer ASN.1 sequence. This second level of parsing is also done using the ASN.1 data types.
  • events
    • ldap::message - called for each LDAP message
    • ldap::bindreq - when a bind request is made
    • ldap::searchreq - basic search request information
    • ldap::searchres - called each time a search result is returned
  • enums
    • ProtocolOpcode
    • ResultCode
    • BindAuthType
    • SearchScope
    • SearchDerefAlias
    • FilterType
  • Zeek log files
    • ldap.log - contains information about all LDAP messages except those that are search-related. Log lines are grouped by connection ID + message ID
      • ts (time)
      • uid (connection UID)
      • id (connection ID 4-tuple)
      • proto (transport protocol)
      • message_id (LDAP message ID)
      • version (LDAP version for bind requests)
      • opcode (set of 1..n operations from this uid+message_id)
      • result (set of 1..n results from this uid+message_id)
      • diagnostic_message (vector of 0..n diagnostic message strings)
      • object (vector of 0..n "objects," the meaning of which depends on the operation)
      • argument (vector of 0..n "argument," the meaning of which depends on the operation)
    • ldap_search.log - contains information about LDAP searches. Log lines are grouped by connection ID + message ID
      • ts (time)
      • uid (connection UID)
      • id (connection ID 4-tuple)
      • proto (transport protocol)
      • message_id (LDAP message ID)
      • scope (set of 1..n search scopes defined in this uid+message_id)
      • deref (set of 1..n search deref alias options defined in this uid+message_id)
      • base_object (vector of 0..n search base objects specified)
      • result_count (number of result entries returned)
      • result (set of 1..n results from this uid+message_id)
      • diagnostic_message (vector of 0..n diagnostic message strings)
  • test
    • basic tests for detecting plugin presence and simple bind and search result/requests

Here's what it doesn't have, which could be added by future parties interested in expanding it:

• although LDAP can use UDP as transport, currently the analyzer only looks for 389/tcp, 3268/tcp; this may be easy to change, but I don't have any examples of traffic to test it with so I haven't bothered
• LDAP referrals are not parsed out of the results
• SASL credentials in bind requests are not being parsed beyond the mechanism string
• SASL information in bind responses are not being parsed; for that matter, SASL-based LDAP stuff hasn't been tested much and may have issues
• Search filters and attributes: while basic parsing is done in the AttributeSelection, AttributeValueAssertion, SubstringFilter and SearchFilter units, I'm not really pulling stuff out from the search filter tree (as the protocols allows arbitrarily complex combinations of AND, OR, substring, etc. that I don't think would be easily representable in a log file)
• the details of SearchResultReference are not being parsed
• the only detail of ModifyRequest being parsed is the object name
• the details of AddRequest are not being parsed
• the details of ModDNRequest are not being parsed
• the details of CompareRequest are not being parsed
• the details of AbandonRequest are not being parsed
• the details of ExtendedRequest are not being parsed
• the details of ExtendedResponse are not being parsed
• the details of IntermediateResponse are not being parsed

[bbannier]

Thanks for this PR. I made a first round through the ASN.1 parser. Will look at the other parts now.

[mmguero]

Finished @bbannier 's review and making changes from it. I'm going to do some more testing and will reply here once I feel things are all as stable as they were pre-review. Thanks a lot

[mmguero]

Made the changes suggested by @bbannier in his review, minus the few of them that I gave reasons for in the replies to his comments. I'm still getting all the same .log output and passing my tests, so that's a good thing ;).

[bbannier]

Thanks for spending the time to clean this up @mmguero.

Overall this looks good to me. Like I mentioned offline we could still make this easier to consume by minimizing data exposed by units (you often use members for parsing, and then set some dependent unit var; ideally we'd wouldn't expose the member used for parsing after that). If you do not have time for that cleanup, I believe we take that on after merging.

Let's also see whether @rsmmr has any additional suggestions.

[mmguero]

Something got a little bungled in merging from upstream, I'll fix whatever it is and comment here again when I'm good-to-go.

[mmguero]

Okay, I got it sorted. I think I'm good whenever @bbannier and/or @rsmmr are.

[rsmmr]

This has come together nicely, good job!

I went ahead and merge it, and created a ticket tracking some TODOs: #58

Also, I turned the PR's description into a README. @mmguero Feel free to file another PR if you want clean that up a bit more.