Merge ssh://github.com/mmguero-dev/spicy-analyzers

Couple tweaks included:

- Turned PR summary into README
- Capitalized module names

* ssh://github.com/mmguero-dev/spicy-analyzers:
  something in 51ec8fd broke something, this will (should) fix it
  something in 51ec8fd broke something, this will (should) fix it
  try to expose less useless stuff in each unit, for #56
  update changes
  Remove analyzer_id from scripts for ipsec.
  update .log files from baseline test to reflect new logs fields
  Added proto (TCP for now, may include UDP in the future ) to the ldap logs
  changes made after @bbanier's review of PR #56. See the comments in that review for the details.
  changes made after @bbanier's review of PR #56. See the comments in that review for the details.
  Moving computation for |self.seq.submessages| to temporary local variable to fix CI integration error.
  void fields never store a value and cannot be named
  void fields never store a value and cannot be named
  fix LDAP test (specify -C to zeek)
  update changes
  added ldap test
  rename ldap.zeek to main.zeek
  don't save both 'bind' and 'bind simple'/'bind SASL' in the operations list
  added more ldap codes
  Added ldap version
  Topic/ldap (#1)

CHANGES

@@ -1,4 +1,8 @@
 
+0.2.12 | 2021-05-19 14:16:26 +0200
+
+  * Add LDAP protocol analyzer. (Seth Grover)
+
 0.2.11 | 2021-05-17 09:39:00 +0200
 
   * Remove `analyzer_id` from scripts for IPSec. (Keith Jones)

README.md

@@ -11,6 +11,7 @@ Currently, the following analyzers are included:
 - DNS <sup>[1]</sup>
 - HTTP <sup>[1]</sup>
 - IPSec
+- LDAP
 - OpenVPN
 - PNG
 - Portable Executable (PE) <sup>[2]</sup>

analyzer/__load__.zeek

@@ -5,6 +5,7 @@
 @load ./protocol/dns
 @load ./protocol/http
 @load ./protocol/ipsec
+@load ./protocol/ldap
 @load ./protocol/openvpn
 @load ./protocol/tftp
 @load ./protocol/wireguard

analyzer/protocol/CMakeLists.txt

@@ -4,6 +4,7 @@ add_subdirectory(dhcp)
 add_subdirectory(dns)
 add_subdirectory(http)
 add_subdirectory(ipsec)
+add_subdirectory(ldap)
 add_subdirectory(openvpn)
 add_subdirectory(tftp)
 add_subdirectory(wireguard)

analyzer/protocol/ldap/CMakeLists.txt

@@ -0,0 +1,3 @@
+# Copyright (c) 2021 by the Zeek Project. See LICENSE for details.
+
+spicy_add_analyzer(LDAP ldap.spicy ldap_zeek.spicy ldap.evt)

analyzer/protocol/ldap/README.md

@@ -0,0 +1,64 @@
+LDAP Analyzer
+=============
+
+Here's what it has:
+
+- ASN.1 structure decoding: this is probably generally useful for more than just the LDAP parser, so it may be of interest for this to be included somehow as part of spicy's standard modules or whatever
+  - everything is working except for the "constructed" forms of `ASN1BitString` and `ASN1OctetString`
+- LDAP: the LDAP parsing is basically "done once" through a single call to `ASN1Message` (which parses itself recursively) and then the application-level data is also parsed via `&parse-from` a byte array belonging to the outer ASN.1 sequence. This second level of parsing is also done using the ASN.1 data types.
+  - events
+    - `ldap::message` - called for each LDAP message
+    - `ldap::bindreq` - when a bind request is made
+    - `ldap::searchreq` - basic search request information
+    - `ldap::searchres` - called each time a search result is returned
+  - enums
+    - `ProtocolOpcode`
+    - `ResultCode`
+    - `BindAuthType`
+    - `SearchScope`
+    - `SearchDerefAlias`
+    - `FilterType`
+  - Zeek log files
+    - `ldap.log` - contains information about all LDAP messages except those that are search-related. Log lines are grouped by connection ID + message ID
+      - `ts` (time)
+      - `uid` (connection UID)
+      - `id` (connection ID 4-tuple)
+      - `proto` (transport protocol)
+      - `message_id` (LDAP message ID)
+      - `version` (LDAP version for bind requests)
+      - `opcode` (set of 1..n operations from this uid+message_id)
+      - `result` (set of 1..n results from this uid+message_id)
+      - `diagnostic_message` (vector of 0..n diagnostic message strings)
+      - `object` (vector of 0..n "objects," the meaning of which depends on the operation)
+      - `argument` (vector of 0..n "argument," the meaning of which depends on the operation)
+    - `ldap_search.log` - contains information about LDAP searches. Log lines are grouped by connection ID + message ID
+      - `ts` (time)
+      - `uid` (connection UID)
+      - `id` (connection ID 4-tuple)
+      - `proto` (transport protocol)
+      - `message_id` (LDAP message ID)
+      - `scope` (set of 1..n search scopes defined in this uid+message_id)
+      - `deref` (set of 1..n search deref alias options defined in this uid+message_id)
+      - `base_object` (vector of 0..n search base objects specified)
+      - `result_count` (number of result entries returned)
+      - `result` (set of 1..n results from this uid+message_id)
+      - `diagnostic_message` (vector of 0..n diagnostic message strings)
+  - test
+    - basic tests for detecting plugin presence and simple bind and search result/requests
+
+Here's what it doesn't have, which could be added by future parties interested in expanding it:
+
+- although LDAP can use UDP as transport, currently the analyzer only looks for `389/tcp, 3268/tcp`; this may be easy to change, but I don't have any examples of traffic to test it with so I haven't bothered
+- LDAP [referrals](https://tools.ietf.org/html/rfc4511#section-4.1.10) are not parsed out of the results
+- [SASL credentials](https://datatracker.ietf.org/doc/html/rfc4511#section-4.2) in bind requests are not being parsed beyond the mechanism string
+- SASL information in bind responses are not being parsed; for that matter, SASL-based LDAP stuff hasn't been tested much and may have issues
+- Search filters and attributes: while basic parsing is done in the `AttributeSelection`, `AttributeValueAssertion`, `SubstringFilter` and `SearchFilter` units, I'm not really pulling stuff out from the search filter tree (as the protocols allows arbitrarily complex combinations of AND, OR, substring, etc. that I don't think would be easily representable in a log file)
+- the details of `SearchResultReference` are not being parsed
+- the only detail of `ModifyRequest` being parsed is the object name
+- the details of `AddRequest` are not being parsed
+- the details of `ModDNRequest` are not being parsed
+- the details of `CompareRequest` are not being parsed
+- the details of `AbandonRequest` are not being parsed
+- the details of `ExtendedRequest` are not being parsed
+- the details of `ExtendedResponse` are not being parsed
+- the details of `IntermediateResponse` are not being parsed

analyzer/protocol/ldap/__load__.zeek

@@ -0,0 +1,3 @@
+# Copyright (c) 2021 by the Zeek Project. See LICENSE for details.
+
+@load ./main

analyzer/protocol/ldap/asn1.spicy

@@ -0,0 +1,286 @@
+# Copyright (c) 2021 by the Zeek Project. See LICENSE for details.
+
+module ASN1;
+
+###############################################################################
+# ASN.1 structure decoding
+#
+# A Layman's Guide to a Subset of ASN.1, BER, and DER
+#   http://luca.ntop.org/Teaching/Appunti/asn1.html
+#
+# ASN.1 Tutorial from Computer Networks and Open Systems:
+#   An Application Development Perspective
+#   https://www.obj-sys.com/asn1tutorial/asn1only.html
+#
+# The ASN1JS tool (http://lapo.it/asn1js and https://github.com/lapo-luchini/asn1js)
+#   is invaluable in debugging ASN.1
+###############################################################################
+
+import spicy;
+
+#- ASN.1 data types ----------------------------------------------------------
+# https://www.obj-sys.com/asn1tutorial/node124.html
+# https://www.obj-sys.com/asn1tutorial/node10.html
+
+public type ASN1Type = enum {
+  Boolean = 1,
+  Integer = 2,
+  BitString = 3,
+  OctetString = 4,
+  NullVal = 5,
+  ObjectIdentifier = 6,
+  ObjectDescriptor = 7,
+  InstanceOf = 8,
+  Real = 9,
+  Enumerated = 10,
+  EmbeddedPDV = 11,
+  UTF8String = 12,
+  RelativeOID = 13,
+  Sequence = 16,
+  Set = 17,
+  NumericString = 18,
+  PrintableString = 19,
+  TeletextString = 20,
+  VideotextString = 21,
+  IA5String = 22,
+  UTCTime = 23,
+  GeneralizedTime = 24,
+  GraphicString = 25,
+  VisibleString = 26,
+  GeneralString = 27,
+  UniversalString = 28,
+  CharacterString = 29,
+  BMPString = 30
+};
+
+#- ASN.1 data classes --------------------------------------------------------
+
+public type ASN1Class = enum {
+  Universal = 0,
+  Application = 1,
+  ContextSpecific = 2,
+  Private = 3
+};
+
+#- ASN.1 tag definition (including length) ------------------------------------
+
+type LengthType = unit {
+  var len: uint64;
+  var tag_len: uint8;
+
+  data : bitfield(8) {
+    num: 0..6;
+    islong: 7;
+  };
+
+
+  switch ( self.data.islong ) {
+    0 -> : void {
+      self.len = self.data.num;
+      self.tag_len = 1;
+    }
+    1 -> : bytes &size=self.data.num
+           &convert=$$.to_uint(spicy::ByteOrder::Network) {
+      self.len = $$;
+      self.tag_len = self.data.num + 1;
+    }
+  };
+};
+
+type ASN1Tag = unit {
+  var type_: ASN1Type;
+  var class: ASN1Class;
+  var constructed: bool;
+
+  : bitfield(8) {
+    num: 0..4;
+    constructed: 5;
+    class: 6..7;
+  } {
+    self.type_ = ASN1Type($$.num);
+    self.class = ASN1Class($$.class);
+    self.constructed = cast<bool>($$.constructed);
+  }
+};
+
+#- ASN.1 bit string -----------------------------------------------------------
+# https://www.obj-sys.com/asn1tutorial/node10.html
+
+type ASN1BitString = unit(len: uint64, constructed: bool) {
+  : uint8; # unused bits
+  value_bits: bytes &size=(len - 1);
+
+  # TODO - constructed form
+  # https://github.com/zeek/spicy/issues/921
+  # `bytes` needs << and >> support before we can implement complex bitstrings
+  #
+};
+
+#- ASN.1 octet string ---------------------------------------------------------
+# https://www.obj-sys.com/asn1tutorial/node10.html
+
+type ASN1OctetString = unit(len: uint64, constructed: bool) {
+  value: bytes &size = len;
+
+  # TODO - constructed form
+};
+
+#- ASN.1 various string types -------------------------------------------------
+# https://www.obj-sys.com/asn1tutorial/node124.html
+
+type ASN1String = unit(tag: ASN1Tag, len: uint64) {
+  var value: string = "";
+
+  : ASN1OctetString(len, tag.constructed) {
+    switch ( tag.type_ ) {
+
+      # see "Restricted Character String Types" in
+      # "Generic String Encoding Rules (GSER) for ASN.1 Types"
+      # (https://datatracker.ietf.org/doc/html/rfc3641#section-3.2)
+
+      case ASN1Type::PrintableString,
+           ASN1Type::GeneralizedTime,
+           ASN1Type::UTCTime: {
+        self.value = $$.value.decode(hilti::Charset::ASCII);
+      }
+
+      case ASN1Type::UTF8String,
+           ASN1Type::GeneralString,
+           ASN1Type::CharacterString,
+           ASN1Type::GraphicString,
+           ASN1Type::IA5String,
+           ASN1Type::NumericString,
+           ASN1Type::TeletextString,
+           ASN1Type::VideotextString,
+           ASN1Type::VisibleString,
+           # TODO: RFC3641 mentions special UTF-8 mapping rules for
+           # BMPString and UniversalString. This *may* not be correct.
+           ASN1Type::BMPString,
+           ASN1Type::UniversalString: {
+        self.value = $$.value.decode(hilti::Charset::UTF8);
+      }
+    }
+  }
+};
+
+#- ASN.1 OID ------------------------------------------------------------------
+# https://www.obj-sys.com/asn1tutorial/node124.html
+
+type ASN1ObjectIdentifierNibble = unit {
+  data : bitfield(8) {
+    num: 0..6;
+    more: 7;
+  };
+} &convert=self.data;
+
+type ASN1ObjectIdentifier = unit(len: uint64) {
+  var oid: vector<uint64>;
+  var temp: uint64;
+  var oidstring: string;
+
+  : uint8 if ( len >= 1 ) {
+    self.temp = $$ / 40;
+    self.oid.push_back( self.temp );
+    self.oidstring = "%d" % (self.temp);
+    self.temp = $$ % 40;
+    self.oid.push_back( self.temp );
+    self.oidstring = self.oidstring + ".%d" % (self.temp);
+    self.temp = 0;
+  }
+
+  sublist: ASN1ObjectIdentifierNibble[len - 1] foreach {
+    self.temp = ( self.temp<<7 ) | $$.num;
+    if ( $$.more != 1 ) {
+      self.oid.push_back(self.temp);
+      self.oidstring = self.oidstring + ".%d" % (self.temp);
+      self.temp = 0;
+    }
+  }
+};
+
+
+#- ASN.1 message header (tag + length information) ----------------------------
+
+public type ASN1Header = unit {
+  tag: ASN1Tag;
+  len: LengthType;
+};
+
+#- ASN.1 message body ---------------------------------------------------------
+
+public type ASN1Body = unit(head: ASN1Header, recursive: bool) {
+  switch ( head.tag.type_ ) {
+
+    ASN1Type::Boolean -> bool_value: uint8 &convert=cast<bool>($$) &requires=head.len.len==1;
+
+    ASN1Type::Integer,
+    ASN1Type::Enumerated -> num_value: bytes &size=head.len.len
+                                             &convert=$$.to_int(spicy::ByteOrder::Big);
+
+    ASN1Type::NullVal -> null_value: bytes &size=0 &requires=head.len.len==0;
+
+    ASN1Type::BitString -> bitstr_value: ASN1BitString(head.len.len, head.tag.constructed);
+
+    ASN1Type::OctetString -> str_value: ASN1OctetString(head.len.len, head.tag.constructed)
+                                        &convert=$$.value.decode(hilti::Charset::ASCII);
+
+    ASN1Type::ObjectIdentifier -> str_value: ASN1ObjectIdentifier(head.len.len)
+                                             &convert=$$.oidstring;
+
+    ASN1Type::BMPString,
+    ASN1Type::CharacterString,
+    ASN1Type::GeneralizedTime,
+    ASN1Type::GeneralString,
+    ASN1Type::GraphicString,
+    ASN1Type::IA5String,
+    ASN1Type::NumericString,
+    ASN1Type::PrintableString,
+    ASN1Type::TeletextString,
+    ASN1Type::UTCTime,
+    ASN1Type::UTF8String,
+    ASN1Type::VideotextString,
+    ASN1Type::VisibleString,
+    ASN1Type::UniversalString -> str_value: ASN1String(head.tag, head.len.len)
+                                            &convert=$$.value;
+
+    ASN1Type::Sequence, ASN1Type::Set -> seq: ASN1SubMessages(head.len.len) if (recursive);
+
+    # TODO: ASN1Type values not handled yet
+    ASN1Type::ObjectDescriptor,
+    ASN1Type::InstanceOf,
+    ASN1Type::Real,
+    ASN1Type::EmbeddedPDV,
+    ASN1Type::RelativeOID -> unimplemented_value: bytes &size=head.len.len;
+
+    # unknown (to me) ASN.1 enumeration, skip over silently
+    * -> unimplemented_value: bytes &size=head.len.len;
+  };
+};
+
+#- ASN.1 array of ASN.1 sequence/set sub-messages (up to msgLen bytes) --------
+
+public type ASN1SubMessages = unit(msgLen: uint64) {
+  submessages: ASN1Message(True)[] &eod;
+} &size=msgLen;
+
+#- ASN.1 message with header and body -----------------------------------------
+# Universal or Application/ContextSpecific/Private
+# - if Universal, body:ASN1Body is parsed
+# - else, application_data:bytes stores data array
+
+public type ASN1Message = unit(recursive: bool) {
+  var application_id: int32;
+
+  head: ASN1Header;
+  switch ( self.head.tag.class ) {
+
+    ASN1Class::Universal -> body: ASN1Body(self.head, recursive);
+
+    ASN1Class::Application,
+    ASN1Class::ContextSpecific,
+    ASN1Class::Private -> application_data: bytes &size=self.head.len.len {
+      self.application_id = cast<int32>(self.head.tag.type_);
+    }
+
+  };
+};