Add ability to correlate dynamic fields between events

[chemamartinez]

This PR aims to solve the use case of #2531 and cover the misfunction reported at #2524.

Description

Two options have been added to the rule options:

• same_field: It matches when the value of a dynamic field from an incoming event is the same as the one of a previous event which matched the same rule.
• not_same_field: The same case but when the value is different between both events.

Use case

Here we can see a simple example of using these two new options:

<rule id="100001" level="3">
    <if_sid>221</if_sid>
    <field name="netinfo.iface.name">ens33</field>
    <description>Testing interface alert</description>
  </rule>

  <rule id="100002" level="7" frequency="3" timeframe="300">
    <if_matched_sid>100001</if_matched_sid>
    <same_field>netinfo.iface.name</same_field>
    <same_field>netinfo.iface.mac</same_field>
    <not_same_field>netinfo.iface.rx_bytes</not_same_field>
    <options>no_full_log</options>
    <description>Testing options for correlating repeated fields</description>
  </rule>

Rule 10002 matches when the third network inventory scan reports the same MAC address for the interface ens33 but the amount of received packets has changed between events. Here we have the associated alert:

{
  "timestamp": "2019-02-25T07:49:50.581-0800",
  "rule": {
    "level": 7,
    "description": "Testing options for correlating repeated fields",
    "id": "100002",
    "frequency": 3,
    "firedtimes": 3,
  ...
  "data": {
    "type": "network",
    "netinfo": {
      "iface": {
        "name": "ens33",
        "type": "ethernet",
        "state": "up",
        "mtu": "1500",
        "mac": "00:0c:29:58:1c:4c",
        "tx_packets": "8718126",
        "rx_packets": "9207404",
        "tx_bytes": "2070054274",
        "rx_bytes": "1340293681",
        "tx_errors": "0",
        "rx_errors": "0",
        "tx_dropped": "0",
        "rx_dropped": "0",
        "ipv4": {
          "gateway": "unknown",
          "dhcp": "disabled",
          "address": "172.16.98.128",
          "netmask": "255.255.255.0",
          "broadcast": "172.16.98.255"
        },
        "ipv6": {
          "dhcp": "enabled",
          "address": "fe80::20c:29ff:fe58:1c4c",
          "netmask": "ffff:ffff:ffff:ffff::"
        }
      }
    }
  },
  "location": "syscollector"
}

Valgrind report

It has been tested by using valgrind on the Analysis daemon and this is the result:

==89372== LEAK SUMMARY:
==89372==    definitely lost: 100 bytes in 9 blocks
==89372==    indirectly lost: 0 bytes in 0 blocks
==89372==      possibly lost: 7,072 bytes in 26 blocks
==89372==    still reachable: 10,214,019 bytes in 46,925 blocks
==89372==         suppressed: 0 bytes in 0 blocks
==89372== Reachable blocks (those to which a pointer was found) are not shown.
==89372== To see them, rerun with: --leak-check=full --show-leak-kinds=all

Lost memory is related to known issues when creating the child threads.

[vikman90]

I think we don't need these goto.

[vikman90]

Please optimize this algorithm: each for will iterate over all the fields even after matching a field.
Surely you can use break and found to achieve it.

[vikman90]

LGTM!

I ran ossec-logtest and ossec-analysisd with Valgrind and a bunch of input logs.

Cheers @chemamartinez !

[ambatshri]

Hey,
Will this even work with WinEventLog, or just with the WinEventChannel and if not is there any similar capability for the wineventlog ?
Another question is why we're having two decoders for windows events.? EventChannel and EventLog.