Add event.start and event.end (elastic#185)

`event.start` and `event.end` are date fields that demarcate the beginning and end, respectively,
of an activity. For example in a network flow the event.start is time time of the first packet in the
flow and `event.end` time the time of the last observed packet in the flow.

The `event.duration` value is then computed as the difference between end and start times.

CHANGELOG.md

@@ -9,6 +9,7 @@ All notable changes to this project will be documented in this file based on the
 
 ### Added
 * Add `host.name` field and clarify usage of `host.hostname`.
+* Add `event.start` and `event.end` date fields.
 
 ### Improvements
 * Improve and clarify the definition of Device fields #192

README.md

@@ -200,8 +200,10 @@ The event fields are used for context information about the data itself.
 | <a name="event.severity"></a>event.severity | Severity describes the severity of the event. What the different severity values mean can very different between use cases. It's up to the implementer to make sure severities are consistent across events. | core | long | `7` |
 | <a name="event.original"></a>event.original | Raw text message of entire event. Used to demonstrate log integrity.<br/>This field is not indexed and doc_values are disabled. It cannot be searched, but it can be retrieved from `_source`. | core | keyword | `Sep 19 08:26:10 host CEF:0&#124;Security&#124; threatmanager&#124;1.0&#124;100&#124; worm successfully stopped&#124;10&#124;src=10.0.0.1 dst=2.1.2.2spt=1232` |
 | <a name="event.hash"></a>event.hash | Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity. | extended | keyword | `123456789012345678901234567890ABCD` |
-| <a name="event.duration"></a>event.duration | Duration of the event in nanoseconds. | core | long |  |
+| <a name="event.duration"></a>event.duration | Duration of the event in nanoseconds.<br/>If event.start and event.end are known this value should be the difference between the end and start time. | core | long |  |
 | <a name="event.created"></a>event.created | event.created contains the date when the event was created.<br/>This timestamp is distinct from @timestamp in that @timestamp contains the processed timestamp. For logs these two timestamps can be different as the timestamp in the log line and when the event is read for example by Filebeat are not identical. `@timestamp` must contain the timestamp extracted from the log line, event.created when the log line is read. The same could apply to package capturing where @timestamp contains the timestamp extracted from the network package and event.created when the event was created.<br/>In case the two timestamps are identical, @timestamp should be used. | core | date |  |
+| <a name="event.start"></a>event.start | event.start contains the date when the event started or when the activity was first observed. | extended | date |  |
+| <a name="event.end"></a>event.end | event.end contains the date when the event ended or when the activity was last observed. | extended | date |  |
 | <a name="event.risk_score"></a>event.risk_score | Risk score or priority of the event (e.g. security solutions). Use your system's original value here. | core | float |  |
 | <a name="event.risk_score_norm"></a>event.risk_score_norm | Normalized risk score or priority of the event, on a scale of 0 to 100.<br/>This is mainly useful if you use more than one system that assigns risk scores, and you want to see a normalized value across all systems. | extended | float |  |
 

fields.yml

@@ -473,6 +473,9 @@
           description: >
             Duration of the event in nanoseconds.
     
+            If event.start and event.end are known this value should be the
+            difference between the end and start time.
+
         - name: created
           level: core
           type: date
@@ -490,6 +493,20 @@
 
             In case the two timestamps are identical, @timestamp should be used.
 
+        - name: start
+          level: extended
+          type: date
+          description: >
+            event.start contains the date when the event started or when the
+            activity was first observed.
+    
+        - name: end
+          level: extended
+          type: date
+          description: >
+            event.end contains the date when the event ended or when the activity
+            was last observed.
+    
         - name: risk_score
           level: core
           type: float

schema.csv

@@ -41,13 +41,15 @@ event.category,keyword,core,metrics
 event.created,date,core,
 event.dataset,keyword,core,stats
 event.duration,long,core,
+event.end,date,extended,
 event.hash,keyword,extended,123456789012345678901234567890ABCD
 event.id,keyword,core,8a4f500d
 event.module,keyword,core,mysql
 event.original,keyword,core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
 event.risk_score,float,core,
 event.risk_score_norm,float,extended,
 event.severity,long,core,7
+event.start,date,extended,
 event.type,keyword,core,nginx-stats-metrics
 file.ctime,date,extended,
 file.device,keyword,extended,

schemas/event.yml

@@ -106,6 +106,9 @@
       description: >
         Duration of the event in nanoseconds.
 
+        If event.start and event.end are known this value should be the
+        difference between the end and start time.
+
     - name: created
       level: core
       type: date
@@ -123,6 +126,20 @@
 
         In case the two timestamps are identical, @timestamp should be used.
 
+    - name: start
+      level: extended
+      type: date
+      description: >
+        event.start contains the date when the event started or when the
+        activity was first observed.
+
+    - name: end
+      level: extended
+      type: date
+      description: >
+        event.end contains the date when the event ended or when the activity
+        was last observed.
+
     - name: risk_score
       level: core
       type: float

template.json

@@ -215,6 +215,9 @@
             "duration": {
               "type": "long"
             },
+            "end": {
+              "type": "date"
+            },
             "hash": {
               "ignore_above": 1024,
               "type": "keyword"
@@ -242,6 +245,9 @@
             "severity": {
               "type": "long"
             },
+            "start": {
+              "type": "date"
+            },
             "type": {
               "ignore_above": 1024,
               "type": "keyword"