Add log.message #3

ruflin · 2018-05-28T11:21:04Z

The field log.message contains the full log message before splitting it up in multiple parts.

In contrast to the message field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message.

This field is not indexed and doc_values are disabled so it can't be
queried but the value can be retrieved from _source.

webmat · 2018-05-29T19:03:26Z

I would prefer event.raw over log.message. I have built an operational monitoring system in the past, that was being fed by SendGrid's Event Webhook (just an example). This is operational data about email delivery and engagement. Having these original events be put in log.message would sound contrived.

I think event.raw sounds more general. Yes, you can stick logs in there, but also any other operational events.

Just my 2¢ :-)

ruflin · 2018-05-30T06:30:55Z

Thanks a lot for the feedback, that is super useful. Currently the way I differentiate between logs and metrics is that metrics are pulled on a predefined period and logs are pushed, but this line is very blurry. Based on this the above would also be logs I'm thinking. But I see that a user is in this case not going to look for the "raw event" under log.

The initial reason I wanted to get it out of event prefix was that it felt like the only object in there that does not contain meta information but actual data. I'm also ok with leaving where it is.

@webmat Out of curiosity: Would you categorise operational events under logs, metrics or it's own category?

ruflin · 2018-05-30T11:15:49Z

@webmat Working on elastic/beats#7207 I realised event.raw and log.message might be even two different fields with different purpose. In elastic/beats#7207 I do some processing on the field like encoding, multiline and strip new lines as the target is to make it possible to reprocess the event. In event.raw I would expect a completely unprocessed event. Perhaps we ned both?

webmat · 2018-05-30T12:47:07Z

? @webmat Out of curiosity: Would you categorise operational events under logs, metrics or it's own category?

For the type of data I was dealing with, I would definitely say "events", in the sense that it was not just numerical data. I was processing email addresses, IP addresses, email headers and email categories. So full on events with lots of juicy data ;-)

On the other hand, I don't have a strong attachment to the "event" part of event.raw. It could totally be something else equally generic, like raw_payload or the like. The part I was reacting to was losing the "generic"-ness of "event" and going to "log", when not all operational events come from logs*.

* Well, depending on how broad you define log. But I think most people think of text files, when they hear "log". And "most people" is our audience for this ;-)

ave19 · 2018-05-30T17:29:58Z

We have some data sources that aren't logs. Some are data base scrapes and some are API calls we make to a service and then index what comes out. Not sure log.message would apply as it was never in a log and was really never a message either. They do describe things that happened, or at least facts that were true at a particular time, though, so event seems ok.

Indexing the original data (before it is parsed and mutilated) as *.raw would allow us to re-parse events later if we find a bug or gain some other new insight. We wouldn't have to go back to the original source, which may or may not have the information anymore.

ruflin · 2018-05-31T12:15:13Z

Based on the feedback above I reverted the change to event.raw. The idea for this one is to really contain the raw initial event.

For log.message I think it fits the logging purpose and is potentially already modified a bit like encoding applied, new lines removed, multiline applied so it does not full fit event.raw.

My suggestion is now to only introduce log.message.

webmat · 2018-05-31T13:52:58Z

Yes that part I agree with. Makes total sense, especially wrt reconstructed messages (like multiline).

praseodym · 2018-05-31T19:33:27Z

schemas/log.yml

+
+        In contrast to the `message` field which can contain an extracted part
+        of the log message, this field contains the raw log message and should
+        not be processed. It can have already some modifications like encoding


This sounds like a contractiction: 'raw log message' and 'not be processed' vs. 'can have already some modifications'

Agree. Any suggestion for a better description instead of raw we could use here?

praseodym · 2018-05-31T19:34:12Z

CHANGELOG.md

@@ -6,6 +6,8 @@ All notable changes to this project will be documented in this file based on the

 ### Breaking changes

+* Rename `event.raw` to `log.message`. #3


This change should be removed now that event.raw stays as is.

fixed, thanks for the review.

ruflin · 2018-06-06T07:23:11Z

PR updated with disabling doc_values and setting index: false to disable indexing. This should reduce the storage needed in Elasticsearch.

praseodym · 2018-06-06T21:41:24Z

README.md

@@ -275,6 +275,7 @@ Fields which are specific to log events.
 | <a name="log.level"></a>`log.level`  | Log level of the log event.<br/>Some examples are `WARN`, `ERR`, `INFO`.  | keyword  |   | `ERR`  |
 | <a name="log.line"></a>`log.line`  | Line number the log event was collected from.  | long  |   | `18`  |
 | <a name="log.offset"></a>`log.offset`  | Offset of the beginning of the log event.  | long  |   | `12`  |
+| <a name="log.message"></a>`log.message`  | This is the log message and contains the full log message before splitting it up in multiple parts.<br/>In contrast to the `message` field which can contain an extracted part of the log message, this field contains the raw log message and should not be processed. It can have already some modifications like encoding applied or new lines removed to clean up the log message.<br/>This field is not index and doc_values are disabled so it can't be queried but the value can be retrieved from `_source`.  | keyword  |   | `Sep 19 08:26:10 localhost My log`  |


"not index" -> "not indexed"

Also, w/ regards to my previous comment: "[..] this field contains the raw log message and should not be processed." -> "[..] this field contains the original, full log message."

The field `log.message` contains the full log message before splitting it up in multiple parts. In contrast to the `message` field which can contain an extracted part of the log message, this field contains the original, full log message. It can have already some modifications applied like encoding or new lines removed to clean up the log message. This field is not indexed and doc_values are disabled so it can't be queried but the value can be retrieved from `_source`.

ruflin · 2018-06-11T08:40:02Z

PR rebased, commit and PR message updated, fixes applied. Read for an other review.

andrewkroh · 2018-06-11T12:10:00Z

template.json

@@ -459,6 +459,12 @@
            "line": {
              "type": "long"
            },
+            "message": {
+              "doc_values": false,
+              "ignore_above": 1024,


For logs, ignore_above: 1024 will be too small.

I checked the docs for this and run some tests: https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html

It seems like ignore_above does not play any role here as it's not index anyway. _source is not affected by ignore_above.

Good. Thanks for testing.

Maybe we shouldn't even set an ignore_above when index: false? I imagine others having a similar reaction to mine.

ruflin added the review label May 28, 2018

ruflin force-pushed the log-message branch 2 times, most recently from f3bf226 to 147b59e Compare May 28, 2018 11:27

ruflin mentioned this pull request May 29, 2018

Feature: Dissect processor elastic/beats#6925

Merged

ruflin added the discuss label May 30, 2018

praseodym reviewed May 31, 2018

View reviewed changes

ruflin changed the title ~~Rename event.raw to log.message~~ Add log.message Jun 4, 2018

ruflin removed the discuss label Jun 6, 2018

ruflin force-pushed the log-message branch from 2e1ab6c to bff926f Compare June 6, 2018 07:22

ruflin force-pushed the log-message branch from bff926f to 312a57a Compare June 6, 2018 07:23

praseodym reviewed Jun 6, 2018

View reviewed changes

ruflin force-pushed the log-message branch from 3d85f81 to ec5f621 Compare June 11, 2018 08:37

ruflin force-pushed the log-message branch from ec5f621 to 2a67f26 Compare June 11, 2018 08:39

andrewkroh reviewed Jun 11, 2018

View reviewed changes

andrewkroh merged commit b058a31 into elastic:master Jun 11, 2018

ruflin deleted the log-message branch June 11, 2018 13:01

Aqualie mentioned this pull request Dec 17, 2019

New use case: Firewall #125

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add log.message #3

Add log.message #3

ruflin commented May 28, 2018 •

edited

webmat commented May 29, 2018

ruflin commented May 30, 2018

ruflin commented May 30, 2018

webmat commented May 30, 2018 •

edited

ave19 commented May 30, 2018

ruflin commented May 31, 2018

webmat commented May 31, 2018

praseodym May 31, 2018

ruflin Jun 1, 2018

praseodym May 31, 2018

ruflin Jun 1, 2018

ruflin commented Jun 6, 2018

praseodym Jun 6, 2018

praseodym Jun 6, 2018

ruflin Jun 11, 2018

ruflin commented Jun 11, 2018

andrewkroh Jun 11, 2018

ruflin Jun 11, 2018

andrewkroh Jun 11, 2018

		@@ -6,6 +6,8 @@ All notable changes to this project will be documented in this file based on the

		### Breaking changes

		* Rename `event.raw` to `log.message`. #3

Add log.message #3

Add log.message #3

Conversation

ruflin commented May 28, 2018 • edited

webmat commented May 29, 2018

ruflin commented May 30, 2018

ruflin commented May 30, 2018

webmat commented May 30, 2018 • edited

ave19 commented May 30, 2018

ruflin commented May 31, 2018

webmat commented May 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jun 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Jun 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 28, 2018 •

edited

webmat commented May 30, 2018 •

edited