A new take on adding missing Syslog fields to ECS. #525

webmat · 2019-08-21T20:59:01Z

This PR is meant to replace and close #301.

Unfortunately #301 is too old to be brought up to date with the current state
of the ECS repo.

While bringing the discussion we had in the original PR to present day ECS,
I'm proposing a few changes to what we had there. Comments welcome, if you
agree/disagree.

In this PR:

Modify description of log.level and associate it with Syslog's severity label.
Modify description of event.severity and associate it with Syslog's severity numeric code.
Add field log.facility as the text-based representation.
Add field log.facility_code as the numeric representation.
Add field log.priority (numeric only: 8 * facility + severity)

Differences with #301:

I'm nesting under log.* instead of event.*, because I feel like these very
Syslog-specific fields work better there than in the already crowded event.*.
I'm not adding a text field for the priority label. There's no such thing as a priority label in Syslog.
I'm specifically calling out the Syslog mapping in the existing field log.level and event.severity.

webmat · 2019-08-21T21:00:51Z

Sorry @kvch for not starting from your original PR. It had been too long and too many things happened in the repo since then :-\

schemas/log.yml

ruflin · 2019-08-22T09:36:07Z

Overall I like the direction this is heading.

PS: Recently also stumbled (again) of the syslog protocol RFC https://tools.ietf.org/html/rfc5424 Mainly putting it here for my own reference :-)

webmat · 2019-08-22T12:52:53Z

Oh jeez, I mistyped it in the descriptions. 5324 should be 5424 :-)

I like the idea of nesting it the facility details together conceptually, but I was actually going for the shortest field name possible for what I think is the most often used field of the two (at least in visualizations / tables). I'd be curious to hear how often people use the facility code vs the facility label.

Fix references to RFC 5424

willemdh · 2019-08-23T06:48:19Z

This is looking good! Glad there will finally be a spot for syslog priority / severity / facility fields. Then we can close #36 :)

Currently, I'm using a dictionary lookup filter to get the severity:

      translate {
        dictionary_path => [ "/etc/logstash/dictionaries/syslogpri.yml" ]
        field => "log.priority"
        destination => "log.severity"
      }

Where /etc/logstash/dictionaries/syslogpri.yml is:

"0": emergency
"8": emergency
"16": emergency
"24": emergency
"32": emergency
"40": emergency
"48": emergency
"56": emergency
"64": emergency
"72": emergency
"80": emergency
"88": emergency
"96": emergency
"104": emergency
"112": emergency
"120": emergency
"128": emergency
"136": emergency
"144": emergency
"152": emergency
"160": emergency
"168": emergency
"176": emergency
"184": emergency
"1": alert
"9": alert
"17": alert
"25": alert
"33": alert
"41": alert
"49": alert
"57": alert
"65": alert
"73": alert
"81": alert
"89": alert
"97": alert
"105": alert
"113": alert
"121": alert
"129": alert
"137": alert
"145": alert
"153": alert
"161": alert
"169": alert
"177": alert
"185": alert
"2": critical
"10": critical
"18": critical
"26": critical
"34": critical
"42": critical
"50": critical
"58": critical
"66": critical
"74": critical
"82": critical
"90": critical
"98": critical
"106": critical
"114": critical
"122": critical
"130": critical
"138": critical
"146": critical
"154": critical
"162": critical
"170": critical
"178": critical
"186": critical
"3": error
"11": error
"19": error
"27": error
"35": error
"43": error
"51": error
"59": error
"67": error
"75": error
"83": error
"91": error
"99": error
"107": error
"115": error
"123": error
"131": error
"139": error
"147": error
"155": error
"163": error
"171": error
"179": error
"187": error
"4": warning
"12": warning
"20": warning
"28": warning
"36": warning
"44": warning
"52": warning
"60": warning
"68": warning
"76": warning
"84": warning
"92": warning
"100": warning
"108": warning
"116": warning
"124": warning
"132": warning
"140": warning
"148": warning
"156": warning
"164": warning
"172": warning
"180": warning
"188": warning
"5": notice
"13": notice
"21": notice
"29": notice
"37": notice
"45": notice
"53": notice
"61": notice
"69": notice
"77": notice
"85": notice
"93": notice
"101": notice
"109": notice
"117": notice
"125": notice
"133": notice
"141": notice
"149": notice
"157": notice
"165": notice
"173": notice
"181": notice
"189": notice
"6": informational
"14": informational
"22": informational
"30": informational
"38": informational
"46": informational
"54": informational
"62": informational
"70": informational
"78": informational
"86": informational
"94": informational
"102": informational
"110": informational
"118": informational
"126": informational
"134": informational
"142": informational
"150": informational
"158": informational
"166": informational
"174": informational
"182": informational
"190": informational
"7": debug
"15": debug
"23": debug
"31": debug
"39": debug
"47": debug
"55": debug
"63": debug
"71": debug
"79": debug
"87": debug
"95": debug
"103": debug
"111": debug
"119": debug
"127": debug
"135": debug
"143": debug
"151": debug
"159": debug
"167": debug
"175": debug
"183": debug
"191": debug

Just noticed this also exists => https://www.elastic.co/guide/en/logstash/current/plugins-filters-syslog_pri.html

I'm guessing this plugin should be updated to reflect the new ecs syslog fields?

For now I'll start using this:

    syslog_pri { } 
    mutate {
        rename => { "syslog_pri" => "log.priority" }
        rename => { "syslog_facility" => "log.facility" }
        rename => { "syslog_facility_code" => "log.facility_code" }
        rename => { "syslog_severity" => "log.level" }
        rename => { "syslog_severity_code" => "event.severity" }
        id => 'mutate-rename-syslogs'
    }

webmat · 2019-08-23T19:28:33Z

@willemdh Yes it should. Please make sure to express this over on the plugin repo as well, to ensure the right people see this :-)

What do you think about the facility & facility_code discussion above? Do you find yourself visualizing mostly the text representation (facility), or do you use the numeric version just as much?

Trying to gauge whether my assumption that the text representation is most often used, as a reason for making that field shorter.

… `log.level`

webmat · 2019-08-26T15:31:44Z

I fixed the incorrect RFC number to be 5424 and added a changelog. I think this is ready for final review.

Only question left to answer is how we want the facility field names to look. Options are:

log.facility (keyword) and log.facility_code (long)
log.facility.name (keyword) and log.facility.code (long)

This PR is currently implementing option 1, as I'm trying to make the field containing the text representation shortest, because I assume this is the one people most often work with & visualize.

Please let me know if you feel we should go with option 2, and why :-)

ruflin · 2019-08-27T07:39:04Z

I have a slight preference for option 2 but both are ok with me. I would like @willemdh do chime in for this as he seems to be the one with most experience related to syslog and it's probably best to follow, what we would expect.

kvch · 2019-08-28T09:52:46Z

@webmat Thank you for picking up this!

willemdh · 2019-08-28T12:12:46Z

@webmat we use the name only.. Codes arent very readable for most. But personally I prefer option 2, because I prefer the dots over underscores..

ruflin · 2019-08-29T06:36:26Z

Thanks for the quick feedback @willemdh . @webmat If you are ok with this, could you change it accordingly?

webmat · 2019-08-30T16:52:14Z

Alright, let's go with the family ;-) Thanks for chiming in, @willemdh!

webmat · 2019-09-04T15:46:17Z

Heads up! After internal discussion, we're changing the proposal a bit.

All of Syslog is now nested under log.*, so there's a new log.severity for the Syslog severity.

We're also clarifying that event.severity is meant to contain the severity according to the source of the event (e.g. firewall, IDS). If the source doesn't provide its own severity, the implementer may optionally copy log.severity to event.severity.

But the proposed mapping for Syslog concepts is now:

severity number: log.severity
severity label: log.level (only outlier, naming wise)
facility number: log.facility.code
facility label: log.facility.name
priority: log.priority

webmat · 2019-09-04T15:58:17Z

Related clarification of the log field set's description: #540.

willemdh · 2019-09-06T07:04:51Z

Hmm I was migrating my esxi filters from our old Logstash servers to a dedicated centralized pipeline and noticed that multiple logs have a different syslog level compared to the vmware esxi level. For example:

<167>2019-09-06T06:41:58.518Z esx49 Rhttpproxy: verbose rhttpproxy[79977592] [Originator@6876 sub=Proxy Req 19530] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x000000917a2a4360] _serverNamespace = /vpxa action = Allow _port = 8089

or

<167>2019-09-06T06:47:36.065Z esx11 Vpxa: verbose vpxa[2198549] [Originator@6876 sub=vpxLro opID=sps-Main-500929-912-84-5f] [VpxLRO] Dispatching Error Handler Functor for haTask--vim.vslm.host.CatalogSyncManager.queryCatalogChange-3734994558 which completed with an error from task-25839

And in this one in syslog it's informational and in the esxi log it's info:

<14>1 2019-09-06T13:04:30.707190+02:00 srvvc01 vpxd 38830 - - Event [12324361] [1-1] [2019-09-06T11:04:30.706935Z] [vim.event.UserLoginSessionEvent] [info] [mydomain\myuser] [] [12324361] [User mydomain\myuser@3.54.4.147 logged in as Java/1.8.0_181]

167 in syslog level means 'debug'. But as you can see vmware outputs the log level as 'verbose'.. As there is no verbose level in syslog, I guess it's actually the same level 'in a different language'. Is there currently a way to catch this in ECS?

This seems like a case where dedicated syslog object, maybe even under log. would have come in handy.

Such as

log.syslog.severity.code
log.syslog.severity.name
log.syslog.facility.code
log.syslog.facility.name
log.syslog.priority

log.level could then contain 'verbose'. when the log does not contain a different level, I could copy log.syslog.severity.name to log.level

But then again I really don't want to be difficult and reopen this discussion... (As you are close to releasing this...) As I already created an esxi object, I can easily put 'verbose' for example in esxi.log.level

webmat · 2019-09-23T18:22:46Z

@willemdh Thanks for chiming in again, don't worry about opening cans of worms. Developing ECS is an exercise in opening cans of worms ;-)

Yeah the more we play with this, the more it looks like we should have all Syslog fields defined. If we acknowledge that a source's severity and a Syslog severity can be different, it specifically means that the text representation of that severity can also differ between the source and the transport.

So thanks for reopening the discussion, actually! :-)

I do like the look of this a lot:

log.syslog.severity.code
log.syslog.severity.name
log.syslog.facility.code
log.syslog.facility.name
log.syslog.priority

Unless there's objections to going in that direction, I think that's the direction I'll take this PR next.

Now I feel vaguely uneasy for adding event.provider as a place for the Windows Event Log's "provider"... Do we need a full field set for Windows Event Logs as well? 🤔 (Anybody reading this who has thoughts about this, please open a new issue. Let's not derail this discussion 😉 )

webmat · 2019-09-25T19:29:13Z

@MikePaquette Updated with the latest discussions. WDYT?

MikePaquette

@webmat this is looking nice - LGTM.

MikePaquette · 2019-08-30T19:37:58Z

schemas/event.yml

+        severity values mean can be different between sources and use cases.
+        It's up to the implementer to make sure severities are consistent across events.
+
+        This field corresponds to Syslog's severity. Note that the text-based


This definition makes me think about what if an event contains both a syslog severity and a different, distinct, event severity? Which one goes here? Given that we're moving all the other syslog fields under log.*, perhaps we should also define log.severity as well? If the event contains only the syslog severity, then both log.severity and event.severity would be populated with the same syslog priority value, but if the event has two severity levels, the syslog severity goes to log.severity and the other severity goes to event.severity WDYT?

The current definitions should have answered that. I take this to mean perhaps the descriptions still needs some work, then :-)

So here's the mapping in the two relevant situations:

1. Syslog event / event source specifies different severity

log.syslog.severity.code: Syslog severity number

log.syslog.severity.name: Syslog severity label

event.severity: Source severity number

log.level: Source severity label

2. Syslog event / event source does not specify different severity

log.syslog.severity.code: Syslog severity number

log.syslog.severity.name: Syslog severity label

event.severity: (optional) Syslog severity number

log.level: (optional) Syslog severity label

And just for completeness

3. Non-syslog event, event source specifies severity

event.severity: Source severity number

log.level: Source severity label

webmat requested a review from MikePaquette August 21, 2019 20:59

webmat self-assigned this Aug 21, 2019

webmat requested a review from ruflin August 21, 2019 20:59

webmat mentioned this pull request Aug 21, 2019

Add syslog protocol fields to event #301

Closed

ruflin reviewed Aug 22, 2019

View reviewed changes

schemas/log.yml Outdated Show resolved Hide resolved

ruflin reviewed Aug 22, 2019

View reviewed changes

schemas/log.yml Outdated Show resolved Hide resolved

Mathieu Martin added 6 commits August 26, 2019 11:17

Add Syslog fields under log., mention Sylog in event.severity and…

d6bf5be

… `log.level`

Generated files

62fbc22

Fix small typo

1a1db1a

Link log.level and event.severity for Syslog

2898df2

Changelog

d3b8d1f

Fix the RFC reference, tweak description of priority.

ca0be23

webmat force-pushed the syslog-take2 branch from 87a2f22 to ca0be23 Compare August 26, 2019 15:23

ruflin approved these changes Aug 27, 2019

View reviewed changes

webmat mentioned this pull request Aug 27, 2019

[Filebeat] [SIEM] Fileset for Cisco FTD logs elastic/beats#13286

Merged

Change to facility.name and facility.code

c9a0f20

ruflin approved these changes Sep 2, 2019

View reviewed changes

Nest everything Syslog under log.*, after recent discussion.

bf04d01

webmat mentioned this pull request Sep 4, 2019

Beef up the description of the log field set. #540

Merged

Re-generate files

51dc49d

ruflin approved these changes Sep 5, 2019

View reviewed changes

willemdh mentioned this pull request Sep 5, 2019

ECS compatibility logstash-plugins/logstash-filter-syslog_pri#6

Closed

webmat mentioned this pull request Sep 23, 2019

Mapping syslog header fields to ECS #559

Closed

Mathieu Martin added 2 commits September 25, 2019 14:59

Merge branch 'master' into syslog-take2

7e2c5e1

Update to explicitly name all Syslog fields under log.

56eb862

MikePaquette approved these changes Sep 25, 2019

View reviewed changes

willemdh mentioned this pull request Sep 26, 2019

syslog PRI #36

Closed

webmat merged commit aadf8db into elastic:master Sep 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A new take on adding missing Syslog fields to ECS. #525

A new take on adding missing Syslog fields to ECS. #525

webmat commented Aug 21, 2019 •

edited

Loading

webmat commented Aug 21, 2019

ruflin commented Aug 22, 2019 •

edited

Loading

webmat commented Aug 22, 2019 •

edited

Loading

willemdh commented Aug 23, 2019 •

edited

Loading

webmat commented Aug 23, 2019 •

edited

Loading

webmat commented Aug 26, 2019

ruflin commented Aug 27, 2019

kvch commented Aug 28, 2019

willemdh commented Aug 28, 2019

ruflin commented Aug 29, 2019

webmat commented Aug 30, 2019

webmat commented Sep 4, 2019

webmat commented Sep 4, 2019

willemdh commented Sep 6, 2019 •

edited

Loading

webmat commented Sep 23, 2019

webmat commented Sep 25, 2019

MikePaquette left a comment

MikePaquette Aug 30, 2019

webmat Sep 26, 2019

A new take on adding missing Syslog fields to ECS. #525

A new take on adding missing Syslog fields to ECS. #525

Conversation

webmat commented Aug 21, 2019 • edited Loading

webmat commented Aug 21, 2019

ruflin commented Aug 22, 2019 • edited Loading

webmat commented Aug 22, 2019 • edited Loading

willemdh commented Aug 23, 2019 • edited Loading

webmat commented Aug 23, 2019 • edited Loading

webmat commented Aug 26, 2019

ruflin commented Aug 27, 2019

kvch commented Aug 28, 2019

willemdh commented Aug 28, 2019

ruflin commented Aug 29, 2019

webmat commented Aug 30, 2019

webmat commented Sep 4, 2019

webmat commented Sep 4, 2019

willemdh commented Sep 6, 2019 • edited Loading

webmat commented Sep 23, 2019

webmat commented Sep 25, 2019

MikePaquette left a comment

Choose a reason for hiding this comment

MikePaquette Aug 30, 2019

Choose a reason for hiding this comment

webmat Sep 26, 2019

Choose a reason for hiding this comment

1. Syslog event / event source specifies different severity

2. Syslog event / event source does not specify different severity

3. Non-syslog event, event source specifies severity

webmat commented Aug 21, 2019 •

edited

Loading

ruflin commented Aug 22, 2019 •

edited

Loading

webmat commented Aug 22, 2019 •

edited

Loading

willemdh commented Aug 23, 2019 •

edited

Loading

webmat commented Aug 23, 2019 •

edited

Loading

willemdh commented Sep 6, 2019 •

edited

Loading