Feature request: remote syslog standard RFC5424

[tkald]

Ability to select syslog standard RFC5424 when sending logs to remote syslog server.

[fichtner]

Just for clarification: that exactly isn't standards-compliant with the current remote syslog system?

[tkald]

To begin with - "hostname" is missing

[fichtner]

Hi tkald,

That's a start. :)

"Remote Logging Options" has been in its current form since ages. A revamp would benefit everybody and consistency. We should do it.

Sketching out the required steps in this ticket will accelerate completion, especially when knowledge can indeed be shared.

Cheers,
Franco

[tkald]

Also log levels are off for some data - for example message:
php: rc.filter_synchronize: Filter sync successfully completed with https://10.169.124.3:443
shows level 3 (error)
...
And some exessive repeated messages - if gui shows 2 messeges, I may get 2000 messages to syslog server for same time period.

[AdSchellevis]

timeout due to inactivity

[fichtner]

Apologies, this wasn’t properly assigned yet. We are doing some reworks in this area with syslog-ng for 18.1

[abraxxa]

19.1 is here but the syslog format still doesn't conform to either of the two RFCs, was syslog-ng postponed again?

[fichtner]

Yes.

[abraxxa]

I'd like to work on OPNsense' logging feature including Suricata JSON logs, where should I start? Is there a roadmap what should be supported and how?

[AdSchellevis]

@abraxxa most ship eve logs using beats, there are constraints in what's possible while keeping core functionality work. When adding improvements, keep the steps small and readable, that helps us validating and increases the chances of being accepted. There where some plans switching syslog, but so far

Best first step would probably be to add a new ticket describing goals and plans how to get there, replacing syslog affects small installs too, which kind of depend on clog (rotating logs) to keep the memory usage manageable.

[fichtner]

The goals so far:

• Move remote logging to syslog-ng and keep syslogd in place (fixes all reported issues with remote log problems)
• Make syslogd clog optional with migration paths
• Maybe some day remove syslogd, or simply remove clog from syslogd

[fichtner]

@AdSchellevis worked on this

[abraxxa]

In which version is it supported?

[fichtner]

We try to add the milestone as a general orientation.

[ypid]

I just checked out the logging changes of 19.7, nice work! As far as I can see, at least in the GUI, there is no option to change the log format to RFC5424. I think that the logging format could still be improved. @fichtner Can you maybe reopen?

Logging is historically only very loosely defined which will become a major challenge for someone trying to make sense out of the logs in an automated way. It is best practice for any IT environment to archive security-related logs. In bigger organizations you will additionally often find automated log processing, indexed logs for searching, reports and alerting. For this, the key aspect is structured logging. Unfortunately there are a number of log formats. I took some time to go through them (ref: https://github.com/geberit/elastic-helpers/blob/master/Logging%20best%20practices.md). Bottom line: RFC5424 would be my recommendation for OPNsense, closely followed by the Elastic Common Schema (ECS, ref: https://github.com/elastic/ecs). It also seems that FreeBSD is moving in that direction rsyslog/rsyslog#3316. For the filterlog that would mean providing all details not as csv but as key value pairs as documented by RFC5424. The filter log would also be the most interesting I would say.

@abraxxa Ref: #3505

@AdSchellevis

most ship eve logs using beats

In my world that would be Elastic Beats but I guess you mean something else?

CC: @fabianfrz You could also be interested in this.

My background: In my day job, I am quite involved with logging using the Elastic Stack. I use OPNsense currently only in my spare time and love it.