Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Cisco and FortiGate syslog messages #3599

Merged
merged 3 commits into from Mar 30, 2017

Conversation

@joschi
Copy link
Contributor

@joschi joschi commented Mar 9, 2017

This PR adds support for (the most common) Cisco and FortiGate/FortiOS syslog messages.

This depends on merging and releasing changes in the syslog4j project: graylog-labs/syslog4j-graylog2#17

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
@joschi
Copy link
Contributor Author

@joschi joschi commented Mar 16, 2017

This should also be back-ported into the 2.2 branch once it's been reviewed and merged.

Loading

@dennisoelkers dennisoelkers added this to the 2.2.3 milestone Mar 24, 2017
@dennisoelkers dennisoelkers removed this from the 2.3.0 milestone Mar 24, 2017
@dennisoelkers dennisoelkers self-assigned this Mar 24, 2017
@joschi joschi added this to the 2.3.0 milestone Mar 30, 2017
@joschi joschi removed this from the 2.2.3 milestone Mar 30, 2017
@dennisoelkers dennisoelkers merged commit 0b94800 into master Mar 30, 2017
4 checks passed
Loading
@dennisoelkers dennisoelkers deleted the syslog-improvements branch Mar 30, 2017
joschi added a commit that referenced this issue Mar 30, 2017
* Add support for Cisco and FortiGate syslog messages

* Use non-SNAPSHOT version of syslog4j 0.9.59

* Allow use of DateTime.getDefault() in SyslogCodecTest

(cherry picked from commit 0b94800)
dennisoelkers added a commit that referenced this issue Mar 30, 2017
* Add support for Cisco and FortiGate syslog messages

* Use non-SNAPSHOT version of syslog4j 0.9.59

* Allow use of DateTime.getDefault() in SyslogCodecTest

(cherry picked from commit 0b94800)
@ysinfras
Copy link

@ysinfras ysinfras commented Apr 6, 2017

Hello,

I am afraid your Fortigate parsing is wrong, I've got several parsing exceptions on the graylog logfile since the 2.2.3 upgrade . I can't get associated messages, but for example when a url contains the string "source=..." your implementation overwrites the source attribute leading to sources like "smarttag&fired=report&confid=kvjrsgal&_kpid=2897a0e1-1f44-4f18-9361-1b730c6292bc&_kcp_s=commentcamarche&_kcp_" for the following log (I changed some values for security reason) :

"date=2017-04-06 time=30:30:30 devname=FOOBAR devid=FGSN logid=1111111111 type=utm subtype=app-ctrl eventtype=app-ctrl-all level=information vd=root appid=5060 user="" srcip=300.300.300.300 srcport=50 srcintf="if" dstip=500.500.500.500 dstport=80 dstintf="if2" profiletype="applist" proto=6 service="HTTP" policyid=222222222222 sessionid=12 applist="block" appcat="Web.Client" app="HTTP.BROWSER_Chrome" action=pass hostname="beacon.krxd.net" url="/pixel.gif?source=smarttag&fired=report&confid=KVJRsGaL&_kpid=2897a0e1-1f44-4f18-9361-1b730c6292bc&_kcp_s=CommentCaMarche&_kcp_" msg="Web.Client: HTTP.BROWSER_Chrome," apprisk=elevated.

Is there any way to deactive this autoparsing without having to re compile ?

Loading

@joschi
Copy link
Contributor Author

@joschi joschi commented Apr 6, 2017

Is there any way to deactive this autoparsing without having to re compile ?

No, but please provide some more example messages so that we can adapt and fix the Fortigate-specific parser.

As a workaround, you can use a Raw/Plaintext input in Graylog and extract the required information using extractors or processing pipeline rules.

Loading

@ysinfras
Copy link

@ysinfras ysinfras commented Apr 7, 2017

Hello !

I'm a bit confused, it looks as if the message didn't make it up to eslaticsearch. For instance, when I look on the logfile I have the following entry :

	2017-04-07T08:59:41.130+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=c5b9ca20-1b5f-11e7-b2d0-005056b8d528 ....
	java.time.format.DateTimeParseException: Text '1491548380' could not be parsed at index 2
			at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) ~[?:1.8.0_121]
			at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) ~[?:1.8.0_121]
			at java.time.LocalTime.parse(LocalTime.java:441) ~[?:1.8.0_121]
			at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parseDate(FortiGateSyslogEvent.java:90) ~[graylog.jar:?]
			at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parse(FortiGateSyslogEvent.java:58) ~[graylog.jar:?]
			at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.<init>(FortiGateSyslogEvent.java:44) ~[graylog.jar:?]
			at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:130) ~[graylog.jar:?]
			at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96) ~[graylog.jar:?]
			at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:146) ~[graylog.jar:?]
			at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:87) [graylog.jar:?]
			at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:79) [graylog.jar:?]
			at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
			at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
			at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
			at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

however when I query in Graylog : _id:c5b9ca20-1b5f-11e7-b2d0-005056b8d528 I have 0 result even when escaping _id:c5b9ca20\-1b5f\-11e7\-b2d0\-005056b8d528 . I hope this will help.

Thank you.

Loading

@joschi
Copy link
Contributor Author

@joschi joschi commented Apr 7, 2017

@ysinfras Please provide some example messages, e. g. captured with Wireshark or tcpdump, so we can tune the syslog parser. Otherwise there's nothing we can do.

Loading

@ysinfras
Copy link

@ysinfras ysinfras commented Apr 7, 2017

I'd be glad to help, but as I said I can't get any messages that raise an exception because they are simply drop due to a processing error. Can you please provide me with a method to get such messages ?

Loading

@joschi
Copy link
Contributor Author

@joschi joschi commented Apr 7, 2017

@ysinfras That's why I've asked for captures of the messages on the wire, not in Graylog.

Loading

@ysinfras
Copy link

@ysinfras ysinfras commented Apr 7, 2017

Ok I see you just edited your message, I am trying to get you an example of failling message, but I can't give you the full packet capture.

Loading

@ysinfras
Copy link

@ysinfras ysinfras commented Apr 7, 2017

@joschi Hello again,

I managed to find one of those log, here is the content of the syslog message :

5041	152.006190	5.5.5.5	7.8.8.8	Syslog	589	LOCAL7.INFO: date=2017-04-07 time=10:36:52 devname=FOOBAR devid=FGSN logid=1111111111 type=utm subtype=app-ctrl eventtype=app-ctrl-all level=information vd=root appid=125 user="" srcip=2.2.2.2 srcport=12 srcintf="sif" dstip=3.3.3.3 dstport=8 dstintf="dif" profiletype="applist" proto=6 service="HTTP" policyid=75555 sessionid=20 applist="profil" appcat="Video/Audio" app="Spotify" action=pass hostname="apresolve.spotify.com" url="/?time=1491554213" msg="Video/Audio: Spotify," apprisk=medium

It looks close to the other bug, the url contains a key value pair and the time parsing in the url fails...

Hope this will help ! Thank you for your time.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants