Fotigate not decoding becasue of quotation marks.

[MalfuncEddie]

Hi,

My Fortigate is not decoding due to the devname and other fields are quoted. Any way to fix this? Or am I the only one that has this issue?

date=2016-06-16 time=16:22:34 devname=Mobipay_Firewall devid=FGTXXXX9999999999 logid=0100032001 type=event subtype=system level=information vd="root" logdesc="Admin login successful" sn=1466090554 user="a@b.com.na" ui=https(10.42.8.253) action=login status=success reason=none profile="super_admin" msg="Administrator a@b.com.na logged in successfully from https(10.42.8.253)"

date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"

[MalfuncEddie]

Seems fortigate changed it log format again:
https://docs.fortinet.com/uploaded/files/3610/FortiOS-5.6.0-Log-Reference.pdf -> page 7
What's new
This section identifies major changes in the FortiOS Log Reference from version 5.6.0 and later.
FortiOS 5.6.0
DNS
l DNS was added as a new log type with a log type ID of 15.
CEF
l CEF (Common Event Format) is supported when sending logs to remote syslog servers.
Log format changes
l All field values are escaped with double quotes, except devname and devid.

[fnuzon]

I have the same issue also.

EDIT: I solved the issue by adding the quotes

decoder name="fortigate-firewall-v5">
date=\S+ time=.+ devname="(\S+)" devid="(FG\w+)" logid="(\d+)"
syslog
/decoder>

and now the logtest decodes it

**Phase 2: Completed decoding.
decoder: 'fortigate-firewall-v5'

**Phase 3: Completed filtering (rules).
Rule id: '81603'
Level: '0'
Description: 'Fortigate messages grouped.'

But now my problem is that the logs aren't shown in kibana discover anymore
They came just as full log from ip address before it couldn't handle the decode

[SitoRBJ]

Hello MalfuncEddie and fnuzon,

First of all, we regret the delay in attending to this issue. In fact, you are receiving an event with a format that the decoders do not recognize.

The solution proposed by fnuzon is fine, because as we can see, the decoder recognizes the event that MalfuncEddie has given as an example and generates a "generic" alert, in this case, the alert id is: 81603, its alert level is: 0, and its description is: 'Fortigate messages grouped.'

date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"


**Phase 1: Completed pre-decoding.
       full event: 'date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"'
       timestamp: '(null)'
       hostname: 'manager'
       program_name: '(null)'
       log: 'date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"'

**Phase 2: Completed decoding.
       decoder: 'fortigate-firewall-v5'

**Phase 3: Completed filtering (rules).
       Rule id: '81603'
       Level: '0'
       Description: 'Fortigate messages grouped.'

But I think this event should "go further" and generate a similar alert to the next one:

date=2016-06-16 time=08:48:28 devname=Device_Name devid=FGTXXXX9999999999 logid=0100032003 type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"


**Phase 1: Completed pre-decoding.
       full event: 'date=2016-06-16 time=08:48:28 devname=Device_Name devid=FGTXXXX9999999999 logid=0100032003 type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"'
       timestamp: '(null)'
       hostname: 'manager'
       program_name: '(null)'
       log: 'date=2016-06-16 time=08:48:28 devname=Device_Name devid=FGTXXXX9999999999 logid=0100032003 type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"'

**Phase 2: Completed decoding.
       decoder: 'fortigate-firewall-v5'
       srcuser: 'a@b.com.na'
       srcip: '4.3.5.253'
       action: 'logout'
       status: 'success'
       extra_data: '"Administrator a@b.com.na logged out from https(2.3.8.1)"'

**Phase 3: Completed filtering (rules).
       Rule id: '81616'
       Level: '4'
       Description: 'Fortigate: User logout successful'
**Alert to be generated.

This is because the event has a different format making it not a match for the regular expressions on the decoders. In this particular case we would have the following situation:

The decoder that doesn't match:

<decoder name="fortigate-firewall-v5-event-system-information">
    <parent>fortigate-firewall-v5</parent>
    <prematch offset="after_parent">type=event subtype=system level=information</prematch>
    <regex offset="after_parent">user="(\S+)" ui=\p*\w+\((\S+)\)\p* action=(\S+) </regex>
    <order>srcuser,srcip,action</order>
</decoder>

Receives in each case the following regular expressions:

Old event:

type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"

New event:

logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"

And start searching for matches from the user field:

<regex offset="after_parent">user="(\S+)" ui=\p*\w+\((\S+)\)\p* action=(\S+) </regex>

user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"

user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"

As we can see, the regular expression changes from the "ui" field, since in the old one the "action" field follows and in the new one the "method" field follows.

From here the decoders have no effect and therefore the alert generated is not the expected one. This also explains why the alerts are not displayed in Kibana. Kibana displays the alerts contained in the file alerts.json", but level 0 alerts are not sent to alerts.json and therefore Kibana cannot show them.

Remember that the alert generated by the change in the decoder is level 0 (As we can see, we do not get the message "**Alert to be generated." underneath the message of phase 3).

**Phase 3: Completed filtering (rules).
       Rule id: '81603'
       Level: '0'
       Description: 'Fortigate messages grouped.'

Thank you very much for reporting this problem, we will modify the necessary decoders so that they can generate the appropriate alerts.

If you have any questions, do not hesitate to contact us.

Kind regards,

Alfonso.

[SitoRBJ]

Hello again,

Until we make the changes effective, you can modify the following decoders to follow the event as desired.

File: decoders\0100-fortigate_decoders.xml

<decoder name="fortigate-firewall-v5">
    <prematch>date=\S+ time=\.+ devname=\S+ devid=FG\w+ logid=\d+ |date=\S+ time=\.+ devname="\S+" devid="FG\w+" logid="\d+" </prematch>
    <type>syslog</type>
</decoder>

. . .

<decoder name="fortigate-firewall-v5-event-system-information">
    <parent>fortigate-firewall-v5</parent>
    <prematch offset="after_parent">type=event subtype=system level=information|type="event" subtype="system" level="information"</prematch>
    <regex offset="after_parent">user="(\S+)" ui=\p*\w+\((\S+)\)\p* action=(\S+) |user="(\S+)" ui=\p*\w+\((\S+)\)\p* \.*action="(\S+)" </regex>
    <order>srcuser,srcip,action</order>
</decoder>

. . .

<decoder name="fortigate-firewall-v5-event-system-information">
    <parent>fortigate-firewall-v5</parent>
    <regex offset="after_regex">status=(\S+) \.*msg=(\.*)|status="(\S+)" \.*msg=(\.*)</regex>
    <order>status,extra_data</order>
</decoder>

If we receive an event with the old format or an event with the new format, we will get the following results:

Old format:

date=2016-06-16 time=08:48:28 devname=Device_Name devid=FGTXXXX9999999999 logid=0100032003 type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"


**Phase 1: Completed pre-decoding.
       full event: 'date=2016-06-16 time=08:48:28 devname=Device_Name devid=FGTXXXX9999999999 logid=0100032003 type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"'
       timestamp: '(null)'
       hostname: 'manager'
       program_name: '(null)'
       log: 'date=2016-06-16 time=08:48:28 devname=Device_Name devid=FGTXXXX9999999999 logid=0100032003 type=event subtype=system level=information vd="root" logdesc="Admin logout successful" sn=1466062693 user="a@b.com.na" ui=https(4.3.5.253) action=logout status=success duration=615 state="Config-Changed" reason=exit msg="Administrator a@b.com.na logged out from https(2.3.8.1)"'

**Phase 2: Completed decoding.
       decoder: 'fortigate-firewall-v5'
       srcuser: 'a@b.com.na'
       srcip: '4.3.5.253'
       action: 'logout'
       status: 'success'
       extra_data: '"Administrator a@b.com.na logged out from https(2.3.8.1)"'

**Phase 3: Completed filtering (rules).
       Rule id: '81616'
       Level: '4'
       Description: 'Fortigate: User logout successful'
**Alert to be generated.

New format:

date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"


**Phase 1: Completed pre-decoding.
       full event: 'date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"'
       timestamp: '(null)'
       hostname: 'manager'
       program_name: '(null)'
       log: 'date=2018-05-19 time=11:58:25 devname="xxxxxxxxx-A" devid="FGxxxxxxxxxxxxxx" logid="0100032003" type="event" subtype="system" level="information" vd="root" eventtime=1526723905 logdesc="Admin logout successful" sn="xxxxxxxxxx" user="admin" ui="ssh(xxx.xxx.xxx.xxx)" method="ssh" srcip=xxx.xxx.xxx.xxx dstip=xxx.xxx.xxx.xxx action="logout" status="success" duration=1 reason="exit" msg="Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"'

**Phase 2: Completed decoding.
       decoder: 'fortigate-firewall-v5'
       srcuser: 'admin'
       srcip: 'xxx.xxx.xxx.xxx'
       action: 'logout'
       status: '"success"'
       extra_data: '"Administrator admin logged out from ssh(xxx.xxx.xxx.xxx)"'

**Phase 3: Completed filtering (rules).
       Rule id: '81616'
       Level: '4'
       Description: 'Fortigate: User logout successful'
**Alert to be generated.

We hope this information will be helpful and if you have more events that do not generate the corresponding alerts as they should, do not hesitate to let us know.

Kind regards,

Alfonso.

[fnuzon]

Hey! Thanks for your reply. I have few more events that do not generate the corresponding alerts.

2018 Jun 21 00:00:35 XXX->127.0.0.1 date=2018-06-21 time=03:00:35 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcport=11111 srcintf="port111" srcintfrole="undefined" dstip=127.0.0.1 dstport=111 dstintf="port11" dstintfrole="undefined" poluuid="111-111-111-111" sessionid=11111111111 proto=1 action="client-rst" policyid=111 policytype="policy" service="XXXX" dstcountry="xxxxx" srccountry="Reserved" trandisp="snat" transip=127.0.0.1 transport=11111 appcat="unknown" applist="xxxxx" duration=11 sentbyte=11 rcvdbyte=11 sentpkt=11

2018 Jun 21 00:00:35 XXX->127.0.0.1 date=2018-06-21 time=03:00:35 devname="xxx" devid="FG123341414414" logid="111111111" type="utm" subtype="webfilter" eventtype="ftgd_allow" level="notice" vd="xxx" eventtime=111111111 policyid=111 sessionid=111111111 srcip=127.0.0.1 srcport=11111 srcintf="port1" srcintfrole="undefined" dstip=127.0.0.1 dstport=111 dstintf="port111" dstintfrole="undefined" proto=1 service="XXX" hostname="xxxxx.com" profile="xx" action="passthrough" reqtype="direct" url="/xxxxxxxxxxxx" sentbyte=11 rcvdbyte=111 direction="outgoing" msg="URL belongs to an allowed category in policy" method="domain" cat=50 catdesc="Information and Computer Security"

2018 Jun 21 00:00:35 XXX->127.0.0.1 date=2018-06-21 time=03:00:35 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcport=11111 srcintf="port111" srcintfrole="undefined" dstip=127.0.0.1 dstport=1111 dstintf="xxxxx" dstintfrole="undefined" sessionid=122111221 proto=6 action="deny" policyid=1 policytype="policy" service="tcp/11111" dstcountry="xxxxx" srccountry="xxxx" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=111 craction=11111 crlevel="high"

2018 Jun 21 00:00:35 XXX->127.0.0.1 date=2018-06-21 time=03:00:35 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcport=11111 srcintf="port1111" srcintfrole="undefined" dstip=127.0.0.1 dstport=1111 dstintf="xxxxx" dstintfrole="undefined" poluuid="111-111-111-111" sessionid=1111111 proto=1 action="close" policyid=111 policytype="policy" service="tcp/111111" dstcountry="xxxxx" srccountry="Reserved" trandisp="noop" duration=11 sentbyte=111 rcvdbyte=111 sentpkt=111 rcvdpkt=111 appcat="unscanned"

2018 Jun 21 00:00:35 XXX->127.0.0.1 date=2018-06-21 time=03:00:35 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcintf="port111" srcintfrole="undefined" dstip=127.0.0.1 dstintf="port111" dstintfrole="undefined" sessionid=121212121 proto=1 action="deny" policyid=1111 policytype="policy" service="PING" dstcountry="xxxx" srccountry="Reserved" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=1111 craction=11111 crlevel="high"

2018 Jun 21 00:00:34 XXX->127.0.0.1 date=2018-06-21 time=03:00:34 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcport=11111 srcintf="xxxxxx" srcintfrole="undefined" dstip=127.0.0.1 dstport=111 dstintf="port111" dstintfrole="undefined" poluuid="111-111-111-111" sessionid=111111 proto=1 action="close" policyid=111 policytype="policy" service="XXX" dstcountry="xxxxxx" srccountry="Reserved" trandisp="snat" transip=127.0.0.1 transport=11111 appid=1111 app="xxxx" appcat="xxxxx" apprisk="medium" applist="xxxxx" duration=111 sentbyte=111 rcvdbyte=1111 sentpkt=111 rcvdpkt=1111 wanin=1111 wanout=11111 lanin=11111 lanout=1111

2018 Jun 21 00:00:34 XXX->127.0.0.1 date=2018-06-21 time=03:00:33 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcport=11111 srcintf="xxxxxxx" srcintfrole="undefined" dstip=127.0.0.1 dstport=11111 dstintf="port1111" dstintfrole="undefined" poluuid="111-1111-111-111" sessionid=111111 proto=1 action="server-rst" policyid=111 policytype="policy" service="tcp/1111" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=5 sentbyte=0 rcvdbyte=212 sentpkt=0 rcvdpkt=5 appcat="unscanned"

2018 Jun 21 00:00:34 XXX->127.0.0.1 date=2018-06-21 time=03:00:33 devname="xxx" devid="FG123341414414" logid="111111111" type="traffic" subtype="forward" level="notice" vd="xxx" eventtime=111111111 srcip=127.0.0.1 srcport=111111 srcintf="port111" srcintfrole="undefined" dstip=127.0.0.1 dstport=1111 dstintf="xxxxx" dstintfrole="undefined" sessionid=111111 proto=11 action="deny" policyid=1 policytype="policy" service="XXXX" dstcountry="xxxx" srccountry="xxxxxx" trandisp="noop" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=11 craction=111111 crlevel="high"

2018 Jun 21 00:11:50 XXX->127.0.0.1 date=2018-06-21 time=03:11:49 devname="xxx" devid="FG123341414414" logid="111111111" type="utm" subtype="virus" eventtype="analytics" level="information" vd="xxx" eventtime=111111111 srcip=127.0.0.1 dstip=127.0.0.1 srcport=1111 dstport=11 action="monitored" service="XXX" filename="xxx.xx" fsaverdict="clean" analyticscksum="19jf2oi1jfokj1iofj189fjofko3kf010fuoifjoi1f" dtype="fortisandbox"

2018 Jun 21 00:11:50 XXX->127.0.0.1 date=2018-06-21 time=03:11:49 devname="xxx" devid="FG123341414414" logid="111111111" type="utm" subtype="virus" eventtype="analytics" level="information" vd="xxx" eventtime=111111111 msg="File submitted to Sandbox." action="analytics" service="XXX" sessionid=11111111 srcip=127.0.0.1 dstip=127.0.0.1 srcport=111111 dstport=111 srcintf="port1" srcintfrole="undefined" dstintf="port1" dstintfrole="undefined" policyid=111 proto=1 direction="incoming" filename="xxx.xx" url="wwww.test.com/xxx.xx" profile="XXXX" agent="XXXX" analyticscksum="12k3jljfi1ljfo1jokfjko1ofk1jf" analyticssubmit="true"

[frgv]

Hi all!

Now our Fortigate rules and decoders supports the new format, and we've included rules for some new event types. Rules and decoders can be found on #147

Thank you very much for your feedback!

Best regards,

Fran G.

[frgv]

Also, I'll proceed to close this issue. If you find any relevant event that is still not in our ruleset, please don't hesitate to open another issue. Thank you very much for helping us to improve Wazuh!

Best regards,

Fran G.