Skip to content

Latest commit

 

History

History
375 lines (243 loc) · 15.7 KB

README-RSYSLOG.md

File metadata and controls

375 lines (243 loc) · 15.7 KB
Raw

FortiDragon (fortinet-2-elasticsearch) 🐉

logo

Engage

Join our community on Discord 🤓. Feel free to ask about anything on the channel.

You are already saving a lot of money by using Fortinet+Elastic, so consider making a contribution to the project. 💰💰💰 (It is NOT a must for asking for help)

FortiDragon 🆚 Filebeat

So you want to take you Fortinet logs to Elasticseach??? You have come to the right place!!! 👍

But wait! Doesn't Elastic provide a Filebeat module for Fortinet??? Why should you go with all the logstash hassle?? 🤷

Well, Filebeat module and Fortidragon are like cousins 👪. The logic for Filebeat module for Fortigate was based on FortiDragon, we colaborated together with Elastic when they made it.

The main differences would be

Category FortiDragon Filebeat
Dashboard We got super cool dashboards!!! None yet 😢
Updates Much more often Dependant to Elastic releases
Installation Harder Easier

The real reason behind is that we use FortiDragon on our day to day operations for threat hunting, so updates and constant evolution is more fluid.

TL;DR

Let's get this party on!!! 🤩

Prerequisites

  • Working Rsyslog-Server
  • Elastic-Search Instance
  • Kibana Instance

On Rsyslog-Server

  1. Create a Rsyslog-Rule that catches the Fortiweb/Fortigate Logs, in our case the config looks like this:

if not ($fromhost contains "monitor1" ) then ?remote-incoming-logs
& ~
else

"monitor1" is our rsyslog servers name, so all incoming logs that don't match its name are written in to /var/log/remotelogs/%HOSTNAME%/%PROGRAMNAME%.log which in case of fortiweb results in a file like /var/log/remotelogs/fortiweb_ip/filename

omelasticsearch

Install omelasticsearch https://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html on Ubuntu the package is called rsyslog-elasticsearch so install it using apt:

apt install rsyslog-elasticsearch

add a rsyslog rule for omelasticsearch like this

action(type="mmjsonparse")

# this is for index names to be like: rsyslog-YYYY.MM.DD
template(name="logstash-index" type="string" string="rsyslog-%$YEAR%.%$MONTH%.%$DAY%")

# this is for formatting our syslog in JSON with @timestamp
template(name="json-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")

      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}

# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch" template="json-syslog" searchIndex="logs-fortinet.fortigate" dynSearchIndex="on" server="127.0.0.1:9200" errorFile="/var/log/omelasticsearchmoduleerror.log" action.resumeretrycount="-1" )

On Fortigate

  1. Configure syslog
    config log syslogd setting
        set status enable
        set server "rsyslog-server-ip"
        set port 514
    end
  1. Extendend logging on webfilter OPTIONAL
    config webfilter profile
        edit "test-webfilter"
            set extended-log enable
            set web-extended-all-action-log enable
        next
    end

No need for syslogd on mode reliable, at least on v6.2 and v.6.4

On Kibana

  1. Load ingest pipeline
PUT _ingest/pipeline/add_event_ingested
{
  "processors": [
    {
      "set": {
        "field": "event.ingested",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}
  1. Create ILM policies according to your needs. You can use these examples. Make sure you name them:
  • logs-fortinet.fortigate.traffic
  • logs-fortinet.fortigate.utm
  • logs-fortinet.fortigate.event

In our experience, type=traffic generates lots of logs, while type=event very few. So it makes sense to have different lifecycles for differente types of logs. Other slicing ideas can be found below.

  1. Load component templates. You can use this script or do it manually one by one:
PUT _component_template/ecs-base
{
  "_meta": {
    "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-base.html",
    "ecs_version": "8.3.1"
  },
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "labels": {
          "type": "object"
        },
        "message": {
          "type": "match_only_text"
        },
        "tags": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    }
  }
}

  1. Load index templates

  2. Load Dashboards: Go to Management --> Stack Management --> Saved Objects --> Import

  3. Enable dashboard controls: Go to Management --> Kibana Advanced Settings --> Presentation Labs --> Enable dashboard controls

Pipelines sequence

The overall pipeline flow is as follows:

graph LR;
    Input-->kv;
    kv-->fortimail_2_ecs;
    kv-->forticlient_2_ecs;
    kv-->fortigate_2_ecs;
    kv-->fortisandbox_2_ecs;
    kv-->fortiweb_2_ecs;
    forticlient_2_ecs-->common_ecs;
    fortimail_2_ecs-->common_ecs;
    fortigate_2_ecs-->common_ecs;
    fortisandbox_2_ecs-->common_ecs;
    fortiweb_2_ecs-->common_ecs;
    common_ecs-->output;
Loading

Input Syslog

Just receives syslog logs and populates event.module and event.dataset fields depending on udp port. You can also uncomment Fortianalyzer tags is you are using it for syslog forwarding. Fortianalyzer stamps its own date format to the log, so it needs to be treated different on next pipeline.

KV Syslog

Splits the original log into key-value pairs and sets the timestamp. Timezone is also obtained from the log itself if FortiOS v6.2+.

Fortigate 2 ECS

  • Validates nulls on IP fields (Fortinet loves to fill with "N/A" null fields, which turns into ingestion errors if your field has IP mapping).
  • Renames Fortigate fields that overlaps with ECS.
  • Translates Fortigate field to ECS by type of logs.
  • Introduces network.protocol_category used on dashboards controls.
  • Populates other ECS fields based on ECS recommendations, like event.kind, event.category, event.type.

Common ECS

Populates several ECS fields based on other present fields.

  • *.locality for private, loopback, link-local, multicast and public addresses. These fields and not ECS official fields.
  • Geo localitzation of IPs. 🌎
  • related.ip and related.user.
  • network.bytes and network.packets.
  • event.duration. ⌛
  • event.hour_of_day and event.day_of_week. These fields and not ECS official fields.
  • Calcualtes network.community_id just for tcp/udp.
  • Registered domain.
  • Url parsing.
  • user_agent.*.
  • network.transport.

Output

This is crucial for index strategy 🤯. On Fortigate datastreams are split by type.

In our experience, type=traffic generates lots of logs, while type=event very few. Even inside type=traffic you might have that most of your logs have action=denied, so you may want to split them even further. Splitting into several datastreams allows to assign different ILMs policies and also will be faster for searching.

Datasets and ECS Translations

Datasets

We need your help for getting the datasets for versions 6.2 and forward. Currently we only got this. Fortinet fields are always "evolving" (changing witout any notice and logic), and not all changes get docummented on Log Reference releases. Any help on taking a Log Refence file and transform it into a csv will be welcome. 🆘

Translations Sheets

Once we got the Log Reference guides turned into spreadsheets we could process the data. We had to denormalize data, merge fields, verify fields mapping (data type), look for fileds that overlap with ECS fields, translate fields to ECS, make mappings and pipelines configs.

We plan to consolidate datasets per major version in a single spreadsheet.

Fortigate

FortiOS_6.2.X_Log_Reference - Public

Fortigate logs are an ugly beast, mainly because its lack of (good) documentation. Altough it has been improving, it is still far from being coherent. For example, starting from 6.2.1, type "utm" was documented, altough it existed long ago.

On top of that, GTP events cause some field mismatch mapping like:

  • checksum: string | uint32

  • from: ip | string

  • to: ip | string

  • version: string | uint32

As far as we are concern, GTP is only part of Fortigate Carrier, which is a different product (¿?) How can Fortigate manage a field that has 2 different data types in its internal relational database? how does fortianalyzer do it? We have no idea because we have never seen GTP events on a real scenario. In order to avoid any data type mismatch, GTP events are not going to be considered, and unless you use Fortigate Carrier, you are not going to see them either.

The spreadsheet goes like:

  1. Data 6.2.X is the denormalized data set obtained from the Log Reference Guide of version 6.2.X. This is the raw dataset of every version.

  2. Data has all the datasets combined from Data 6.2.X sheets. You can look at it as the denormalize version of all datasets of major release 6.2.

  3. On Overlap ECS - Summary of fields, we look for any field on Fortigate dataset that could overlap with ECS fields. First we consolidate all fields with a dynamic table, and then lookup for it over root fields on ECS 1.X. For example, Fortigate dataset has field agent, which overlaps with field agent on ECS. If we find an overlap, we propose a rename for Fortigate field: fortios.agent.

  4. We have decided to slice the full dataset by type, resulting in 3 datasets: traffic, utm and event. Each of them has its own translation. So, on sheets Summary of "traffic type" fields, Summary of "utm type" fields and Summary of "event type" fields we consolidate the fields of each dataset independently.

  5. On ECS Translation of "XXX type" fields is where the magic happens. Based on our criteria, we translate to ECS the fields we consider can fit.

  6. On logstash - XXX we consolidate the translated fields of previous sheets and generate logstash code.

  7. On fortigate mapping we filter all fortigate fields that are not string and. Based on its type, mapping is generated. The template we use consider keyword as default mapping, this is why we only explicitly define non-keyword fields.

Translation is where we need more help from the community!!! Any suggestions are welcome.

Fortisandbox

FortiSandbox - Log Reference v3.1.2 - Public

Not updated in a while 😕

Fortiweb

FortiWeb_6.2.0_Log_Reference - Public

Not updated in a while 😕

Dashboards

We have tried to follow Fortigate´s Logs & Report section. Main objective of these dashboards is to analyze key KPIs in order to spot anomalies on it.

We have migrated eveythigh to Lens now, so that has helped a lot on performance, but still it is very recommended to fine tune the dashboards with the relevant info to your needs. There a lot of visualizations on each dashboard so keep in mind performance can be impacted (loading times).

Structure

All dashboards are connected via its header structure. Making it easy to navigate trough them.

header

Dashboards follow a (max) 3 layer structure, going from more general to more specific.

  1. Top level reference Fortinet´s type field: traffic, utm or event. UTM is already disaggregated so it can be easier to go to an specif UTM type, just like in Fortigate´s Logs & Report section.

  2. Second level dives into traffic direction (if possible). For example: on traffic dashboard, we have Outbound | Inbound | LAN 2 LAN. It makes total sense to analyze it separetly.

  3. Third level refers to which metric are we using for exploring the dataset: we only use sessions and bytes.

  • sessions: we consider each log as a unique session.

  • bytes: we analyze source.bytes and destination.bytes by both sum and average.

We need to filter out logid=20, so we dont get duplicate data when running aggregations.

You can do it on the firewall itself

config log syslogd filter
        set filter "logid(00020)"
        set filter-type exclude
   end

We have filter it out on logstash.

You could also do it on Kibana itself.

  1. Controls, above header structure, let you quickly filter your data.

Visualizations

Dashboards have 2 sections, the upper visualizations are specific fields for the dataset that it is been analyzed on the dashboard: on webfilter we would see catdesc and url.domain for example. The lower visualizations are entity specific, on the first row there will always be source.ip, destination.ip, network.protocol which are fields that are present on all logs. The second raw has entities that might be useful on the analysis of that specific dashboard.

Limitations

We have not tested it yet on FortiOS v7+

Roadmap 🛣️

  • ECS compatibility for pipelines.
  • More dashboards: SD-WAN, traffic shapping, consolidated risk-score, etc.
  • Canvas for reports and C-level presentations.
  • Explore other ingest options: Kafka and Rsyslog
  • Explore other visualization alternatives: Grafana
  • IoC enrichment. IoC in general has 2 sides: enriching a log that is currently being ingested, and enriching a log that has already been ingested. Both approachs are needed, and both have very different challenges.
  • Transforms: go from logs to Key Security Indicators (KSI) 💡💡💡

Authors

Logstash pipelines and Elasticsearch config @hoat23 and @enotspe 🐉

Dataset analysis and Kibana @enotspe 🐉

Current maintenance @enotspe 🐉