Add Suricata module to Filebeat #8693

andrewkroh · 2018-10-23T11:26:10Z

This adds a Filebeat module for ingesting the logs created by Suricata IDS/IPS/NSM. The module collects the logs from the Suricata Eve JSON output.

#8153

This is the code as of commit #c346ff7 over [there](https://github.com/elastic/filebeat-module-suricata/pull/1)

See #7251 and #7224 for more details.

This patch updates the packaging scripts to copy modules from x-pack/filebeat into Elastic-licensed packages.

* Vendorize goimports (#8619) * Fix make fmt after vendoring goimports (#8623)

This updates the suricata module: - Added fields from eve.json - Copy fields of interest into ECS equivalent - Updated dashboards

This PR adds an intermediate solution for packaging X-Pack modules with Filebeat. In this PR the dashboards, configuration files, and fields are generated in x-pack/filebeat. Packaging is still done entirely from the OSS filebeat directory by making the build run `mage update` in x-pack/filebeat then customizing the packaging configuration to point to different dashboards, config, and fields.yml for the Elastic licensed packages. Long term we will build, test, and package the OSS and Elastic licensed Beats from their respective directories, but this gives us a smaller step in order to be able to release the X-Pack content before the build system is fully transitioned. Co-authored-by: Adrian Serrano <adrisr83@gmail.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Mostly making all vis and searches end in [Suricata]. Part of #8153.

The ingest pipeline for suricata's eve fileset uses the user-agent plugin. This updates the manifest to include this requirement.

Added Filebeat module test cases for suricata/eve. To support to running Filebeat module tests from x-pack/filebeat an env var MODULES_PATH was added to allow the existing tests to run against a different module directory than the OSS module dir. Added some missing fields to pass validation. Added sample eve.json files for validation.

andrewkroh

~~Needs a changelog.~~

libbeat/scripts/Makefile

andrewkroh · 2018-10-23T11:51:15Z

x-pack/filebeat/module/suricata/_meta/docs.asciidoc

+
+This module comes with a sample dashboard. For example:
+
+TODO: include an image of a sample dashboard


The docs still have some work to be done. But we can update them in a separate PR. From what I see they are not yet included in the Filebeat docs tree.

@dedemorton This might be the first documentation for an X-Pack only module. Should I simply link it into the filebeat/docs/module_list.asciidoc?

@andrewkroh You should use the x-pack role to flag X-pack features.

[role="xpack"]

Here's an example where we use the flag: https://github.com/elastic/beats/edit/6.4/libbeat/docs/security/securing-beats.asciidoc.

AFAIK modules_list.asciidoc is a generated file, so your module should get added when you run make update.

Thanks. That works. I now see the tag.

andrewkroh · 2018-10-23T13:59:17Z

CI is green for this, but there is a Jenkins issue with archiving artifacts from the Filebeat build. I opened a ticket to get some help debugging this on Jenkins.

ruflin

I mainly skimmed through the changes especially didn't check the instrumentation stuff in detail. Overall LGTM.

We should add this to our manual testing checklist.

ruflin · 2018-10-23T19:10:57Z

CHANGELOG-developer.asciidoc

@@ -54,3 +55,9 @@ The list below covers the major changes between 6.3.0 and master only.
 - Set current year in generator templates. {pull}8396[8396]
 - You can now override default settings of libbeat by using instance.Settings. {pull}8449[8449]
 - Add `-space-id` option to `export_dashboards.go` script to support Kibana Spaces {pull}7942[7942]
+- Add `-name` option to `asset.go` script to explicitly name the asset rather than using its filename. {pull}8693[8693]


Was at first surprised by this long dev changelog but it seems all of these actually go into this PR.

ruflin · 2018-10-23T19:20:39Z

x-pack/filebeat/filebeat.reference.yml

+    #var.paths:
+
+#-------------------------------- Haproxy Module --------------------------------
+- module: haproxy


Not related to this PR but I realised we have a bit of inconsistency here related to what we comment out and what not across modules.

ruflin · 2018-10-23T19:21:25Z

x-pack/filebeat/module/suricata/README.md

@@ -0,0 +1,60 @@
+# Suricata module
+
+Shove the content of this repo under `filebeats/module/suricata` for testing,


Still needed?

I'll clean this up along with the documentation in a separate PR.

I cleaned up this file now. But perhaps you're questioning if we should remove this file entirely?

Yeah I put this file in there as a convenience for the handoff. I tend to think that having a bit of a readme this deep down in the module can be helpful to people who will collaborate on this in the future, while not being in the way otherwise. I would leave it in there after a cleanup. I'd probably remove the whole "Caveats" section but leave the getting started tips.

ruflin · 2018-10-23T19:23:26Z

x-pack/filebeat/module/suricata/eve/ingest/pipeline.json

+    }
+  , {"convert":
+      {"field": "suricata.eve.http.hostname"
+      ,"target_field": "url.host.name"


This will be url.hostname in the next days in ECS. @webmat Please correct me if I'm wrong.

We can still adjust this a bug fix if needed.

It's imminent indeed: elastic/ecs#147

webmat

LGTM

Jenkins is failing during artifact archival. This might fix it.

Run make fix-permissions on the x-pack/filebeat after executing tests.

The problem was the ownership of files rather than the symlinks.

This adds a Filebeat module for ingesting the logs created by Suricata IDS/IPS/NSM. The module collects the logs from the Suricata Eve JSON output (https://suricata.readthedocs.io/en/latest/output/eve/eve-json-format.html). elastic#8153 The module is included in the Elastic licensed Filebeat package. It is considered beta at this stage. It includes two sample dashboards. It uses Elastic Common Schema (ECS) for field naming were applicable, but it has to work-around the conflict with the existing source field in Filebeat. source_ecs holds the data that goes into the ECS source field. This will be rectified in the next major release when we can make a breaking change. The development tooling for the building/testing/packaging of x-pack modules is still a bit of a WIP. So at the moment testing and packaging continues to happen through the OSS filebeat directory. Co-authored-by: Adrian Serrano <adrisr83@gmail.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> Co-authored-by: Tudor Golubenco <tudor@elastic.co> (cherry picked from commit 3e1b03e)

* Add Suricata module to Filebeat (#8693) This adds a Filebeat module for ingesting the logs created by Suricata IDS/IPS/NSM. The module collects the logs from the Suricata Eve JSON output (https://suricata.readthedocs.io/en/latest/output/eve/eve-json-format.html). #8153 The module is included in the Elastic licensed Filebeat package. It is considered beta at this stage. It includes two sample dashboards. It uses Elastic Common Schema (ECS) for field naming were applicable, but it has to work-around the conflict with the existing source field in Filebeat. source_ecs holds the data that goes into the ECS source field. This will be rectified in the next major release when we can make a breaking change. The development tooling for the building/testing/packaging of x-pack modules is still a bit of a WIP. So at the moment testing and packaging continues to happen through the OSS filebeat directory. Co-authored-by: Adrian Serrano <adrisr83@gmail.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> Co-authored-by: Mathieu Martin <mathieu.martin@elastic.co> Co-authored-by: Tudor Golubenco <tudor@elastic.co> (cherry picked from commit 3e1b03e) * Change url.host.name to url.hostname (#8732) This update the Filebeat Suricata module to use url.hostname instead of url.host.name. * Add fields used by Suricata module Add fields used by Suricata module to fields.yml. Some of these are in ECS. event.type destination.ip destination.port user_agent.original user_agent.device user_agent.version user_agent.major user_agent.minor user_agent.patch user_agent.name user_agent.os.name user_agent.os.full_name (non-ECS) user_agent.os.version user_agent.os.major user_agent.os.minor file.path file.size

Mathieu Martin and others added 14 commits October 2, 2018 21:09

Import suricata module from temp repo

33e7c59

This is the code as of commit #c346ff7 over [there](https://github.com/elastic/filebeat-module-suricata/pull/1)

Generated doc (no actual documentation for it the module yet)

a4da08b

Update the kibana objects to the new format.

3cfcc11

See #7251 and #7224 for more details.

Remove symlink to Suricata module and re-generate the doc

9652a66

Add a simple compatibility comment in the doc.

167aacf

Ooops, off by 3 error. Version 4.0.4 :-)

a1d05e0

Package filebeat x-pack modules in the non-oss distribution

2ba9706

This patch updates the packaging scripts to copy modules from x-pack/filebeat into Elastic-licensed packages.

Add modules.d to filebeat's x-pack tree

3ab125f

Fix build in feature-suricata branch (#8625)

0211eda

* Vendorize goimports (#8619) * Fix make fmt after vendoring goimports (#8623)

[Suricata] Update fields and paths (#8550)

4d60173

This updates the suricata module: - Added fields from eve.json - Copy fields of interest into ECS equivalent - Updated dashboards

Use naming conventions for the Suricata dashboards (#8675)

2430239

Mostly making all vis and searches end in [Suricata]. Part of #8153.

Add ingest-user-agent to the required plugins list (#8674)

c8d1ab7

The ingest pipeline for suricata's eve fileset uses the user-agent plugin. This updates the manifest to include this requirement.

andrewkroh added the Filebeat Filebeat label Oct 23, 2018

andrewkroh added 2 commits October 23, 2018 07:39

Merge branch 'master' of github.com:elastic/beats into feature-suricata

3e2f40b

Format magefile.go for x-pack/filebeat

576bb2d

andrewkroh commented Oct 23, 2018

View reviewed changes

andrewkroh added 3 commits October 23, 2018 07:52

rely on go to resolve beats repo for goimports install

563a289

Update CHANGELOG files

8033e99

Merge branch 'master' of github.com:elastic/beats into feature-suricata

3a5ef22

andrewkroh added the review label Oct 23, 2018

andrewkroh requested review from tsg and ruflin October 23, 2018 13:22

ruflin reviewed Oct 23, 2018

View reviewed changes

andrewkroh added 4 commits October 23, 2018 19:18

Clean-up module documentation

0f33ff3

Update config file

ed53c42

Fix xpack role on docs

96bc0f7

Merge branch 'master' of github.com:elastic/beats into feature-suricata

73936dd

Fix heartbeat test

151572b

webmat approved these changes Oct 24, 2018

View reviewed changes

andrewkroh added 4 commits October 23, 2018 21:56

Remove system-test symlinks

c239c8d

Jenkins is failing during artifact archival. This might fix it.

Remove system-test symlinks - take 2

b808dc0

Jenkins is failing during artifact archival. This might fix it.

Fix permissions

ab7266c

Run make fix-permissions on the x-pack/filebeat after executing tests.

Undo symlink deletion

9b4f8e8

The problem was the ownership of files rather than the symlinks.

andrewkroh merged commit 3e1b03e into master Oct 24, 2018

andrewkroh mentioned this pull request Oct 24, 2018

Cherry-pick #8693 to 6.x: Add Suricata module to Filebeat #8728

Merged

andrewkroh added the v6.5.0 label Oct 24, 2018

ph mentioned this pull request Oct 24, 2018

Fix an issue where we were trying to use a non existing directory when generating the fields.go #8729

Merged

andrewkroh added the v7.0.0 label Nov 1, 2018

andrewkroh deleted the feature-suricata branch February 1, 2019 21:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Suricata module to Filebeat #8693

Add Suricata module to Filebeat #8693

andrewkroh commented Oct 23, 2018 •

edited

andrewkroh left a comment •

edited

andrewkroh Oct 23, 2018

andrewkroh Oct 23, 2018 •

edited

dedemorton Oct 23, 2018

andrewkroh Oct 23, 2018

andrewkroh commented Oct 23, 2018

ruflin left a comment

ruflin Oct 23, 2018

ruflin Oct 23, 2018

ruflin Oct 23, 2018

andrewkroh Oct 23, 2018

andrewkroh Oct 23, 2018

webmat Oct 24, 2018

ruflin Oct 23, 2018

webmat Oct 24, 2018

webmat left a comment


		This module comes with a sample dashboard. For example:

		TODO: include an image of a sample dashboard

		@@ -0,0 +1,60 @@
		# Suricata module

		Shove the content of this repo under `filebeats/module/suricata` for testing,

Add Suricata module to Filebeat #8693

Add Suricata module to Filebeat #8693

Conversation

andrewkroh commented Oct 23, 2018 • edited

andrewkroh left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Oct 23, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Oct 23, 2018

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

andrewkroh commented Oct 23, 2018 •

edited

andrewkroh left a comment •

edited

andrewkroh Oct 23, 2018 •

edited