[Filebeat] Add Pensando DFW Module #21063

punisherVX · 2020-09-10T23:08:09Z

Enhancement

What does this PR do?

Utilized the instructions found here: https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html
This adds the Pensando distributed firewall (fileset) beat to the release.

Why is it important?

Many of our customers want an easy way to implement our FW logging in/on their Elastic instances.

Checklist

My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

All files are in the correct place (dashboards, docs, module files, etc.)

How to test this PR locally

All tests were run using these guidelines to verify logs worked correctly: https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html#_test

Related issues

None

Use cases

Screenshots

Logs

Update from original

elasticmachine · 2020-09-10T23:08:11Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-09-10T23:08:11Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-09-10T23:13:10Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: marc-gr commented: jenkins run tests
Start Time: 2021-02-15T15:52:08.727+0000
Duration: 54 min 32 sec
Commit: 494e8cc

Test stats 🧪

Test	Results
Failed	0
Passed	13041
Skipped	2047
Total	15088

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13041
Skipped	2047
Total	15088

elasticmachine · 2020-09-11T09:43:19Z

Pinging @elastic/siem (Team:SIEM)

adriansr · 2020-09-21T08:07:37Z

jenkins run the tests please

botelastic · 2020-10-21T08:36:17Z

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

punisherVX · 2020-10-21T12:21:47Z

This is still relevant (to us at least). 👍

punisherVX · 2020-11-13T17:38:53Z

This is still relevant and needs to be looked at. Thanks.

botelastic · 2020-12-13T18:15:42Z

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

punisherVX · 2020-12-14T00:03:43Z

Yep - still relevant 👍

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

marc-gr · 2021-01-12T11:02:47Z

@punisherVX could you please update your branch with master?

marc-gr

It is looking good!

After the comments I left and finishing up the docs should be in pretty good shape.

Maybe could be interesting to map some of the fields under pensando to ECS (I was thinking about filling up some network.* fields based on pensando.dfw.direction and pensando.dfw.protocol).

LMK if you need anything from our side to move this forward.

Thanks for your patience and hope to get it merged soon 😄

filebeat/module/pensando/_meta/config.yml

filebeat/module/pensando/dfw/config/dfw.yml

filebeat/module/pensando/dfw/ingest/pipeline.yml

filebeat/module/pensando/dfw/_meta/fields.yml

filebeat/module/pensando/dfw/test/test.log-expected.json

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

marc-gr

I checked it out locally and some changes are required to fix the pipeline. Once done, you will need to run:

cd filebeat
TESTING_FILEBEAT_MODULES=pensando MODULES_PATH=module GENERATE=true mage -v pythonIntegTest

to regenerate the test golden files.

Please lmk if you need any help. Thanks!!

filebeat/module/pensando/dfw/ingest/pipeline.yml

filebeat/module/pensando/_meta/kibana/7/dashboard/pensando-dfw-overview.json2

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

marc-gr · 2021-02-02T07:47:09Z

jenkins run tests

marc-gr · 2021-02-08T16:41:28Z

jenkins run tests

marc-gr

Sorry I missed that the first time!

Can you re-generate the golden files again after that change?

cd filebeat
TESTING_FILEBEAT_MODULES=pensando MODULES_PATH=module GENERATE=true mage -v pythonIntegTest

This time to double check all is good, run te tests without the GENERATE=true flag.

Thanks!!

filebeat/module/pensando/dfw/ingest/pipeline.yml

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

marc-gr · 2021-02-10T09:57:10Z

jenkins run tests

marc-gr

LGTM

andrewkroh

Just one minor comment to fix then it I think it will be good to go.

filebeat/module/pensando/dfw/_meta/fields.yml

marc-gr · 2021-02-12T08:20:31Z

Please @punisherVX commit the changes after running cd filebeat; mage fmt update

marc-gr · 2021-02-15T15:51:31Z

jenkins run tests

Changes done

marc-gr · 2021-02-15T16:48:20Z

Thanks @punisherVX for the hard work!

punisherVX · 2021-02-15T18:06:55Z

Thanks @punisherVX for the hard work!

Thanks for all the help @marc-gr!! Great learning experience on how this is all done as well.

* Add Pensando module init * explicitly define the ECS version per testing * updates to docs from make update * updates for pensando module * updates to documentation and db screenshot * add dashboard export to repo * update to add pensando beat * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update pipeline.yml Condensed all "remove" fields to 1 list of fields. * Update pipeline.yml Do not remove the payload_raw field. * Update filebeat/module/pensando/_meta/docs.asciidoc Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * Update config.yml Added syslog_host and syslog_port values as suggested. * Update docs.asciidoc Added documentation for syslog_host and syslog_port as suggested. * Update pipeline.yml Removing payload_raw - this and json are, essentially, the same field and no longer needed after parsing. * Update pipeline.yml Changed checks if values are != null to use the filebeat specific ignore_empty_value: true instead. * Remove set of event.module Remove the set param for event.module. Filebeat should add this automatically. * Apply suggestions from code review Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * Update test.log * Use convert instead of set for some fields Changed ECS sets for IP addresses and ports to converts of type ip and integer respectively. * Updates for geoip and autonomous system * add pensando dfw fields * fixes from make -C filebeat update * fixes for filebeat check * make update changes * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * remove old json file * ran tests * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * gen after run of 'mage -v pythonIntegTest' * Update fields.yml * mage fmt update request Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 4194408)

* Add Pensando module init * explicitly define the ECS version per testing * updates to docs from make update * updates for pensando module * updates to documentation and db screenshot * add dashboard export to repo * update to add pensando beat * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update pipeline.yml Condensed all "remove" fields to 1 list of fields. * Update pipeline.yml Do not remove the payload_raw field. * Update filebeat/module/pensando/_meta/docs.asciidoc Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * Update config.yml Added syslog_host and syslog_port values as suggested. * Update docs.asciidoc Added documentation for syslog_host and syslog_port as suggested. * Update pipeline.yml Removing payload_raw - this and json are, essentially, the same field and no longer needed after parsing. * Update pipeline.yml Changed checks if values are != null to use the filebeat specific ignore_empty_value: true instead. * Remove set of event.module Remove the set param for event.module. Filebeat should add this automatically. * Apply suggestions from code review Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> * Update test.log * Use convert instead of set for some fields Changed ECS sets for IP addresses and ports to converts of type ip and integer respectively. * Updates for geoip and autonomous system * add pensando dfw fields * fixes from make -C filebeat update * fixes for filebeat check * make update changes * Update filebeat/module/pensando/dfw/config/dfw.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * remove old json file * ran tests * Update filebeat/module/pensando/dfw/ingest/pipeline.yml Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> * gen after run of 'mage -v pythonIntegTest' * Update fields.yml * mage fmt update request Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com> Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 4194408) Co-authored-by: Edward Arcuri <7133080+punisherVX@users.noreply.github.com>

…-arm * upstream/master: [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905) [CI] googleStorageUploadExt step (elastic#24048) Check fields are documented for aws metricsets (elastic#23887) Update go-concert to 0.1.0 (elastic#23770) [Libbeat][New Processor] XML Decode (elastic#23678) Fix: bad substitution of API key (elastic#24036) [Filebeat] Add Pensando DFW Module (elastic#21063) [Filebeat] Check if processor is supported by ES version (elastic#23763) Syslog system tests: be more forgiving (elastic#24021)

punisherVX and others added 8 commits September 9, 2020 12:42

Merge pull request #1 from elastic/master

4750fc3

Update from original

Add Pensando module init

182ffa5

explicitly define the ECS version per testing

a653242

updates to docs from make update

60c9682

updates for pensando module

2e56ff8

updates to documentation and db screenshot

22c09d1

add dashboard export to repo

110235b

update to add pensando beat

1b2d4a4

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 10, 2020

andresrc added the Team:SIEM label Sep 11, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 11, 2020

adriansr requested a review from a team September 21, 2020 08:06

botelastic bot added the Stalled label Oct 21, 2020

botelastic bot removed the Stalled label Oct 21, 2020

botelastic bot added the Stalled label Dec 13, 2020

botelastic bot removed the Stalled label Dec 14, 2020

marc-gr self-assigned this Jan 12, 2021

marc-gr reviewed Jan 12, 2021

View reviewed changes

Update filebeat/module/pensando/dfw/config/dfw.yml

812bdd5

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

marc-gr reviewed Jan 27, 2021

View reviewed changes

punisherVX and others added 6 commits February 1, 2021 13:28

Update filebeat/module/pensando/dfw/ingest/pipeline.yml

d9140e3

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

Update filebeat/module/pensando/dfw/ingest/pipeline.yml

57307d5

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

Update filebeat/module/pensando/dfw/ingest/pipeline.yml

22a3ea3

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

Update filebeat/module/pensando/dfw/ingest/pipeline.yml

074f0aa

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

remove old json file

6e07cd9

ran tests

8e55c35

marc-gr reviewed Feb 9, 2021

View reviewed changes

filebeat/module/pensando/dfw/ingest/pipeline.yml Outdated Show resolved Hide resolved

punisherVX and others added 2 commits February 9, 2021 07:36

Update filebeat/module/pensando/dfw/ingest/pipeline.yml

f29d158

Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

gen after run of 'mage -v pythonIntegTest'

ee1b59f

marc-gr approved these changes Feb 10, 2021

View reviewed changes

marc-gr requested a review from andrewkroh February 10, 2021 11:44

andrewkroh previously requested changes Feb 12, 2021

View reviewed changes

filebeat/module/pensando/dfw/_meta/fields.yml Outdated Show resolved Hide resolved

Update fields.yml

0a619b9

mage fmt update request

494e8cc

marc-gr merged commit 4194408 into elastic:master Feb 15, 2021

marc-gr mentioned this pull request Feb 15, 2021

Cherry-pick #21063 to 7.x: [Filebeat] Add Pensando DFW Module #24045

Merged

6 tasks

marc-gr added the v7.12.0 label Feb 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add Pensando DFW Module #21063

[Filebeat] Add Pensando DFW Module #21063

punisherVX commented Sep 10, 2020

elasticmachine commented Sep 10, 2020

elasticmachine commented Sep 10, 2020

elasticmachine commented Sep 10, 2020 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

elasticmachine commented Sep 11, 2020

adriansr commented Sep 21, 2020

botelastic bot commented Oct 21, 2020

punisherVX commented Oct 21, 2020

punisherVX commented Nov 13, 2020

botelastic bot commented Dec 13, 2020

punisherVX commented Dec 14, 2020

marc-gr commented Jan 12, 2021

marc-gr left a comment •

edited

marc-gr left a comment

marc-gr commented Feb 2, 2021

marc-gr commented Feb 8, 2021

marc-gr left a comment

marc-gr commented Feb 10, 2021

marc-gr left a comment

andrewkroh left a comment

marc-gr commented Feb 12, 2021

marc-gr commented Feb 15, 2021

marc-gr commented Feb 15, 2021

punisherVX commented Feb 15, 2021

[Filebeat] Add Pensando DFW Module #21063

[Filebeat] Add Pensando DFW Module #21063

Conversation

punisherVX commented Sep 10, 2020

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

None

Use cases

Screenshots

Logs

elasticmachine commented Sep 10, 2020

elasticmachine commented Sep 10, 2020

elasticmachine commented Sep 10, 2020 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Sep 11, 2020

adriansr commented Sep 21, 2020

botelastic bot commented Oct 21, 2020

punisherVX commented Oct 21, 2020

punisherVX commented Nov 13, 2020

botelastic bot commented Dec 13, 2020

punisherVX commented Dec 14, 2020

marc-gr commented Jan 12, 2021

marc-gr left a comment • edited

Choose a reason for hiding this comment

marc-gr left a comment

Choose a reason for hiding this comment

marc-gr commented Feb 2, 2021

marc-gr commented Feb 8, 2021

marc-gr left a comment

Choose a reason for hiding this comment

marc-gr commented Feb 10, 2021

marc-gr left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

marc-gr commented Feb 12, 2021

marc-gr commented Feb 15, 2021

marc-gr commented Feb 15, 2021

punisherVX commented Feb 15, 2021

elasticmachine commented Sep 10, 2020 •

edited by jenkins-beats-ci bot

marc-gr left a comment •

edited