New fileset for googlecloud firewall logs #14553

adriansr · 2019-11-15T17:19:34Z

This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps parsing firewall logs generated by rules under VPC Network -> Firewall Rules.

Note that GCP only logs firewall events under the following conditions:

Logging needs to be enabled for each individual rule in order to log.
Only TCP and UDP rules can be logged.

See https://cloud.google.com/vpc/docs/using-firewall-rules-logging.

x-pack/filebeat/module/googlecloud/firewall/config/pipeline.js

x-pack/filebeat/module/googlecloud/firewall/_meta/fields.yml

elasticmachine · 2019-11-18T15:27:24Z

Pinging @elastic/siem (Team:SIEM)

andrewkroh

I like the addition of log.name and the keep_original_message. Either here or in a separate PR, can you please add those to vpcflow to keep them consistent?

x-pack/filebeat/module/googlecloud/firewall/config/input.yml

x-pack/filebeat/module/googlecloud/firewall/config/pipeline.js

andrewkroh

LGTM

webmat · 2019-11-19T15:04:14Z

x-pack/filebeat/module/googlecloud/firewall/config/pipeline.js

+            ALLOWED: "allow",
+            DENIED: "deny"
+        },
+        default: "unknown"


Please note that event.outcome is still reserved.

The current thinking is that expected values in this field (when present) should be: "success" or "failure". Finer grained information such as "allow" or "deny" should be in another place, perhaps in event.action or a custom field.

As usual, populating a reserved field signs you up to having to do a breaking change later.

cc @MikePaquette

webmat · 2019-11-19T15:11:22Z

x-pack/filebeat/module/googlecloud/firewall/test/rare.log-expected.json

+        "destination.domain": "local-adrian-test",
+        "destination.ip": "10.128.0.16",
+        "destination.port": 80,
+        "event.category": "firewall-rule",


event.category is still reserved, and at this time, there is no plan to have a category named "firewall-rule".

Leaving this here signs you up to have to do a breaking change, once the expected values are published for event.category. The name of the category that will encapsulate firewall rule events is still very much in flux.

cc @MikePaquette

webmat · 2019-11-19T15:14:15Z

x-pack/filebeat/module/googlecloud/firewall/test/test.log-expected.json

+        "related.ip": [
+            "10.128.0.16",
+            "8.8.8.8"
+        ],


OMG thanks for filling that ❤️ 🙂

webmat · 2019-11-19T15:15:28Z

Other than me harping on the reserved fields, I'm really excited about this! Good work, as usual :-)

adriansr · 2019-11-19T16:50:38Z

Thanks @webmat

Regarding event.outcome / event.category, those are the fields and values already in use by other firewall data sources (cisco, iptables, panos), so I think its better to keep compatibility now and when those fields are standarized we can rename them all.

This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps parsing firewall logs generated by rules under VPC Network -> Firewall Rules. Note that GCP only logs firewall events under the following conditions: - Logging needs to be enabled for each individual rule in order to log. - Only TCP and UDP rules can be logged. (cherry picked from commit 4a66f0b)

…15621) * New fileset for googlecloud firewall logs (#14553) This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps parsing firewall logs generated by rules under VPC Network -> Firewall Rules. Note that GCP only logs firewall events under the following conditions: - Logging needs to be enabled for each individual rule in order to log. - Only TCP and UDP rules can be logged. (cherry picked from commit 4a66f0b) * googlecloud/vpcflow fileset: Populate additional log fields (#14608) To keep the vpcflow fileset of the googlecloud module aligned with the new firewall fileset, a `var.keep_original_message` option is added. Also the log.logger ECS field is now filled.

…l logs (elastic#15621) * New fileset for googlecloud firewall logs (elastic#14553) This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps parsing firewall logs generated by rules under VPC Network -> Firewall Rules. Note that GCP only logs firewall events under the following conditions: - Logging needs to be enabled for each individual rule in order to log. - Only TCP and UDP rules can be logged. (cherry picked from commit 4a66f0b) * googlecloud/vpcflow fileset: Populate additional log fields (elastic#14608) To keep the vpcflow fileset of the googlecloud module aligned with the new firewall fileset, a `var.keep_original_message` option is added. Also the log.logger ECS field is now filled. (cherry picked from commit 22fb66d)

…15621) (#15625) * New fileset for googlecloud firewall logs (#14553) This PR adds a new fileset, firewall, to the googlecloud module in Filebeat. It helps parsing firewall logs generated by rules under VPC Network -> Firewall Rules. Note that GCP only logs firewall events under the following conditions: - Logging needs to be enabled for each individual rule in order to log. - Only TCP and UDP rules can be logged. (cherry picked from commit 4a66f0b) * googlecloud/vpcflow fileset: Populate additional log fields (#14608) To keep the vpcflow fileset of the googlecloud module aligned with the new firewall fileset, a `var.keep_original_message` option is added. Also the log.logger ECS field is now filled. (cherry picked from commit 22fb66d)

houndci-bot reviewed Nov 15, 2019

View reviewed changes

adriansr force-pushed the feature_fb_gcp_firewall branch 3 times, most recently from 864113e to 6b3ef3c Compare November 15, 2019 17:35

adriansr commented Nov 15, 2019

View reviewed changes

x-pack/filebeat/module/googlecloud/firewall/_meta/fields.yml Outdated Show resolved Hide resolved

adriansr marked this pull request as ready for review November 18, 2019 14:09

adriansr requested a review from a team as a code owner November 18, 2019 14:09

adriansr requested review from a team and removed request for a team November 18, 2019 15:26

adriansr added enhancement Filebeat Filebeat module review Team:SIEM labels Nov 18, 2019

andrewkroh approved these changes Nov 18, 2019

View reviewed changes

adriansr added 10 commits November 19, 2019 10:30

New fileset for googlecloud firewall logs

6dea670

Don't save labels

11ae94b

Share source/destination objects for all filesets

c50fc64

Make update

5d3a8c7

Update config comments

779f244

More update

7881acc

Add params for debug and to keep log.original

6564746

Add docs

0bf1af6

Address review comments

6ad8698

Update changelog

9c724fc

adriansr force-pushed the feature_fb_gcp_firewall branch from 9a8ac3b to 9c724fc Compare November 19, 2019 09:30

andrewkroh approved these changes Nov 19, 2019

View reviewed changes

Merge branch 'master' into feature_fb_gcp_firewall

fad9f8b

webmat reviewed Nov 19, 2019

View reviewed changes

adriansr merged commit 4a66f0b into elastic:master Nov 19, 2019

andrewkroh mentioned this pull request Nov 19, 2019

[Filebeat] Generate googlecloud.yml.disabled #14626

Merged

webmat mentioned this pull request Nov 20, 2019

Add Rule Fields for Observers and Agents elastic/ecs#598

Closed

7 tasks

ChrsMark mentioned this pull request Nov 21, 2019

Cherry-pick #14330 to 7.x: [Metricbeat] Add Kafka JMX metricsets #14656

Merged

3 tasks

andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Jan 16, 2020

adriansr mentioned this pull request Jan 16, 2020

Cherry-pick #14553 to 7.6: New fileset for googlecloud firewall logs #15620

Closed

adriansr added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jan 16, 2020

adriansr mentioned this pull request Jan 16, 2020

Cherry-pick #14553 to 7.6: New fileset for googlecloud firewall logs #15621

Merged

adriansr mentioned this pull request Jan 17, 2020

Cherry-pick #15621 to 7.x: Cherry-pick #14553 to 7.6: New fileset for googlecloud firewall logs #15625

Merged

urso mentioned this pull request Feb 8, 2020

[docs] Add 7.6 breaking changes and release highlights #16202

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New fileset for googlecloud firewall logs #14553

New fileset for googlecloud firewall logs #14553

adriansr commented Nov 15, 2019 •

edited

Loading

elasticmachine commented Nov 18, 2019

andrewkroh left a comment

andrewkroh left a comment

webmat Nov 19, 2019 •

edited

Loading

webmat Nov 19, 2019

webmat Nov 19, 2019

webmat Nov 19, 2019

webmat commented Nov 19, 2019

adriansr commented Nov 19, 2019

New fileset for googlecloud firewall logs #14553

New fileset for googlecloud firewall logs #14553

Conversation

adriansr commented Nov 15, 2019 • edited Loading

elasticmachine commented Nov 18, 2019

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

webmat Nov 19, 2019 • edited Loading

Choose a reason for hiding this comment

webmat Nov 19, 2019

Choose a reason for hiding this comment

webmat Nov 19, 2019

Choose a reason for hiding this comment

webmat Nov 19, 2019

Choose a reason for hiding this comment

webmat commented Nov 19, 2019

adriansr commented Nov 19, 2019

adriansr commented Nov 15, 2019 •

edited

Loading

webmat Nov 19, 2019 •

edited

Loading