Support Apache Kafka output #124

KeisukeYamashita · 2020-11-23T16:23:27Z

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area config

/area outputs

/area tests

/area helm

What this PR does / why we need it:

Apache Kafka is one of the most famous open-source distributed event streaming platform.
I use it a lot so I want falcosidekick to support. My goal is not requesting falcosidekick to support all products. But IMO, it's worth supporting Kafka.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This is just a suggestion so please feel free to reject.

I tested as below:

# Terminal 1. Wait for the event
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

# Terminal 1
$ go run ./ -c /tmp/falcosidekick.yaml

After accessing /test endpoint, it can see below

# Terminal 1. Wait for the event
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
{"output":"08:12:46.746475122: Noticcce System user ran an interactive command (user=bin user_loginuid=-1 command=login container_id=79de7a68b24a image=falcosecurity/event-generator) k8s.ns=falco k8s.pod=falco-event-generator-f476f6d5d-gwp2p container=79de7a68b24a k8s.ns=falco k8s.pod=falco-event-generator-f476f6d5d-gwp2p container=79de7a68b24a","priority":"Notice","rule":"System user interactive","time":"2020-11-17T08:12:46.746475122Z","output_fields":{"akey":"AValue","bkey":"BValue","ckey":"CValue","container.id":"79de7a68b24a","container.image.repository":"falcosecurity/event-generator","evt.time":1605600766746475000,"k8s.ns.name":"falco","k8s.pod.name":"falco-event-generator-f476f6d5d-gwp2p","proc.cmdline":"login","user.loginuid":-1,"user.name":"bin"}}

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

poiana · 2020-11-23T16:23:30Z

@KeisukeYamashita: The label(s) area/outputs cannot be applied, because the repository doesn't have them

In response to this:

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area config

/area outputs

/area tests

/area helm

What this PR does / why we need it:

Apache Kafka is one of the most famous open-source distributed event streaming platform.
I use it a lot so I want falcosidekick to support.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This is just a suggestion so please feel free to reject.

I tested as below:
# Terminal 1. Wait for the event
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning

# Terminal 1
$ go run ./ -c /tmp/falcosidekick.yaml 
After accessing /test endpoint, it can see below
# Terminal 1. Wait for the event
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
{"output":"08:12:46.746475122: Noticcce System user ran an interactive command (user=bin user_loginuid=-1 command=login container_id=79de7a68b24a image=falcosecurity/event-generator) k8s.ns=falco k8s.pod=falco-event-generator-f476f6d5d-gwp2p container=79de7a68b24a k8s.ns=falco k8s.pod=falco-event-generator-f476f6d5d-gwp2p container=79de7a68b24a","priority":"Notice","rule":"System user interactive","time":"2020-11-17T08:12:46.746475122Z","output_fields":{"akey":"AValue","bkey":"BValue","ckey":"CValue","container.id":"79de7a68b24a","container.image.repository":"falcosecurity/event-generator","evt.time":1605600766746475000,"k8s.ns.name":"falco","k8s.pod.name":"falco-event-generator-f476f6d5d-gwp2p","proc.cmdline":"login","user.loginuid":-1,"user.name":"bin"}}

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Issif · 2020-11-23T21:39:47Z

README.md

@@ -247,6 +248,12 @@ googlechat:
  # outputformat: "" # all (default), text
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  messageformat: "Alert : rule *{{ .Rule }}* triggered by user *{{ index .OutputFields \"user.name\" }}*" # a Go template to format Google Chat Text above Attachment, displayed in addition to the output from `GOOGLECHAT_OUTPUTFORMAT`, see [Slack Message Formatting](#slack-message-formatting) in the README for details. If empty, no Text is displayed before Attachment.
+
+kafka:
+  url: "" # Apache Kafka URL (ex: http://kafka). Defaults to port 9092 if no port is specified after the domain.


It misses a message to specify it enables the output

Thank you, I added it 👍

Issif · 2020-11-23T21:40:16Z

config_example.yaml

@@ -137,3 +137,9 @@ googlechat:
  # outputformat: "" # all (default), text
  # minimumpriority: "" # minimum priority of event for using this output, order is emergency|alert|critical|error|warning|notice|informational|debug or "" (default)
  messageformat: 'Alert : rule *{{ .Rule }}* triggered by user *{{ index .OutputFields "user.name" }}*' # a Go template to format Slack Text above Attachment, displayed in addition to the output from `SLACK_OUTPUTFORMAT`, see [Slack Message Formatting](#slack-message-formatting) in the README for details. If empty, no Text is displayed before Attachment.
+
+kafka:
+  url: "" # Apache Kafka URL (ex: http://kafka). Defaults to port 9092 if no port is specified after the domain.


Same comment about enabling

main.go

outputs/client.go

outputs/kafka.go

config.go

outputs/kafka.go

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

Co-authored-by: Thomas Labarussias <issif+github@gadz.org> Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

Issif

❤️

poiana · 2020-11-24T10:36:19Z

LGTM label has been added.

Git tree hash: 420ee89bcef33c46194e34f27d592cc3aef57c92

poiana · 2020-11-24T10:36:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Issif

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Issif]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Support Apache kafka

455c679

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

poiana added the kind/feature New feature or request label Nov 23, 2020

poiana added the dco-signoff: yes label Nov 23, 2020

poiana requested review from Issif and leodido November 23, 2020 16:23

poiana added the size/L label Nov 23, 2020

KeisukeYamashita changed the title ~~Support Apache kafka output~~ Support Apache Kafka output Nov 23, 2020

Issif requested changes Nov 23, 2020

View reviewed changes

poiana assigned Issif Nov 23, 2020

KeisukeYamashita added 2 commits November 24, 2020 09:45

Add missing kafka document

2aebf83

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

Add description on enabling kafka

42d7362

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

poiana added dco-signoff: no and removed dco-signoff: yes labels Nov 24, 2020

Apply suggestions from code review

154ebbb

Co-authored-by: Thomas Labarussias <issif+github@gadz.org> Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

KeisukeYamashita force-pushed the support-kafka branch from 5ff6e9b to 154ebbb Compare November 24, 2020 00:48

poiana added dco-signoff: yes and removed dco-signoff: no labels Nov 24, 2020

Gofmt kafka output

0a3c2ea

Signed-off-by: KeisukeYamashita <19yamashita15@gmail.com>

KeisukeYamashita requested a review from Issif November 24, 2020 08:08

Issif approved these changes Nov 24, 2020

View reviewed changes

poiana added the lgtm label Nov 24, 2020

poiana added the approved label Nov 24, 2020

poiana merged commit bc6f5da into falcosecurity:master Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Apache Kafka output #124

Support Apache Kafka output #124

KeisukeYamashita commented Nov 23, 2020 •

edited

poiana commented Nov 23, 2020

Issif Nov 23, 2020

KeisukeYamashita Nov 24, 2020

Issif Nov 23, 2020

Issif left a comment

poiana commented Nov 24, 2020

poiana commented Nov 24, 2020

Support Apache Kafka output #124

Support Apache Kafka output #124

Conversation

KeisukeYamashita commented Nov 23, 2020 • edited

poiana commented Nov 23, 2020

Issif Nov 23, 2020

Choose a reason for hiding this comment

KeisukeYamashita Nov 24, 2020

Choose a reason for hiding this comment

Issif Nov 23, 2020

Choose a reason for hiding this comment

Issif left a comment

Choose a reason for hiding this comment

poiana commented Nov 24, 2020

poiana commented Nov 24, 2020

KeisukeYamashita commented Nov 23, 2020 •

edited