Reorg Headers, Add Basic Auth for Elasticsearch #245

distortedsignal · 2021-06-19T01:19:20Z

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area outputs

/area tests

What this PR does / why we need it:

Elasticsearch primarily supports Basic Authentication as the means of authenticating to the server. This PR should allow users to use HTTP Basic Authentication with Elasticsearch. There was an attempt to make it easy to apply Basic Authentication to other providers as well, though no other output providers are configured in this PR.

This PR also cleans up the Post method in client.go to force the outputs to set the headers in their own Post methods.

Which issue(s) this PR fixes:

N/A

Special notes for your reviewer:

Previously: I'm going to get to the tests eventually. Trust me.

Now: Maybe I got to the tests? Maybe?

poiana · 2021-06-19T01:19:28Z

Welcome @distortedsignal! It looks like this is your first PR to falcosecurity/falcosidekick 🎉

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

distortedsignal · 2021-06-19T08:33:53Z

I don't know if this will give anyone any kind of confidence in the quality of this change, but I got this output on my machine:

➜  falcosidekick git:(add-es-basic-auth) date && make falcosidekick && date && make test && date && make test-coverage && date
Sat Jun 19 01:31:01 PDT 2021
go build -gcflags all=-trimpath=/src -asmflags all=-trimpath=/src -a -installsuffix cgo -o falcosidekick .
Sat Jun 19 01:31:27 PDT 2021
go vet ./...
go test -v -race ./...
?   	github.com/falcosecurity/falcosidekick	[no test files]
=== RUN   TestNewAlertmanagerPayloadO
--- PASS: TestNewAlertmanagerPayloadO (0.00s)
=== RUN   TestNewClient
2021/06/19 01:31:34 [ERROR] : test - Bad Endpoint
--- PASS: TestNewClient (0.00s)
=== RUN   TestPost
2021/06/19 01:31:34 [ERROR] :  - Header missing (400)
2021/06/19 01:31:34 [ERROR] :  - Authentication Error (401)
2021/06/19 01:31:34 [ERROR] :  - Access Denied (403)
2021/06/19 01:31:34 [ERROR] :  - Resource not found (404)
2021/06/19 01:31:34 [ERROR] :  - Bad Request (422)
2021/06/19 01:31:34 [ERROR] :  - Exceeding post rate limit (429)
2021/06/19 01:31:34 [ERROR] :  - Unexpected Response  (502)
2021/06/19 01:31:34 [INFO]  :  - Post OK (200)
--- PASS: TestPost (0.01s)
=== RUN   TestAddHeader
2021/06/19 01:31:34 [INFO]  :  - Post OK (200)
--- PASS: TestAddHeader (0.00s)
=== RUN   TestAddBasicAuth
2021/06/19 01:31:34 [INFO]  :  - Post OK (200)
--- PASS: TestAddBasicAuth (0.00s)
=== RUN   TestMutualTlsPost
2021/06/19 01:31:48 [INFO]  :  - Post OK (200)
--- PASS: TestMutualTlsPost (14.07s)
=== RUN   TestNewDatadogPayload
--- PASS: TestNewDatadogPayload (0.00s)
=== RUN   TestNewDiscordPayload
--- PASS: TestNewDiscordPayload (0.00s)
=== RUN   TestNewGoogleChatPayload
--- PASS: TestNewGoogleChatPayload (0.00s)
=== RUN   TestNewInfluxdbPayload
--- PASS: TestNewInfluxdbPayload (0.00s)
=== RUN   TestNewLokiPayload
--- PASS: TestNewLokiPayload (0.00s)
=== RUN   TestMattermostPayload
--- PASS: TestMattermostPayload (0.00s)
=== RUN   TestNewOpsgeniePayload
--- PASS: TestNewOpsgeniePayload (0.00s)
=== RUN   TestPagerdutyPayload
--- PASS: TestPagerdutyPayload (0.00s)
=== RUN   TestNewRocketchatPayload
--- PASS: TestNewRocketchatPayload (0.00s)
=== RUN   TestNewSlackPayload
--- PASS: TestNewSlackPayload (0.00s)
=== RUN   TestNewTeamsPayload
--- PASS: TestNewTeamsPayload (0.00s)
PASS
ok  	github.com/falcosecurity/falcosidekick/outputs	14.752s
=== RUN   TestPriorityType_MarshalJSON
=== RUN   TestPriorityType_MarshalJSON/Default
=== RUN   TestPriorityType_MarshalJSON/Debug
=== RUN   TestPriorityType_MarshalJSON/Informational
=== RUN   TestPriorityType_MarshalJSON/Notice
=== RUN   TestPriorityType_MarshalJSON/Warning
=== RUN   TestPriorityType_MarshalJSON/Error
=== RUN   TestPriorityType_MarshalJSON/Critical
=== RUN   TestPriorityType_MarshalJSON/Alert
=== RUN   TestPriorityType_MarshalJSON/Emergency
=== RUN   TestPriorityType_MarshalJSON/Unknown_Key
--- PASS: TestPriorityType_MarshalJSON (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Default (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Debug (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Informational (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Notice (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Warning (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Error (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Critical (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Alert (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Emergency (0.00s)
    --- PASS: TestPriorityType_MarshalJSON/Unknown_Key (0.00s)
=== RUN   TestPriorityType_UnmarshalJSON
=== RUN   TestPriorityType_UnmarshalJSON/Default
=== RUN   TestPriorityType_UnmarshalJSON/Debug
=== RUN   TestPriorityType_UnmarshalJSON/Informational
=== RUN   TestPriorityType_UnmarshalJSON/Notice
=== RUN   TestPriorityType_UnmarshalJSON/Warning
=== RUN   TestPriorityType_UnmarshalJSON/Error
=== RUN   TestPriorityType_UnmarshalJSON/Critical
=== RUN   TestPriorityType_UnmarshalJSON/Alert
=== RUN   TestPriorityType_UnmarshalJSON/Emergency
=== RUN   TestPriorityType_UnmarshalJSON/Unknown_Key
=== RUN   TestPriorityType_UnmarshalJSON/an_error
--- PASS: TestPriorityType_UnmarshalJSON (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Default (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Debug (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Informational (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Notice (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Warning (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Error (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Critical (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Alert (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Emergency (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/Unknown_Key (0.00s)
    --- PASS: TestPriorityType_UnmarshalJSON/an_error (0.00s)
=== RUN   TestPriority
=== RUN   TestPriority/Default
=== RUN   TestPriority/Debug
=== RUN   TestPriority/Informational
=== RUN   TestPriority/Notice
=== RUN   TestPriority/Warning
=== RUN   TestPriority/Error
=== RUN   TestPriority/Critical
=== RUN   TestPriority/Alert
=== RUN   TestPriority/Emergency
=== RUN   TestPriority/Debug#01
=== RUN   TestPriority/Informational#01
=== RUN   TestPriority/Notice#01
=== RUN   TestPriority/Warning#01
=== RUN   TestPriority/Error#01
=== RUN   TestPriority/Critical#01
=== RUN   TestPriority/Alert#01
=== RUN   TestPriority/Emergency#01
=== RUN   TestPriority/Unknown_Key
--- PASS: TestPriority (0.01s)
    --- PASS: TestPriority/Default (0.00s)
    --- PASS: TestPriority/Debug (0.00s)
    --- PASS: TestPriority/Informational (0.00s)
    --- PASS: TestPriority/Notice (0.00s)
    --- PASS: TestPriority/Warning (0.00s)
    --- PASS: TestPriority/Error (0.00s)
    --- PASS: TestPriority/Critical (0.00s)
    --- PASS: TestPriority/Alert (0.00s)
    --- PASS: TestPriority/Emergency (0.00s)
    --- PASS: TestPriority/Debug#01 (0.00s)
    --- PASS: TestPriority/Informational#01 (0.00s)
    --- PASS: TestPriority/Notice#01 (0.00s)
    --- PASS: TestPriority/Warning#01 (0.00s)
    --- PASS: TestPriority/Error#01 (0.00s)
    --- PASS: TestPriority/Critical#01 (0.00s)
    --- PASS: TestPriority/Alert#01 (0.00s)
    --- PASS: TestPriority/Emergency#01 (0.00s)
    --- PASS: TestPriority/Unknown_Key (0.00s)
PASS
ok  	github.com/falcosecurity/falcosidekick/types	(cached)
Sat Jun 19 01:31:48 PDT 2021
go test ./outputs -count=1 -cover -v ./...
=== RUN   TestNewAlertmanagerPayloadO
--- PASS: TestNewAlertmanagerPayloadO (0.00s)
=== RUN   TestNewClient
2021/06/19 01:32:00 [ERROR] : test - Bad Endpoint
--- PASS: TestNewClient (0.00s)
=== RUN   TestPost
2021/06/19 01:32:00 [INFO]  :  - Post OK (200)
2021/06/19 01:32:00 [ERROR] :  - Header missing (400)
2021/06/19 01:32:00 [ERROR] :  - Authentication Error (401)
2021/06/19 01:32:00 [ERROR] :  - Access Denied (403)
2021/06/19 01:32:00 [ERROR] :  - Resource not found (404)
2021/06/19 01:32:00 [ERROR] :  - Bad Request (422)
2021/06/19 01:32:00 [ERROR] :  - Exceeding post rate limit (429)
2021/06/19 01:32:00 [ERROR] :  - Unexpected Response  (502)
--- PASS: TestPost (0.00s)
=== RUN   TestAddHeader
2021/06/19 01:32:00 [INFO]  :  - Post OK (200)
--- PASS: TestAddHeader (0.00s)
=== RUN   TestAddBasicAuth
2021/06/19 01:32:00 [INFO]  :  - Post OK (200)
--- PASS: TestAddBasicAuth (0.00s)
=== RUN   TestMutualTlsPost
2021/06/19 01:32:03 [INFO]  :  - Post OK (200)
--- PASS: TestMutualTlsPost (3.21s)
=== RUN   TestNewDatadogPayload
--- PASS: TestNewDatadogPayload (0.00s)
=== RUN   TestNewDiscordPayload
--- PASS: TestNewDiscordPayload (0.00s)
=== RUN   TestNewGoogleChatPayload
--- PASS: TestNewGoogleChatPayload (0.00s)
=== RUN   TestNewInfluxdbPayload
--- PASS: TestNewInfluxdbPayload (0.00s)
=== RUN   TestNewLokiPayload
--- PASS: TestNewLokiPayload (0.00s)
=== RUN   TestMattermostPayload
--- PASS: TestMattermostPayload (0.00s)
=== RUN   TestNewOpsgeniePayload
--- PASS: TestNewOpsgeniePayload (0.00s)
=== RUN   TestPagerdutyPayload
--- PASS: TestPagerdutyPayload (0.00s)
=== RUN   TestNewRocketchatPayload
--- PASS: TestNewRocketchatPayload (0.00s)
=== RUN   TestNewSlackPayload
--- PASS: TestNewSlackPayload (0.00s)
=== RUN   TestNewTeamsPayload
--- PASS: TestNewTeamsPayload (0.00s)
PASS
coverage: 25.9% of statements
ok  	github.com/falcosecurity/falcosidekick/outputs	3.600s	coverage: 25.9% of statements
Sat Jun 19 01:32:03 PDT 2021

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

distortedsignal · 2021-06-19T08:39:12Z

outputs/gcpcloudrun.go

@@ -10,6 +10,10 @@ import (
 func (c *Client) CloudRunFunctionPost(falcopayload types.FalcoPayload) {
 	c.Stats.GCPCloudRun.Add(Total, 1)

+	if c.Config.GCP.CloudRun.JWT != "" {
+		c.AddHeader(AuthorizationHeaderKey, "Bearer "+c.Config.GCP.CloudRun.JWT)
+	}


Compare to the following hunk in the current main branch:

falcosidekick/outputs/client.go

Lines 181 to 183 in 970a905

if c.OutputType == "GCPCloudRun" && c.Config.GCP.CloudRun.JWT != "" {

req.Header.Add("Authorization", "Bearer "+c.Config.GCP.CloudRun.JWT)

}

distortedsignal · 2021-06-19T08:40:22Z

outputs/kubeless.go

@@ -72,6 +80,11 @@ func (c *Client) KubelessCall(falcopayload types.FalcoPayload) {
 		}
 		log.Printf("[INFO]  : Kubeless - Function Response : %v\n", string(rawbody))
 	} else {
+		c.AddHeader(KubelessEventIDKey, uuid.New().String())
+		c.AddHeader(KubelessEventTypeKey, KubelessEventTypeValue)
+		c.AddHeader(KubelessEventNamespaceKey, c.Config.Kubeless.Namespace)


Compare to the following hunk in the current main branch:

falcosidekick/outputs/client.go

Lines 175 to 179 in 970a905

if c.OutputType == Kubeless {

req.Header.Add("event-id", uuid.New().String())

req.Header.Add("event-type", "falco")

req.Header.Add("event-namespace", c.Config.Kubeless.Namespace)

}

distortedsignal · 2021-06-19T08:41:08Z

outputs/kubeless.go

+		c.AddHeader(KubelessEventIDKey, uuid.New().String())
+		c.AddHeader(KubelessEventTypeKey, KubelessEventTypeValue)
+		c.AddHeader(KubelessEventNamespaceKey, c.Config.Kubeless.Namespace)
+		c.ContentType = KubelessContentType


Compare to the following hunk in the current main branch:

falcosidekick/outputs/client.go

Lines 165 to 168 in 970a905

contentType := "application/json; charset=utf-8"

if c.OutputType == "Loki" || c.OutputType == Kubeless {

contentType = "application/json"

}

distortedsignal · 2021-06-19T08:41:44Z

outputs/loki.go

@@ -47,6 +50,7 @@ func newLokiPayload(falcopayload types.FalcoPayload, config *types.Configuration
 // LokiPost posts event to Loki
 func (c *Client) LokiPost(falcopayload types.FalcoPayload) {
 	c.Stats.Loki.Add(Total, 1)
+	c.ContentType = LokiContentType


Compare to the following hunk in the current main branch:

falcosidekick/outputs/client.go

Lines 165 to 168 in 970a905

contentType := "application/json; charset=utf-8"

if c.OutputType == "Loki" || c.OutputType == Kubeless {

contentType = "application/json"

}

distortedsignal · 2021-06-19T08:42:54Z

outputs/opsgenie.go

@@ -51,6 +51,7 @@ func newOpsgeniePayload(falcopayload types.FalcoPayload, config *types.Configura
 // OpsgeniePost posts event to OpsGenie
 func (c *Client) OpsgeniePost(falcopayload types.FalcoPayload) {
 	c.Stats.Opsgenie.Add(Total, 1)
+	c.AddHeader(AuthorizationHeaderKey, "GenieKey "+c.Config.Opsgenie.APIKey)


Compare to the following hunk in the current main branch:

falcosidekick/outputs/client.go

Lines 171 to 173 in 970a905

if c.OutputType == "Opsgenie" {

req.Header.Add("Authorization", "GenieKey "+c.Config.Opsgenie.APIKey)

}

distortedsignal · 2021-06-19T08:43:38Z

outputs/webhook.go

+		for i, j := range c.Config.Webhook.CustomHeaders {
+			c.AddHeader(i, j)
+		}
+	}


Compare to the following hunk in the current main branch:

falcosidekick/outputs/client.go

Lines 187 to 191 in 970a905

if len(c.Config.Webhook.CustomHeaders) != 0 && c.OutputType == "Webhook" {

for i, j := range c.Config.Webhook.CustomHeaders {

req.Header.Add(i, j)

}

}

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Issif

Really nice PR, thank you a lot. I had in my todo list for a while to refactore client.go. I was just too lazy for 😛 .

It's really nice to add sources for comparison, it's so convenient for the review, thank you.

I have a just a small comment for consistency of descriptions between files, else is good

README.md

distortedsignal · 2021-06-21T02:00:44Z

The last commit message should be "Standardize wording across documentation", but I can't spell right now.

Issif · 2021-06-21T07:19:44Z

The last commit message should be "Standardize wording across documentation", but I can't spell right now.

You can fix the last commit with git commit --amend && git push origin add-es-basic-auth --force

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Issif

well done 👍

poiana · 2021-06-21T20:40:15Z

LGTM label has been added.

Git tree hash: b1210d5f4c499f9a0a2af650db46f096728b6006

poiana · 2021-06-21T20:40:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: distortedsignal, Issif

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Issif]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

poiana added do-not-merge/work-in-progress kind/feature New feature or request dco-signoff: no area/outputs area/tests labels Jun 19, 2021

poiana requested review from developer-guy and Issif June 19, 2021 01:19

poiana added the size/M label Jun 19, 2021

Reorg Headers, Add Basic Auth for Elasticsearch

2f9bf74

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

distortedsignal force-pushed the add-es-basic-auth branch from 9fa38d8 to 2f9bf74 Compare June 19, 2021 01:27

poiana added dco-signoff: yes and removed dco-signoff: no labels Jun 19, 2021

Make consts CapCamelCase, add more consts, misc small tweaks.

2cba5ec

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

poiana added size/L and removed size/M labels Jun 19, 2021

distortedsignal added 4 commits June 19, 2021 00:40

Set default un/pw to empty string.

194a21e

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Doc updates.

834707e

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Add test

a9466f7

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Fix a test that I broke.

87a595b

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

distortedsignal marked this pull request as ready for review June 19, 2021 08:33

poiana removed the do-not-merge/work-in-progress label Jun 19, 2021

Make the Linter happy.

4a7d72b

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

distortedsignal commented Jun 19, 2021

View reviewed changes

distortedsignal added 2 commits June 19, 2021 01:47

Add function comments.

b7c2693

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Add a check that the digest is being b64 encoded.

bc109c7

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

Issif added this to In progress in 2.x via automation Jun 19, 2021

Issif added this to the 2.23.0 milestone Jun 19, 2021

Issif requested changes Jun 19, 2021

View reviewed changes

README.md Show resolved Hide resolved

poiana assigned Issif Jun 19, 2021

Standardize wording across documentation

0f2a0f2

Signed-off-by: Tom Kelley <distortedsignal@gmail.com>

distortedsignal force-pushed the add-es-basic-auth branch from f4b49dc to 0f2a0f2 Compare June 21, 2021 15:35

Issif approved these changes Jun 21, 2021

View reviewed changes

poiana added the lgtm label Jun 21, 2021

poiana added the approved label Jun 21, 2021

poiana merged commit c02fdbb into falcosecurity:master Jun 21, 2021

2.x automation moved this from In progress to Done Jun 21, 2021

distortedsignal mentioned this pull request Jun 23, 2021

Prevent memory leak in headers. #252

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reorg Headers, Add Basic Auth for Elasticsearch #245

Reorg Headers, Add Basic Auth for Elasticsearch #245

distortedsignal commented Jun 19, 2021 •

edited

poiana commented Jun 19, 2021

distortedsignal commented Jun 19, 2021

distortedsignal Jun 19, 2021 •

edited

distortedsignal Jun 19, 2021

distortedsignal Jun 19, 2021

distortedsignal Jun 19, 2021

distortedsignal Jun 19, 2021

distortedsignal Jun 19, 2021

Issif left a comment

distortedsignal commented Jun 21, 2021

Issif commented Jun 21, 2021

Issif left a comment

poiana commented Jun 21, 2021

poiana commented Jun 21, 2021

	if c.OutputType == "GCPCloudRun" && c.Config.GCP.CloudRun.JWT != "" {
	req.Header.Add("Authorization", "Bearer "+c.Config.GCP.CloudRun.JWT)
	}

	if c.OutputType == Kubeless {
	req.Header.Add("event-id", uuid.New().String())
	req.Header.Add("event-type", "falco")
	req.Header.Add("event-namespace", c.Config.Kubeless.Namespace)
	}

	contentType := "application/json; charset=utf-8"
	if c.OutputType == "Loki" \|\| c.OutputType == Kubeless {
	contentType = "application/json"
	}

	if c.OutputType == "Opsgenie" {
	req.Header.Add("Authorization", "GenieKey "+c.Config.Opsgenie.APIKey)
	}

	if len(c.Config.Webhook.CustomHeaders) != 0 && c.OutputType == "Webhook" {
	for i, j := range c.Config.Webhook.CustomHeaders {
	req.Header.Add(i, j)
	}
	}

Reorg Headers, Add Basic Auth for Elasticsearch #245

Reorg Headers, Add Basic Auth for Elasticsearch #245

Conversation

distortedsignal commented Jun 19, 2021 • edited

poiana commented Jun 19, 2021

distortedsignal commented Jun 19, 2021

distortedsignal Jun 19, 2021 • edited

Choose a reason for hiding this comment

distortedsignal Jun 19, 2021

Choose a reason for hiding this comment

distortedsignal Jun 19, 2021

Choose a reason for hiding this comment

distortedsignal Jun 19, 2021

Choose a reason for hiding this comment

distortedsignal Jun 19, 2021

Choose a reason for hiding this comment

distortedsignal Jun 19, 2021

Choose a reason for hiding this comment

Issif left a comment

Choose a reason for hiding this comment

distortedsignal commented Jun 21, 2021

Issif commented Jun 21, 2021

Issif left a comment

Choose a reason for hiding this comment

poiana commented Jun 21, 2021

poiana commented Jun 21, 2021

distortedsignal commented Jun 19, 2021 •

edited

distortedsignal Jun 19, 2021 •

edited