Logstash pipelines #6480

kaisecheng · 2023-03-03T02:25:16Z

This PR allows configuration of Logstash pipeline.yml in CRD. pipeline.yml is an array of map of string/interface{} defining multiple pipelines. . in between the key is treated as string.

- pipeline.id: stdin_stdout
  config.string: input { stdin {} } output { stdout{} }
- pipeline.id: demo
  pipeline.workers: 1
  path.config: "/usr/share/logstash/pipeline"

Pipelines can be set from either inline in Pipelines or secret referencing inPipelinesRef.

This PR includes tests

e2e TestPipelineConfigRefLogstash, TestPipelineConfigLogstash
unit test

Initial commit of a simple operator. The first operator will create: * A Service exposing the logstash metrics API, so it can be used for monitoring purposes * Secrets holding logstash.yml * A StatefulSet deploying the logstash pods * Pods with default liveness and readiness probes The sample logstash yml file as located in config/samples/logstash/logstash.yaml will create: ``` ✗ kubectl tree logstash logstash-sample NAMESPACE NAME READY REASON AGE default Logstash/logstash-sample - 4m5s default ├─Secret/logstash-sample-ls-config - 4m4s default ├─Service/logstash-sample-ls-http - 4m5s default │ └─EndpointSlice/logstash-sample-ls-http-kwfsg - 4m5s default └─StatefulSet/logstash-sample-ls - 4m4s default ├─ControllerRevision/logstash-sample-ls-79bd59f869 - 4m4s default ├─Pod/logstash-sample-ls-0 True 3m59s default ├─Pod/logstash-sample-ls-1 True 3m59s default └─Pod/logstash-sample-ls-2 True 3m59s ``` And shows status: ``` ✗ kubectl get logstash logstash-sample NAME AVAILABLE EXPECTED AGE VERSION logstash-sample 3 3 9m1s 8.6.1 ``` Still TODO: * Unit Tests * End to end Tests * Certificate handling on the HTTP Metrics API Tidy up Co-authored-by: Kaise Cheng <kaise.cheng@elastic.co>

`

Rudimentary tests for validation and naming

This reverts commit 6eabf4c.

This reverts commit 8016b7b. Remving e2e tests to get e2e tests prior to logstash working

This reverts commit b5c775e. And re-adds the e2e tests

This reverts commit d4908b7.

Add a test to make sure that Logstash is running, by testing the metrics API endpoint and checking that the value of status is "green"

This commit fixes the readiness probe, by setting the default visibility of the logstash API to 0.0.0.0, allowing access to the metrics API to the probe

fix syntax

pkg/apis/logstash/v1alpha1/logstash_types.go

test/e2e/logstash/pipeline_test.go

Co-authored-by: Michael Morello <michael.morello@gmail.com>

add comment to e2e

robbavey

Great stuff, just a couple of nits/suggestions around clarifying naming around PipelinesRef and ConfigRef

robbavey · 2023-04-05T12:59:17Z

pkg/controller/logstash/pipelines/config.go

+}
+
+// MustParse parses the given pipeline content into a Pipelines.
+// Expects content to be in YAML format. Panics on error.


Nit: Add comment to state that this should be used in tests only

robbavey · 2023-04-05T13:01:37Z

pkg/controller/logstash/pipelines/configref.go

+)
+
+// ConfigRefWatchName returns the name of the watch registered on the secret referenced in `pipelinesRef`.
+func ConfigRefWatchName(resource types.NamespacedName) string {


Nit: Maybe PipelinesRef instead of ConfigRef?

Maybe for the filename here too

Suggested change

func ConfigRefWatchName(resource types.NamespacedName) string {

func PipelinesRefWatchName(resource types.NamespacedName) string {

lint suggested not to use Pipelines prefix. Renamed to ref.go and RefWatchName()

robbavey · 2023-04-05T13:03:11Z

pkg/controller/logstash/pipelines/configref.go

+
+// ParseConfigRef retrieves the content of a secret referenced in `pipelinesRef`, sets up dynamic watches for that secret,
+// and parses the secret content into a PipelinesConfig.
+func ParseConfigRef(


Nit:

Suggested change

func ParseConfigRef(

func ParsePipelinesRef(

kaisecheng · 2023-04-05T16:49:18Z

@barkbay I think it is ready 2nd round

barkbay

LGTM! Nice work 👍

pkg/controller/logstash/pipelines/config.go

test/e2e/logstash/pipeline_test.go

barkbay · 2023-04-06T06:49:06Z

test/e2e/test/logstash/checks.go

@@ -9,6 +9,8 @@ import (
 	"encoding/json"
 	"fmt"

+	"github.com/elastic/cloud-on-k8s/v2/pkg/controller/common/settings"


nit: import grouping seems off

Co-authored-by: Michael Morello <michael.morello@gmail.com>

robbavey · 2023-04-11T12:55:56Z

#6674 adds pipeline reload functionality

robbavey and others added 30 commits February 15, 2023 13:26

Comment out certs for HTTPS for now

11f0f79

Fix up linter issus

7414098

Generate API docs

2304fa3

`

First set of unit tests

9459a70

Rudimentary tests for validation and naming

Fix goimports

285c7bf

Add version check

d6e7ef2

Add e2e tests

8016b7b

Temporarily take out probes

6eabf4c

Revert "Temporarily take out probes"

d4908b7

This reverts commit 6eabf4c.

Revert "Add e2e tests"

b5c775e

This reverts commit 8016b7b. Remving e2e tests to get e2e tests prior to logstash working

Fix linter

33371ad

Add logstash config details

8e6e707

Fix up typos

4afe0de

Revert "Revert "Add e2e tests""

5f9040f

This reverts commit b5c775e. And re-adds the e2e tests

Revert "Revert "Temporarily take out probes""

803b9ab

This reverts commit d4908b7.

Tidying up

0d79bc9

Add Logstash to sample and stack tests

4e73a41

Added basic logstash verification

5713bb4

Add a test to make sure that Logstash is running, by testing the metrics API endpoint and checking that the value of status is "green"

Fix readiness probe

e66b380

This commit fixes the readiness probe, by setting the default visibility of the logstash API to 0.0.0.0, allowing access to the metrics API to the probe

Tidy up

9fb1e20

add stack monitoring

34c2aca

update test

131f17c

fix sidecar ES ref namespace

b553970

allow podTemplate update

d9f8f90

fix stack monitoring e2e

f31ba83

doc and lint

7ff0f46

lint

9cf60b1

fix doc gen

d7a587e

bring back lint

0b41a42

move test func to test class

6ceda8c

fix syntax

barkbay reviewed Apr 4, 2023

View reviewed changes

pkg/apis/logstash/v1alpha1/logstash_types.go Outdated Show resolved Hide resolved

barkbay reviewed Apr 4, 2023

View reviewed changes

test/e2e/logstash/pipeline_test.go Outdated Show resolved Hide resolved

barkbay reviewed Apr 4, 2023

View reviewed changes

test/e2e/logstash/pipeline_test.go Outdated Show resolved Hide resolved

test/e2e/logstash/pipeline_test.go Show resolved Hide resolved

test/e2e/logstash/pipeline_test.go Show resolved Hide resolved

barkbay reviewed Apr 4, 2023

View reviewed changes

test/e2e/logstash/pipeline_test.go Show resolved Hide resolved

test/e2e/logstash/pipeline_test.go Show resolved Hide resolved

kaisecheng and others added 8 commits April 4, 2023 13:51

Update pkg/apis/logstash/v1alpha1/logstash_types.go

9922520

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update test/e2e/logstash/pipeline_test.go

7a0072d

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update test/e2e/logstash/pipeline_test.go

4a47fb4

Co-authored-by: Michael Morello <michael.morello@gmail.com>

update metrics api checking

754a74e

add comment to e2e

fix naming pipelines.yml

d8cf066

delete pipeline secret at the end of test

a976bda

fix naming pipelines.yml

b6b4ad0

move pipelines_*.go to pipelines package

eebc8a6

kaisecheng requested a review from barkbay April 5, 2023 09:58

robbavey reviewed Apr 5, 2023

View reviewed changes

kaisecheng added 2 commits April 5, 2023 17:33

rename & comment

8d52421

lint

991e636

kaisecheng mentioned this pull request Apr 5, 2023

Logstash add ElasticsearchRefs #6662

Merged

barkbay approved these changes Apr 6, 2023

View reviewed changes

kaisecheng and others added 7 commits April 6, 2023 11:12

Update pkg/controller/logstash/pipelines/config.go

7bb3f5d

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update pkg/controller/logstash/pipelines/config.go

7e816da

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update pkg/controller/logstash/pipelines/config.go

791e02d

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update pkg/controller/logstash/pipelines/config.go

9967ca4

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update test/e2e/logstash/pipeline_test.go

203a3b4

Co-authored-by: Michael Morello <michael.morello@gmail.com>

Update test/e2e/logstash/pipeline_test.go

f756314

Co-authored-by: Michael Morello <michael.morello@gmail.com>

import grouping

3cc5964

kaisecheng merged commit 7ce5d9a into elastic:feature/logstash Apr 6, 2023

thbkrkr added the :logstash label Jun 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logstash pipelines #6480

Logstash pipelines #6480

kaisecheng commented Mar 3, 2023 •

edited

Loading

robbavey left a comment

robbavey Apr 5, 2023

robbavey Apr 5, 2023

kaisecheng Apr 5, 2023

robbavey Apr 5, 2023

kaisecheng commented Apr 5, 2023

barkbay left a comment

barkbay Apr 6, 2023

robbavey commented Apr 11, 2023

	func ConfigRefWatchName(resource types.NamespacedName) string {
	func PipelinesRefWatchName(resource types.NamespacedName) string {

Logstash pipelines #6480

Logstash pipelines #6480

Conversation

kaisecheng commented Mar 3, 2023 • edited Loading

robbavey left a comment

Choose a reason for hiding this comment

robbavey Apr 5, 2023

Choose a reason for hiding this comment

robbavey Apr 5, 2023

Choose a reason for hiding this comment

kaisecheng Apr 5, 2023

Choose a reason for hiding this comment

robbavey Apr 5, 2023

Choose a reason for hiding this comment

kaisecheng commented Apr 5, 2023

barkbay left a comment

Choose a reason for hiding this comment

barkbay Apr 6, 2023

Choose a reason for hiding this comment

robbavey commented Apr 11, 2023

kaisecheng commented Mar 3, 2023 •

edited

Loading