WIP: LOG-1491:Vector config generator #1211

ajaygupta978 · 2021-10-08T18:38:22Z

Description

This PR generates configuration for vector log collector.

Depending on PR(s):
Bugzilla:
Github issue:
JIRA: https://issues.redhat.com/browse/LOG-1491
Enhancement proposal:

ajaygupta978 · 2021-10-08T18:45:47Z

@vimalk78 This PR generates following simple config. This is just first version and there is 1-2 hard coding as of now like pipeline name. As discussed there might be some restructuring required in config.go so that pipeline names/ID can be passed to subsequent vector components.
I modified few test cases config_test.go , kafka_test.go and output_conf_kafka_test.go .

# Logs from containers (including openshift containers)
[sources.src_application]
  type = "kubernetes_logs"
  auto_partial_merge = true
  exclude_paths_glob_patterns = ["/var/log/containers/collector-*_openshift-logging_*.log", "/var/log/containers/elasticsearch-*_openshift-logging_*.log", "/var/log/containers/kibana-*_openshift-logging_*.log"]

[transforms.transform_application]
inputs = ["src_application"]
type = "remap"
source = """
. = parse_json!(.message)
.log_type = application"""

# Ship logs to specific outputs
[sinks.kafka]
type = "kafka"
input = ["transform_application"]
bootstrap_servers = "broker1-kafka.svc.messaging.cluster.local:9092"
topic = "build_complete"
sasl.enabled = false

vimalk78 · 2021-10-08T18:48:00Z

@ajaygupta978 if this is a Work in progress, please add [WIP] to the beginning of PR title

[WIP] LOG-1491:Vector config generator

jcantrill

I believe there is a bunch of info here that needs to be removed for the sake of it's not fluentd

internal/generator/vector/config.go

internal/generator/vector/elements/label.go

internal/generator/vector/elements/match.go

internal/generator/vector/elements/record_modifier.go

openshift-ci · 2021-10-21T08:29:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ajaygupta978
To complete the pull request process, please ask for approval from jcantrill after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ajaygupta978 · 2021-10-21T08:35:33Z

@vimalk78 @jcantrill
In previous version there was hard coding and design was also rigid because I was thinking to use fluentd design. Now in this I have changed the code and it covers application and infrastructure pipeline with Kafka output.
Apart from junit test following is kind of bit more generic pipeline specification which collects application and infra logs and send to kafka. I am adding audit pipeline and other output types.
Could you please review and let me know your comments ?

# Logs from containers
[sources.kubernetes_logs]
  type = "kubernetes_logs"
  auto_partial_merge = true
  exclude_paths_glob_patterns = ["/var/log/containers/collector-*_openshift-logging_*.log", "/var/log/containers/elasticsearch-*_openshift-logging_*.log", "/var/log/containers/kibana-*_openshift-logging_*.log"]

# Logs from journald
[sources.journald]
  type = "journald"

[transforms.transform_kubernetes_logs]
  inputs = ["kubernetes_logs"]
  type = "route"
  route.app = '!(starts_with!(.kubernetes.pod_namespace,"kube") && starts_with!(.kubernetes.pod_namespace,"openshift") && .kubernetes.pod_namespace == "default")'

[transforms.transform_journald]
  inputs = ["kubernetes_logs", "journald"]
  type = "route"
  route.infra = '(starts_with!(.kubernetes.pod_namespace,"kube") && starts_with!(.kubernetes.pod_namespace,"openshift") && .kubernetes.pod_namespace == "default")'

# Ship logs to specific outputs
[sinks.kafka]
  type = "kafka"
  input = ["transform_kubernetes_logs.app", "transform_journald.infra"]
  bootstrap_servers = "broker1-kafka.svc.messaging.cluster.local:9092"
  topic = "build_complete"
  sasl.enabled = false

ajaygupta978 · 2021-10-21T08:35:57Z

/hold

internal/generator/vector/config_test.go

internal/generator/vector/helpers/helpers.go

vimalk78 · 2021-10-21T09:17:01Z

internal/generator/vector/inputs_to_pipeline.go

better to do this matching on CLF spec rather than the config we are trying to produce

Ok will take care.

internal/generator/vector/inputs_to_pipeline.go

internal/generator/vector/helpers/helpers.go

internal/generator/vector/keyval_element.go

internal/generator/vector/labelrouter.go

LOG-1491:Vector config generator LOG-1491:Vector config generator LOG-1491:Vector config generator

ajaygupta978 · 2021-10-25T11:14:42Z

@jcantrill @vimalk78 could you please review it?

LOG-1491:Vector config generator

jcantrill

There still appears to be a bunch of fluent conf here

jcantrill · 2021-10-25T19:55:30Z

internal/generator/vector/helpers/helpers.go

+
+func ConcatArrays(input []string) string {
+	out := make([]string, len(input))
+	for i, a := range input {


If the input is an array string and the output is a string, why the need to spin through the input instead of just strings.Join()

This was required because each of the array element needed double quote.

inputs = ["pipeline1", "pipeline2"]

jcantrill · 2021-10-25T19:56:03Z

internal/generator/vector/inputs_to_pipeline.go

+
+ for _, source := range sources {
+
+  if source.Type() == "journald" {


please make a constant

jcantrill · 2021-10-25T19:56:28Z

internal/generator/vector/inputs_to_pipeline.go

@@ -0,0 +1,36 @@
+package vector


run make fmt to fix formatting

jcantrill · 2021-10-25T19:57:09Z

internal/generator/vector/inputs_to_pipeline.go

+    instance = transform.JournalTransform{
+    SourceID:      helpers.PipelineName("transform_" + source.Type()),
+    SrcType:       source.ComponentID(),
+    InputPipeline: []string{"kubernetes_logs", helpers.PipelineName(source.Type())},


jcantrill · 2021-10-25T19:58:47Z

internal/generator/vector/output/elasticsearch/elasticsearch.go

+[sinks.{{.StoreID}}]
+  type = "elasticsearch"
+  inputs = ` + helpers.ConcatArrays(e.InputPipeline) + `
+  endpoint = "{{.Host}}:{{.Port}}"


Might this need a protocol?

jcantrill · 2021-10-25T19:59:31Z

internal/generator/vector/output/elasticsearch/elasticsearch.go

+	return conf
+}
+
+func generateRubyDigArgs(path string) string {


Not ruby so not required

jcantrill · 2021-10-25T20:00:36Z

internal/generator/vector/output/elasticsearch/elasticsearch_test.go

+  pipeline = "pipeline-name"
+  compression = "none"
+  auth.strategy = "basic"
+auth.user = "#{File.exists?('/var/run/ocp-collector/secrets/es-1/username') ? open('/var/run/ocp-collector/secrets/es-1/username','r') do |f|f.read end : ''}"


this will have to be some equivalent vector scripting not ruby

jcantrill · 2021-10-25T20:01:38Z

internal/generator/vector/output/elasticsearch/tls.go

+
+func (t TLS) Template() string {
+	https := `{{define "elasticsearchTLSTemplate" -}}
+scheme https


Looks like fluentd conf

openshift-ci · 2021-10-28T03:11:20Z

@ajaygupta978: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-11-12T16:00:06Z

@ajaygupta978: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/lint	`9dfc152`	link	true	`/test lint`
ci/prow/unit	`9dfc152`	link	true	`/test unit`
ci/prow/e2e	`9dfc152`	link	true	`/test e2e`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ajaygupta978 · 2021-12-07T14:35:59Z

this is duplicate of #1230 so closing this.

openshift-ci bot assigned jcantrill Oct 8, 2021

openshift-ci bot requested a review from vimalk78 October 8, 2021 18:38

ajaygupta978 changed the title ~~LOG-1491:Vector config generator~~ LOG-1491:Vector config generator[WIP] Oct 8, 2021

jcantrill reviewed Oct 8, 2021

View reviewed changes

ajaygupta978 changed the title ~~LOG-1491:Vector config generator[WIP]~~ WIP: LOG-1491:Vector config generator Oct 13, 2021

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 13, 2021

ajaygupta978 force-pushed the Vector_Conf branch from 31187b5 to d6a4bbc Compare October 21, 2021 08:28

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 21, 2021

vimalk78 reviewed Oct 21, 2021

View reviewed changes

ajaygupta978 force-pushed the Vector_Conf branch from d6a4bbc to 81ec3dc Compare October 21, 2021 12:35

LOG-1491:Vector config generator

045f0af

LOG-1491:Vector config generator LOG-1491:Vector config generator LOG-1491:Vector config generator

ajaygupta978 force-pushed the Vector_Conf branch from 81ec3dc to cfcbe6f Compare October 25, 2021 07:06

LOG-1491:Vector config generator

9dfc152

LOG-1491:Vector config generator

ajaygupta978 force-pushed the Vector_Conf branch from cfcbe6f to 9dfc152 Compare October 25, 2021 11:53

jcantrill reviewed Oct 25, 2021

View reviewed changes

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2021

ajaygupta978 closed this Dec 7, 2021


		for _, source := range sources {

		if source.Type() == "journald" {

WIP: LOG-1491:Vector config generator #1211

WIP: LOG-1491:Vector config generator #1211

Uh oh!

Conversation

ajaygupta978 commented Oct 8, 2021

Description

Uh oh!

ajaygupta978 commented Oct 8, 2021

Uh oh!

vimalk78 commented Oct 8, 2021

Uh oh!

jcantrill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci bot commented Oct 21, 2021

Uh oh!

ajaygupta978 commented Oct 21, 2021

Uh oh!

ajaygupta978 commented Oct 21, 2021

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajaygupta978 commented Oct 25, 2021

Uh oh!

jcantrill left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Oct 28, 2021

Uh oh!

openshift-ci bot commented Nov 12, 2021

Uh oh!

ajaygupta978 commented Dec 7, 2021

Uh oh!

Uh oh!