update docs-gen cmd && update README #199

zhu733756 · 2022-02-15T03:45:19Z

Signed-off-by: zhu733756 zhu733756@kubesphere.io

update docs-gen cmd
update README.md
update RoadMap

Part of #190

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 · 2022-02-15T04:26:20Z

@FeynmanZhou @Felixnoo Could you help to review this pr, I updated the README part.

zhu733756 · 2022-02-15T04:28:43Z

@FeynmanZhou @Felixnoo Could you help to review this pr, I updated the README part.

Or you can pull another pr to implement it.

Felixnoo

@zhu733756 Hi, please see my comments.

Felixnoo · 2022-02-15T05:37:06Z

README.md

-![Kubesphere-logging-fluentbit](docs/images/kubesphere-logging-fluentbit.svg)
+![fluentbit-operator](docs/images/fluentbit-operator.svg)
+
+Besides, we have finished [the pr #189](https://github.com/fluent/fluentbit-operator/pull/189) to integrate fluent-operator as a forward log layer, aims to collect logs from fluentbit or other apps. The whole workflow could be described as below.


pr -> PR
aims to -> aiming to
fluentbit -> Fluent Bit (If there is no specific reason to use fluentbit or Fluentbit, I suggest changing all fluentbit or Fluentbit to Fluent Bit for consistency.)
could be -> is

I can't understand the meaning of a forward log layer. Is it a term? 👀

Unified Logging Layer may be better.

Felixnoo · 2022-02-15T05:38:22Z

README.md

+        - [Deploy the Kubernetes logging stack with Helm](#deploy-the-kubernetes-logging-stack-with-helm)
+    - [Collect auditd logs](#collect-auditd-logs)
+    - [Fluentd](#fluentd)
+      - [Collect logs from Fluentbit](#collect-logs-from-fluentbit)


Fluentbit -> Fluent Bit

Felixnoo · 2022-02-15T05:58:51Z

README.md

@@ -212,6 +230,63 @@ $ curl localhost:9200/_cat/indices
 green open ks-logstash-log-2021.04.06 QeI-k_LoQZ2h1z23F3XiHg  5 1 404879 0 298.4mb 149.2mb
 ```

+### Fluentd 
+
+Fluentd acts as a forward log layer to collect logs from fluentbit and other Apps. 


Apps -> apps

Felixnoo · 2022-02-15T06:03:55Z

README.md

+##### Fluentd ClusterFluentdConfig example
+
+```shell
+kubectl apply -f manifests/fluentd/fluentd-cluster-cfg-output-es.yaml
+```
+
+##### Fluentd FluentdConfig example
+
+```shell
+kubectl apply -f manifests/fluentd/fluentd-namespaced-cfg-output-es.yaml
+```
+
+##### Fluentd mixed FluentdConfig and ClusterFluentdConfig example
+
+```shell
+kubectl apply -f manifests/fluentd/fluentd-mixed-cfgs-output-es.yaml
+```
+
+##### Fluentd ClusterOutput and Output example in multi-tenant scenario
+
+```shell
+kubectl apply -f manifests/fluentd/fluentd-mixed-cfgs-multi-tenant-output.yaml
+```
+
+##### Fluentd outputs to kafka/es example
+
+```shell
+kubectl apply -f manifests/fluentd/fluentd-cluster-cfg-output-kafka.yaml
+kubectl apply -f manifests/fluentd/fluentd-cluster-cfg-output-es.yaml
+```
+
+##### Fluentd buffer example


In line 241, you wrote At first. I'm wondering what's the logic here?

Fluent bit sends log to fluentd with the inner forward plugin, and fluentd also needs to use the inner forward plugin to accept the input logs.

So, each level-5 title in my comment here stands for an option that users can choose to run the command if they want to enable the inner forward plugin for Fluentd to accept the input logs?

See below:

Enable Fluent Bit forward plugin

At first, we should enable the forward plugin in Fluent Bit to send logs to Fluentd.

kubectl apply -f manifests/fluentd/fluentbit-output-forward.yaml

And secondly, Fluentd also needs to use the forward input plugin to receive these input logs. This part has been combined into the following examples.

Fluentd ClusterFluentdConfig example

benjaminhuo · 2022-02-15T05:20:48Z

apis/fluentbit/v1alpha2/plugins/secret_types.go

 	ValueFrom ValueSource `json:"valueFrom,omitempty"`
 }

 // +kubebuilder:object:generate:=true
-
+// ValueSource defines a value source of a secret.


secret and valuesource have similar description like defines a value source of a secret. ?

benjaminhuo · 2022-02-15T05:24:56Z

apis/fluentd/v1alpha1/plugins/common/buffer_types.go

@@ -22,6 +22,7 @@ type BufferCommon struct {
 	LogLevel *string `json:"logLevel,omitempty"`
 }

+// Buffer defines all types for all buffer Plugins


=> Buffer defines various parameters for buffer Plugins

benjaminhuo · 2022-02-15T05:26:37Z

apis/fluentd/v1alpha1/plugins/common/common_types.go

@@ -18,6 +18,7 @@ type CommonFields struct {
 	LogLevel *string `json:"logLevel,omitempty"`
 }

+// Time defines the common parameters for time plugin


for xx plugin => for the xx plugin

benjaminhuo · 2022-02-15T05:29:37Z

apis/fluentd/v1alpha1/plugins/input/forward.go

@@ -4,7 +4,7 @@ import (
 	"fluent.io/fluent-operator/apis/fluentd/v1alpha1/plugins/common"
 )

-// The in_forward Input plugin listens to a TCP socket to receive the event stream
+// Forward defines the in_forward Input plugin listens to a TCP socket to receive the event stream


// Forward defines the in_forward Input plugin listens to a TCP socket to receive the event stream
=>
// Forward defines the in_forward Input plugin that listens on a TCP socket to receive the event stream

benjaminhuo · 2022-02-15T05:30:06Z

apis/fluentd/v1alpha1/plugins/input/http.go

@@ -4,6 +4,7 @@ import (
 	"fluent.io/fluent-operator/apis/fluentd/v1alpha1/plugins/common"
 )

+// Http defines the in_http Input plugin listens to a TCP socket to receive the event stream


that listens on a xxx

benjaminhuo · 2022-02-15T06:14:24Z

README.md

+kubectl apply -f manifests/fluentd/fluentd-namespaced-cfg-output-es.yaml
+```
+
+##### Fluentd mixed FluentdConfig and ClusterFluentdConfig example


Combining Fluentd cluster-wide and namespaced-wide configuration

benjaminhuo · 2022-02-15T06:16:17Z

README.md

+kubectl apply -f manifests/fluentd/fluentd-mixed-cfgs-output-es.yaml
+```
+
+##### Fluentd ClusterOutput and Output example in multi-tenant scenario


Combining Fluentd cluster-wide output and namespace-wide output for the multi-tenant scenario

benjaminhuo · 2022-02-15T06:16:53Z

README.md

+kubectl apply -f manifests/fluentd/fluentd-mixed-cfgs-multi-tenant-output.yaml
+```
+
+##### Fluentd outputs to kafka/es example


Outputing logs to Kafka or Elasticsearch

benjaminhuo · 2022-02-15T06:17:15Z

README.md

+kubectl apply -f manifests/fluentd/fluentd-cluster-cfg-output-es.yaml
+```
+
+##### Fluentd buffer example


Using buffer for Fluentd output

benjaminhuo · 2022-02-15T06:17:33Z

README.md

+kubectl apply -f manifests/fluentd/fluentd-cluster-cfg-output-buffer-example.yaml
+```
+
+#### Collect logs over HTTP


Collecting logs over HTTP

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 requested review from wenchajun, wanjunlei and benjaminhuo February 15, 2022 03:45

zhu733756 force-pushed the master branch 2 times, most recently from b807840 to 0f3623b Compare February 15, 2022 04:02

update docs-gen cmd && update README

ce33ab9

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 force-pushed the master branch from 0f3623b to ce33ab9 Compare February 15, 2022 04:03

fix fluentbit svg

644107d

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 force-pushed the master branch from 83cbfc6 to 644107d Compare February 15, 2022 04:17

update Roadmap

bc0611f

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 requested a review from FeynmanZhou February 15, 2022 04:26

Felixnoo reviewed Feb 15, 2022

View reviewed changes

benjaminhuo requested changes Feb 15, 2022

View reviewed changes

fix comments and README

b84ca58

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

benjaminhuo approved these changes Feb 15, 2022

View reviewed changes

benjaminhuo merged commit 55730d0 into fluent:master Feb 15, 2022

zhu733756 mentioned this pull request Feb 15, 2022

Renaming fluentbit-operator to fluent-operator #189 #190

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update docs-gen cmd && update README #199

update docs-gen cmd && update README #199

zhu733756 commented Feb 15, 2022 •

edited

zhu733756 commented Feb 15, 2022 •

edited by FeynmanZhou

zhu733756 commented Feb 15, 2022

Felixnoo left a comment

Felixnoo Feb 15, 2022

zhu733756 Feb 15, 2022

Felixnoo Feb 15, 2022

Felixnoo Feb 15, 2022

Felixnoo Feb 15, 2022

zhu733756 Feb 15, 2022

Felixnoo Feb 15, 2022 •

edited

zhu733756 Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

benjaminhuo Feb 15, 2022

update docs-gen cmd && update README #199

update docs-gen cmd && update README #199

Conversation

zhu733756 commented Feb 15, 2022 • edited

zhu733756 commented Feb 15, 2022 • edited by FeynmanZhou

zhu733756 commented Feb 15, 2022

Felixnoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Felixnoo Feb 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Enable Fluent Bit forward plugin

Fluentd ClusterFluentdConfig example

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhu733756 commented Feb 15, 2022 •

edited

zhu733756 commented Feb 15, 2022 •

edited by FeynmanZhou

Felixnoo Feb 15, 2022 •

edited