Add fluentd support #189

zhu733756 · 2022-01-25T06:23:00Z

Signed-off-by: zhu733756 zhu733756@kubesphere.io

This pr add Fluentd-operator to implement the former proposal #138.

Backgrouds

See the former proposal mentioned in #138.

This proposal brought the basic concept to answer how to add fluentd-operator as a forward log layer to collect logs from fluentbit or other apps.

See below：

During the log Aggregation & Forwarding Phase, the Fluentd Operator defines five custom resources using CustomResourceDefinition (CRD):

Fluentd: Defines Fluentd instances and its associated config.
- defines common properties like pvc, replicas, resources, etc.
- select FluentdClusterConfig/FluentdConfig CRDs to bind with this instance.
FluentdClusterConfig:
- Support for multiple namespaces isolation and cloud native selectors.
- Integrate the logic of input and filter sections.
- Select any cluster/namespaced scope output crds.
FluentdConfig:
- Support for single namespace isolation and cloud native selectors in this namespace.
- Integrate the logic of input and filter sections
- Select any cluster/namespaced scope output crds, the output log should be from this namespace.
ClusterFilter: Global filters for fluentd.
Filter: Namespaced filters for fluentd.
ClusterOutput: Defines an output section without namespace restriction.
Output: Defines an output section with a specified namespace.

But we made some changes during actual development.

Namespaced fluentdconfig CR can select the namespaced CRs and the cluster CRs. Cluster fluentdconfig CR only selects clusterfilters or clusteroutputs.

Besides, you will see inputs sections combined into the fluentd CR named as globalInputs. Since the globalInputs is manily used for collecting the logs from fluentbit through forward plugin or other apps through http plugin. And we will dynamically
add the related service ports(forward/http, see demo below) according to this configuration.

Now two reconcilers are brought to finish this work.

Fluentd reconciler watches the related resources like statefulset/secrect/service/buffer PVC etc, it setups fluentd instances once a secret is generated by FluendConfig reconciler. FluendConfig reconciler watches the created fluentd CRs, picks out the fluentdCfgSelector to select the related resources, and combines to a configuration that would be mounted as a secret for this fluentd.

Another useful idea activated by the former work is that we support a hot config reloader for fluentbit-operator. So if the related CRs changed, the fluentd-watcher agent will gracefully reload the fluentd instance.

That's the whole change, the other idea is welcome and if I lost the ideas from the proposal, pls remind me.

The practice part

Since the output plugins are huge, I will use es for example to test the whole logic. And for the input layer, I directly use the forward from fluentbit to test the workflow.

1 Setup elasticsearch cluster

If you use Kubesphere Container platform, you can install it through ks-installer.

2 Setup fluent operator

$ kubectl apply -f manifests/setup/setup.yaml
---
$ kubectl -n kubesphere-logging-system get po
NAME                                  READY   STATUS    RESTARTS   AGE
elasticsearch-logging-data-0                                      1/1     Running     1          13d
elasticsearch-logging-discovery-0                                 1/1     Running     1          13d
fluent-bit-6wrv7                                                  1/1     Running     0          69m
fluentd-forward-85f6bcb488-7rhrq                                  1/1     Running     0          80m
fluent-operator-6f959849c7-fq2lp     1/1     Running   0          16

3 Setup fluentbit logging-stack, deleting the fluentbit output plugin

$ vim manifests/logging-stack/kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- fluentbit-fluentBit.yaml
- fluentbitconfig-fluentBitConfig.yaml
- input-tail.yaml
- input-systemd-docker.yaml
- input-systemd-kubelet.yaml
- filter-kubernetes.yaml
- filter-systemd.yaml
# - output-elasticsearch.yaml
# - output-forward.yaml
# - output-kafka.yaml
- filter-containerd.yaml
- fluentbit-containerd-config.yaml
- systemd-lua-config.yaml
---
$ kubectl apply -k manifests/logging-stack

4 Test the forward demo

Namespaced fluentdconfig example see follows, it only collects the namespaced logs.

apiVersion: fluentd.fluent.io/v1alpha1
kind: Fluentd
metadata:
  name: fluentd-forward
  namespace: kubesphere-logging-system
  labels:
    app.kubernetes.io/name: fluentd
spec:
  globalInputs:
    - forward: 
        bind: 0.0.0.0
        port: 24224
  replicas: 1
  image: zhu733756/fluentd:config-reloader 
  fluentdCfgSelector: 
    matchLabels:
      config.fluentd.fluent.io/enabled: "true"
   
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: FluentdConfig
metadata:
  name: fluentd-config
  namespace: kubesphere-logging-system
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  outputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"

---
apiVersion: fluentd.fluent.io/v1alpha1
kind: Output
metadata:
  name: fluentd-stdout
  namespace: kubesphere-logging-system
  labels:
    output.fluentd.fluent.io/enabled: "true"
spec: 
  outputs: 
    - elasticsearch:
        host: elasticsearch-logging-data.kubesphere-logging-system.svc
        port: 9200
        logstashFormat: true
        logstashPrefix: ks-logstash-log

CluserFluentdConfig exmaple see follows, it collects logs from the watched namespaces.

apiVersion: fluentd.fluent.io/v1alpha1
kind: Fluentd
metadata:
  name: fluentd-forward
  namespace: kubesphere-logging-system
  labels:
    app.kubernetes.io/name: fluentd
spec:
  globalInputs:
    - forward: 
        bind: 0.0.0.0
        port: 24224
  replicas: 1
  image: zhu733756/fluentd:config-reloader
  fluentdCfgSelector: 
    matchLabels:
      config.fluentd.fluent.io/enabled: "true"
   
---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterFluentdConfig
metadata:
  name: fluentd-config
  labels:
    config.fluentd.fluent.io/enabled: "true"
spec:
  watchedNamespaces: 
      - kube-system
      - kubesphere-logging-system # will collect logs from these namespaces.
  outputSelector:
    matchLabels:
      output.fluentd.fluent.io/enabled: "true"

---
apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: fluentd-stdout
  labels:
    output.fluentd.fluent.io/enabled: "true"
spec: 
  outputs: 
    -  elasticsearch:
        host: elasticsearch-logging-data.kubesphere-logging-system.svc
        port: 9200
        logstashFormat: true
        logstashPrefix: ks-logstash-log

I will use the cluster scope CRs to test it. Manifests located at manifests/forward.

kubectl apply -f manifests/forward/fluentd-cluster-cfg-output-es.yaml

After a while, a default buffer pvc(1G) bound to this statefulset, and a service named fluentd-forward created. You may disable pvc/service, change the volume type according the definition in fluentd CR.

$ kubectl -n kubesphere-logging-system get po 
NAME                                  READY   STATUS    RESTARTS   AGE
elasticsearch-logging-data-0                                      1/1     Running     1          13d
elasticsearch-logging-discovery-0                                 1/1     Running     1          13d
fluent-bit-6wrv7                                                  1/1     Running     0          72m
fluentd-forward-0                                1/1     Running     0          83m
fluent-operator-6f959849c7-fq2lp     1/1     Running   0          8m3s
$ kubectl -n kubesphere-logging-system get pvc
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-elasticsearch-logging-data-0        Bound    pvc-48f6bdfa-bed0-448e-88be-cd284a113529   8Gi        RWO            standard       78m
data-elasticsearch-logging-discovery-0   Bound    pvc-b2f89a45-f94a-4b49-93f6-f48fc0c17645   4Gi        RWO            standard       78m
fluentd-forward-buffer-pvc-fluentd-forward-0   Bound    pvc-059e0c27-c009-4caf-a160-06c6ab0ce32c   1Gi        RWO            standard       22h
$ kubectl -n kubesphere-logging-system get svc
NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
elasticsearch-logging-data        ClusterIP   10.96.98.206   <none>        9200/TCP    80m
elasticsearch-logging-discovery   ClusterIP   None           <none>        9300/TCP    80m
fluentd-forward                   ClusterIP   10.96.244.52   <none>        24224/TCP   3m10s

5 apply fluentbit output forward

Forward example:

apiVersion: logging.kubesphere.io/v1alpha2
kind: Output
metadata:
  name: fluentd
  namespace: kubesphere-logging-system
  labels:
    logging.kubesphere.io/enabled: "true"
    logging.kubesphere.io/component: logging
spec:
  matchRegex: (?:kube|service)\.(.*)
  forward:
    host: fluentd-forward.kubesphere-logging-system.svc # modify here according to the fluentd name
    port: 24224

6 Check results

$ kubectl -n kubesphere-logging-system exec -it elasticsearch-logging-data-0 -c elasticsearch -- curl -X GET "localhost:9200/ks-logstash*/_search?pretty" -H 'Content-Type: application/json' -d '{                                                           
  "size" : 0,
  "aggs" : {
     "kubernetes_ns": {
        "terms" : {
          "field": "kubernetes.namespace_name.keyword"
        }
     }
  }
}'

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 58,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "kubernetes_ns" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "kube-system",
          "doc_count" : 53
        },
        {
          "key" : "kubesphere-logging-system",
          "doc_count" : 5
        }
      ]
    }
  }
}

Todo

See #190

/cc @benjaminhuo @wanjunlei @wenchajun

wenchajun · 2022-01-25T07:17:54Z

api/fluentdoperator/v1alpha1/fluentd_types.go

+	// Tolerations
+	Tolerations []corev1.Toleration `json:"tolerations,omitempty"`
+	// RuntimeClassName represents the container runtime configuration.
+	RuntimeClassName string `json:"runtimeClassName,omitempty"`


Do fluentd images have different configurations when different container runtime?

Added to TODO list.

no, it's a log aggregator, not a log agent. So it only receives logs via network.

wenchajun · 2022-01-25T07:22:26Z

api/fluentdoperator/v1alpha1/fluentd_types.go

+
+// FluentdStatus defines the observed state of Fluentd
+type FluentdStatus struct {
+	Errors string `json:"errs,omitempty"`


maybe errors is better

zhu733756 · 2022-01-25T07:18:53Z

PROJECT

  version: v1alpha2
 - api:
    crdVersion: v1
    namespaced: true
  domain: kubesphere.io
  group: logging
  kind: Parser
-  path: kubesphere.io/fluentbit-operator/api/v1alpha2
+  path: kkubesphere.io/fluentbit-operator/api/fluentbitoperator/v1alpha2


modified to kubesphere.io

Why is kubesphere.io? Shouldn't it be fluent.io?

I see @wenchajun make this change in another pr :) /cc @wenchajun

Is fluentd's domain also kubesphere.io, if so, it may need to be changed as well.

zhu733756 · 2022-01-25T07:20:05Z

api/fluentdoperator/v1alpha1/clusterfluentdconfig_types.go

+	// Select output plugins
+	OutputSelector metav1.LabelSelector `json:"outputSelector,omitempty"`
+	// // The generated configuration style can be label/tag, default will use tag
+	// // +kubebuilder:validation:Enum:=label;tag


clean up these

zhu733756 · 2022-01-25T07:22:15Z

api/fluentdoperator/v1alpha1/clusteroutput_types.go

@@ -0,0 +1,60 @@
+/*
+Copyright 2021.


wanjunlei · 2022-01-25T07:43:42Z

The crd version of the new fluent bit is v2alpha1 or v1beta1, maybe the version of fluentd should be the same as the version of fluent bit?

zhu733756 · 2022-01-25T07:47:04Z

The crd version of the new fluent bit is v2alpha1 or v1beta1, maybe the version of fluentd should be the same as the version of fluent bit?

I think no need to. Since the two operators can be deployed by themselves, there is no strong attachment. And this is the first version it published. What do you think @benjaminhuo @wenchajun

wanjunlei · 2022-01-25T08:05:45Z

Please change the director api to apis.

zhu733756 · 2022-01-25T08:21:11Z

Please change the director api to apis.

OK

benjaminhuo · 2022-01-25T09:22:59Z

We'd better combine operator of fluentbit and fluentd together into one single deployment/image with fluentbit/fluentd crds/controllers in it and then rename the project to fluent-operator

zhu733756 · 2022-01-25T11:19:15Z

We'd better combine operator of fluentbit and fluentd together into one single deployment/image with fluentbit/fluentd crds/controllers in it and then rename the project to fluent-operator

I will combine the two operators into one

benjaminhuo · 2022-01-25T10:44:51Z

api/fluentdoperator/v1alpha1/clusterfluentdconfig_types.go

+	// Sticky tags will match only one record from an event stream. The same tag will be treated the same way.
+	// will make no effect if EnableFilterKubernetes is set false.
+	StickyTags string `json:"stickyTags,omitempty"`
+	// Comma separated list of hosts. Ignored if left empty.


hosts => namespaces

benjaminhuo · 2022-01-25T10:46:14Z

api/fluentdoperator/v1alpha1/clusterfluentdconfig_types.go

+	// will make no effect if EnableFilterKubernetes is set false.
+	StickyTags string `json:"stickyTags,omitempty"`
+	// Comma separated list of hosts. Ignored if left empty.
+	WatchedNamespaces string `json:"watchedNamespaces,omitempty"`


Is it better to use []string for WatchedNamespaces, WatchedHosts, WatchedContainers?

benjaminhuo · 2022-01-25T11:01:41Z

api/fluentdoperator/v1alpha1/fluentd_types.go

+	// Fluentd Watcher command line arguments.
+	Args []string `json:"args,omitempty"`
+	// MatchCfgSelector defines the selectors to select the fluentd config CRs.
+	MatchCfgSelector metav1.LabelSelector `json:"matchCfgSelector,omitempty"`


Call MatchCfgSelector FluentdConfigSelector is better?

I wonder whether FluentdConfigSelector is a bit confusing since there also has a ClusterFluentdConfig.

FluentdCfgSelector?

Seems better.

benjaminhuo · 2022-01-25T11:16:07Z

api/fluentdoperator/v1alpha1/helper.go

+}
+
+func (pgr *PluginResources) CombineGlobalInputsPlugins(sl plugins.SecretLoader, inputs []input.Input) []string {
+	errs := make([]string, 0)


Is the []input are sorted to make sure the config generated are the same if no changes are made for every reconcile?
The same concerns applies to outputs/filters/configs

I will fix it.

benjaminhuo · 2022-01-25T12:03:15Z

api/fluentdoperator/v1alpha1/helper.go

+// PatchAndFilterClusterLevelResources will combine and patch all the cluster CRs that the fluentdconfig selected,
+// convert the related filter/output pluginstores to the global pluginresources.
+func (pgr *PluginResources) PatchAndFilterClusterLevelResources(sl plugins.SecretLoader, cfgId string,
+	clusterfilters []ClusterFilter, clusteroutputs []ClusterOutput) (*CfgResources, []string) {


clusterfilters/clusteroutputs/filtersList/outputsList all should be sorted using the same standard to make sure the config generated are the same for every reconcile.

filtersList/outputsList => filters/outputs ?

benjaminhuo · 2022-01-25T12:25:28Z

cmd/fluent-manager/fluentbit/Dockerfile

+
+# Use distroless as minimal base image to package the manager binary
+# Refer to https://github.com/GoogleContainerTools/distroless for more details
+FROM gcr.io/distroless/static:nonroot


We'd better sync gcr distroless to https://hub.docker.com/repository/docker/kubesphere/distroless-static and use that instead

Let's replace it when the dockerhub address supports both arm/amd arches in one manifest. Because we use buildx to push image.

now it's multi-arch

benjaminhuo · 2022-01-25T12:28:57Z

This is a great enhancement to the upcoming fluent-operator, thanks @zhu733756

patrick-stephens

I think pretty good initial commit but I think documentation and testing need a fair bit of work. However, this could be done with future PRs.
We also need to consider how to integrate into Github actions well but also allow local development.

My concern is particularly the CRDs are not well spec'd so hard to use - testing will both stabilise them but also highlight issues and usage.
CRD upgrade is hard so we want to avoid it as much as possible and get them good to start with rather than evolve when people have them deployed.
kubectl explain ... should give a user enough information to fill in their CRD values, having to refer to somewhere else for simple questions is not good. It is also something that shows up in editors, etc. so really very helpful to have good context-sensitive documentation.

patrick-stephens · 2022-01-25T20:30:28Z

Makefile

@@ -1,7 +1,9 @@
 VERSION?=$(shell cat VERSION | tr -d " \t\n\r")
 # Image URL to use all building/pushing image targets
 FB_IMG ?= kubesphere/fluent-bit:v1.8.3
-OP_IMG ?= kubesphere/fluentbit-operator:$(VERSION)
+FD_IMG ?= kubesphere/fluentd:v1.14.4


Probably need to sort these to use the fluent organisation ones and likely latest too for dev but with tagged versions for release.

Yeah, we'll change this eventually maybe after we finish adjusting the CI

patrick-stephens · 2022-01-25T20:32:40Z

Makefile

-	go run cmd/manager/main.go
+	go run cmd/fluent-manager/main.go
+
+# Build amd64/arm64 Fluent Operator container image


I can't remember if there was a specific reason we could not support arm32 or just that it is uncommon for K8S? That may change though...

We have few user requirements for arm32 version actually, but we can provide that if there is such requirement in the fluent community

patrick-stephens · 2022-01-25T20:35:56Z

Makefile

+
+# Build amd64 Fluent Operator container image
+build-op-amd64:
+	docker build -f cmd/fluent-manager/Dockerfile . -t ${FO_IMG}${AMD64}


Bit of a mix of BuildKit and classic docker, although this depends on whether DOCKER_BUILDKIT is set as well. Is the intention just to do a local build of the operator images then push later?
Why do we build & push the others rather than offer the same? Just because they're more stable or would we want to use locally too during dev?

patrick-stephens · 2022-01-25T20:42:28Z

apis/fluentbit/v1alpha2/filter_types.go

+// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
+// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.
+
+// FilterSpec defines the desired state of Filter


Probably a general comment but I think this needs a lot more documentation in the comments with examples, defaults, etc. well covered for every field and structure. It should explain usage rather than just being a simple data dictionary, if there are any rules or exclusions to consider they should be covered. Linkage to in-depth documentation and examples as well.

I'm assuming this all feeds into the CRD as well so we definitely want that being very well documented. The goal should be that kubectl explain ... gives the user enough information to actually use the CRD rather than having to reference out to other documentation.

Agree, we need better comments

patrick-stephens · 2022-01-25T20:43:53Z

apis/fluentbit/v1alpha2/filter_types.go

+// +kubebuilder:object:root=true
+// +genclient
+
+// Filter defines a Filter configuration.


This is the kind of comment that should have a linkage to Fluent documentation explaining what filters are but also expanding here to provide a bit more detail to help users of kubectl explain .... Are there any things to watch out for, etc.?

patrick-stephens · 2022-01-25T20:45:37Z

apis/fluentbit/v1alpha2/filter_types.go

+}
+
+type FilterItem struct {
+	// Grep defines Grep Filter configuration.


These comments are pretty self-explanatory, they should link out at the very least to reference documentation I think. Currently they do not provide much help to the user, the name tells as much as the comment.

patrick-stephens · 2022-01-25T20:48:13Z

apis/fluentbit/v1alpha2/filter_types.go

+	FilterItems []FilterItem `json:"filters,omitempty"`
+}
+
+type FilterItem struct {


Potentially you can define a FilterItem with multiple fields in it, what happens then? I think the intent is this is similar to a union in that you should only have one of the internal fields set - is that right? Having a single item of multiple filters seems strange if not. Either way I think it needs more documentation explaining what it is and how to create/use it.

patrick-stephens · 2022-01-25T20:51:42Z

apis/fluentbit/v1alpha2/filter_types.go

+	Match string `json:"match,omitempty"`
+	// A regular expression to match against the tags of incoming records.
+	// Use this option if you want to use the full regex syntax.
+	MatchRegex string `json:"matchRegex,omitempty"`


I think Match and MatchRegex are mutually exclusive but also required - you must have one and only one? Again, needs a bit more documentation here I think. Is this the right type structure to show that? They're both strings so you could have a regex as well in either so would it be better to have a specific type that is one or the other?

patrick-stephens · 2022-01-25T20:53:41Z

apis/fluentbit/v1alpha2/fluentbitconfig_types_test.go

+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+var expected = `[Service]


We probably want to expand this a bit to some data driven tests with the config provided via files or similar. We need to exercise a few different combinations and failure cases I think.

patrick-stephens · 2022-01-25T20:55:29Z

cmd/fluent-watcher/fluentd/Dockerfile.complete

@@ -0,0 +1,76 @@
+# Fluentd watcher agent
+FROM golang:1.13.6-alpine3.11 as buildergo


Probably need to update Golang version here

zhu733756 · 2022-01-26T01:06:31Z

I think pretty good initial commit but I think documentation and testing need a fair bit of work. However, this could be done with future PRs. We also need to consider how to integrate into Github actions well but also allow local development.

My concern is particularly the CRDs are not well spec'd so hard to use - testing will both stabilise them but also highlight issues and usage. CRD upgrade is hard so we want to avoid it as much as possible and get them good to start with rather than evolve when people have them deployed. kubectl explain ... should give a user enough information to fill in their CRD values, having to refer to somewhere else for simple questions is not good. It is also something that shows up in editors, etc. so really very helpful to have good context-sensitive documentation.

@patrick-stephens Good suggestions！ We would add multi images/binaries build, enhance docs explanations, and e2e tests to the schedule marked by an issue. Someone would like to contribute it.

zhu733756 · 2022-01-26T04:29:03Z

.github/workflows/build-pr.yaml

@@ -1,94 +0,0 @@
-name: Build-PullRequest


Seems useless, removed. Will add it later.

zhu733756 · 2022-01-26T04:35:28Z

Hi folks, come to #190 to implement the to-do list.

benjaminhuo · 2022-01-26T04:45:29Z

That's good. Please also take a look at @patrick-stephens‘ comments and add todo items into #190 if necessary

benjaminhuo · 2022-01-26T05:11:58Z

config/crd/patches/cainjection_in_clusterfluentdconfigs.yaml

+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: clusterfluentdconfigs.kubesphere.io


clusterfluentdconfigs.kubesphere.io should be clusterfluentdconfigs.fluentd.fluent.io?

benjaminhuo · 2022-01-26T05:12:35Z

config/crd/patches/cainjection_in_clusterfluentds.yaml

+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: clusterfluentds.kubesphere.io


Do we have clusterfluentds crd?

benjaminhuo · 2022-01-26T05:13:04Z

config/crd/patches/cainjection_in_clusterinputs.yaml

+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: clusterinputs.kubesphere.io


we have clusterinputs crd?

benjaminhuo · 2022-01-26T05:14:43Z

config/crd/patches/webhook_in_clusterfilters.yaml

+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: clusterfilters.kubesphere.io


Please check all crd/patches file if the domain is correct: clusterfilters.kubesphere.io?
These files are auto-generated?

benjaminhuo · 2022-01-26T05:23:12Z

config/rbac/clusterfilter_editor_role.yaml

+  name: clusterfilter-editor-role
+rules:
+- apiGroups:
+  - fluentbit.fluent.io


fluentbit.fluent.io doesn't have clusterfilters now

benjaminhuo · 2022-01-26T05:34:20Z

config/samples/fluentd_v1alpha1_clusterfilter.yaml

@@ -0,0 +1,7 @@
+apiVersion: kubesphere.io/v1alpha1


All the apiVersion in config/samples are not correct

benjaminhuo · 2022-01-26T05:34:51Z

config/samples/fluentd_v1alpha1_clusterinput.yaml

@@ -0,0 +1,7 @@
+apiVersion: kubesphere.io/v1alpha1
+kind: ClusterInput


ClusterInput doesn't exist

benjaminhuo · 2022-01-26T05:47:14Z

manifests/forward-to-fluentd/fluentd-cluster-cfg-output-es.yaml

+  labels:
+    output.fluentd.fluent.io/enabled: "true"
+spec: 
+  outputs: 


we might need to consider if the outputs: layer is necessary and if the type field could be removed。

This part will fix after I come back from the holidays. :)

What I think is that the outputs field is somehow duplicated with the CRD name, maybe we can change it to items?
The same applies to filters.

Your original idea of using an array to store outputs or filters is good, we can keep using the array.
Let's see if it's possible to deal with the type part, it's also ok to keep current implementation if we cannot find a better way

I think the type field combined into the plugin is reasonable, I will try to fix it.

benjaminhuo · 2022-01-26T05:55:30Z

manifests/setup/fluent.io/fluent-operator-clusterRole.yaml

+  labels:
+    app.kubernetes.io/component: controller
+    app.kubernetes.io/name: fluent-operator
+  name: kubesphere:operator:fluent-operator


kubesphere:operator:fluent-operator => fluent-operator is ok?

fix these done

benjaminhuo · 2022-01-26T05:56:12Z

manifests/setup/fluent.io/fluent-operator-clusterRoleBinding.yaml

+  labels:
+    app.kubernetes.io/component: controller
+    app.kubernetes.io/name: fluent-operator
+  name: kubesphere:operator:fluent-operator


kubesphere:operator:fluent-operator => fluent-operator

benjaminhuo · 2022-01-26T06:02:26Z

I think we can check out a branch fluentbit-operator from the current master branch for the legacy fluent-bit operator(with the old kubesphere.io domain) maintenance purpose, maybe someday in the future, this branch can be deleted once the fluent-operator is mature enough.

And then the master branch is used for the development of fluent-operator under the new domain fluent.io

This way, we'll have fewer works to do on each branch and the directory structure will be simplified.

@zhu733756 @wanjunlei @wenchajun

benjaminhuo · 2022-01-26T06:34:27Z

apis/fluentbit/v1alpha2/filter_types.go

+// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
+// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.
+
+// FilterSpec defines the desired state of Filter


Agree, we need better comments

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

patrick-stephens · 2022-01-26T17:08:22Z

I think pretty good initial commit but I think documentation and testing need a fair bit of work. However, this could be done with future PRs. We also need to consider how to integrate into Github actions well but also allow local development.
My concern is particularly the CRDs are not well spec'd so hard to use - testing will both stabilise them but also highlight issues and usage. CRD upgrade is hard so we want to avoid it as much as possible and get them good to start with rather than evolve when people have them deployed. kubectl explain ... should give a user enough information to fill in their CRD values, having to refer to somewhere else for simple questions is not good. It is also something that shows up in editors, etc. so really very helpful to have good context-sensitive documentation.

@patrick-stephens Good suggestions！ We would add multi images/binaries build, enhance docs explanations, and e2e tests to the schedule marked by an issue. Someone would like to contribute it.

Yeah that seems a good idea to me, I would say nothing can go past beta until docs and tests are done. Ideally probably not past alpha as people tend to adopt even beta.
This is a fairly substantial change too so it would be better to get the base in for it then iterate in smaller PRs otherwise it is too large to review.

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 · 2022-01-27T02:40:19Z

charts/fluentbit-operator/crds/fluentbit.fluent.io_inputs.yaml

@@ -211,6 +212,11 @@ spec:
                      messages
                    format: int64
                    type: integer
+                  multilineParser:
+                    description: This will help to reassembly multiline messages originally


These CRDs copies from the generated CRDs with make manifest

benjaminhuo · 2022-01-27T06:31:46Z

Yeah that seems a good idea to me, I would say nothing can go past beta until docs and tests are done. Ideally probably not past alpha as people tend to adopt even beta. This is a fairly substantial change too so it would be better to get the base in for it then iterate in smaller PRs otherwise it is too large to review.

Agree with the comments, a test fluentd config has been added there apis/fluentd/v1alpha1/tests/expected-main-cfgs.cfg

benjaminhuo · 2022-01-27T06:57:56Z

After changing the domain from logging.kubesphere.io to fluent.io, The new directory structure looks better and simpler, thanks @zhu733756!

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 · 2022-02-07T09:29:06Z

Signed-off-by: zhu733756 zhu733756@kubesphere.io

This pr add Fluentd-operator to implement the former proposal #138.

Backgrouds

...

Docs updated. Please help to review the new commit. If no other requested changes, we can push the official images at first, then continue working for the todos #190

benjaminhuo · 2022-02-07T10:05:36Z

The new commit looks good to me, thanks @zhu733756
Would you push fluent-operator:v1.0.0 to your own registry and then I'll sync it to kubesphere, maybe later to fluent org

zhu733756 · 2022-02-07T10:16:01Z

fluent-operator:v1.0.0

OK.

benjaminhuo · 2022-02-07T10:20:10Z

Merge this first, we'll continue to work on items in #190 in separate PRs

zhu733756 requested review from wanjunlei, wenchajun and benjaminhuo January 25, 2022 06:23

zhu733756 force-pushed the fluentd-integration branch 2 times, most recently from fb4506c to 353d18d Compare January 25, 2022 06:39

wenchajun reviewed Jan 25, 2022

View reviewed changes

zhu733756 force-pushed the fluentd-integration branch from bfb0ac1 to b51bda2 Compare January 25, 2022 07:29

zhu733756 commented Jan 25, 2022

View reviewed changes

zhu733756 changed the title ~~Fluentd integration~~ [WIP]Fluentd-operator integration Jan 25, 2022

benjaminhuo requested changes Jan 25, 2022

View reviewed changes

patrick-stephens reviewed Jan 25, 2022

View reviewed changes

zhu733756 force-pushed the fluentd-integration branch 2 times, most recently from 4df73e1 to 3451282 Compare January 26, 2022 04:27

zhu733756 commented Jan 26, 2022

View reviewed changes

.github/workflows/build-pr.yaml

@@ -1,94 +0,0 @@

name: Build-PullRequest

Copy link

Member Author

zhu733756 Jan 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems useless, removed. Will add it later.

zhu733756 mentioned this pull request Jan 26, 2022

Renaming fluentbit-operator to fluent-operator #189 #190

Closed

8 tasks

benjaminhuo requested changes Jan 26, 2022

View reviewed changes

zhu733756 added 2 commits January 27, 2022 00:33

support domain fluent.io && group fluentd API

1d5a484

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

fix and add manifests

d7d1c03

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

suport fluentd watcher hot config reloader

6871e64

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 force-pushed the fluentd-integration branch from 2a6ae15 to 6871e64 Compare January 26, 2022 16:52

fix files naming && controller suite test

c60d911

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 commented Jan 27, 2022

View reviewed changes

Optimize type in CRD

f44774c

Signed-off-by: zhu733756 <zhu733756@kubesphere.io>

zhu733756 force-pushed the fluentd-integration branch from d70f6f5 to f44774c Compare February 7, 2022 09:19

benjaminhuo changed the title ~~[WIP]Fluentd-operator integration~~ Add fluentd support Feb 7, 2022

benjaminhuo merged commit c3f4d88 into fluent:master Feb 7, 2022

zhu733756 deleted the fluentd-integration branch February 7, 2022 10:21

wenchajun mentioned this pull request Feb 10, 2022

Added secret option to fluentbit depployment in helm chart #195

Merged

		@@ -0,0 +1,76 @@
		# Fluentd watcher agent
		FROM golang:1.13.6-alpine3.11 as buildergo

		@@ -0,0 +1,7 @@
		apiVersion: kubesphere.io/v1alpha1
		kind: ClusterInput

Add fluentd support #189

Add fluentd support #189

Conversation

zhu733756 commented Jan 25, 2022 • edited

Backgrouds

The practice part

1 Setup elasticsearch cluster

2 Setup fluent operator

3 Setup fluentbit logging-stack, deleting the fluentbit output plugin

4 Test the forward demo

5 apply fluentbit output forward

6 Check results

Todo

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanjunlei commented Jan 25, 2022

zhu733756 commented Jan 25, 2022

wanjunlei commented Jan 25, 2022

zhu733756 commented Jan 25, 2022

benjaminhuo commented Jan 25, 2022

zhu733756 commented Jan 25, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjaminhuo commented Jan 25, 2022

patrick-stephens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhu733756 commented Jan 26, 2022 • edited

Choose a reason for hiding this comment

zhu733756 commented Jan 26, 2022 • edited

benjaminhuo commented Jan 26, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhu733756 commented Jan 25, 2022 •

edited

zhu733756 commented Jan 25, 2022 •

edited

zhu733756 commented Jan 26, 2022 •

edited

zhu733756 commented Jan 26, 2022 •

edited

benjaminhuo commented Jan 26, 2022 •

edited

benjaminhuo commented Jan 26, 2022 •

edited

benjaminhuo commented Jan 27, 2022 •

edited

zhu733756 commented Feb 7, 2022 •

edited