From 3f7e042a4a56d94d1b9d93f8126211c63c2fdac7 Mon Sep 17 00:00:00 2001 From: zhu733756 Date: Thu, 10 Feb 2022 18:13:45 +0800 Subject: [PATCH] update architecture diagram Signed-off-by: zhu733756 --- docs/images/fluent-operator.svg | 3 +- docs/proposals/fluent_operator_20210907.md | 69 ++++++++++++++-------- 2 files changed, 47 insertions(+), 25 deletions(-) diff --git a/docs/images/fluent-operator.svg b/docs/images/fluent-operator.svg index de8b759b3..b11317c8f 100644 --- a/docs/images/fluent-operator.svg +++ b/docs/images/fluent-operator.svg @@ -1,3 +1,4 @@ + -
Fluentd : Log Aggregation & Forwarding
Fluentd : Log Aggregation & Forwarding
Fluentd CRDs
Fluentd CRDs
FluentBit: Log Collection
FluentBit: Log Collection
Sinks
Sinks
Fluent Operator : Build Cloud Native Log Processing Pipeline
Fluent Operator : Build Cloud Native Log Processing Pipeline
Fluentd
Fluentd
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
FluentdConfig%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%26lt%3Bfont%20color%3D%26quot%3B%23000000%26quot%3B%26gt%3BFluend%20CRD%26lt%3B%2Ffont%26gt%3B%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BfontSize%3D11%3BfontColor%3D%239E9E9E%3Balign%3Dcenter%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22280%22%20y%3D%22600%22%20width%3D%2290%22%20height%3D%2240%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
FluentdConfig%3CmxG...
FluentdClusterConfig
FluentdClusterConfig
Output
Output
ClusterOutput
ClusterOutput
mount secret 
mount secret 
ES
ES
         FluentBit
         FluentBit
         FluentBit
         FluentBit
         FluentBit
         FluentBit
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
Kafka
Kafka
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
Loki
Loki
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
S3
S3
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
...
...
Log Producers
Log Producers
forward
forward
Apps
Apps
mount
mount
mount
mount
secret
secret
forward
forward
forward
forward
      Fluend
http
http
forward
forward
mount
mount
Syslog
Syslog
Syslog
FluentBit CRDs
FluentBit CRDs
FluentBit
FluentBit
FluentBitConfig
FluentBitConfig
FluentBitClusterConfig
FluentBitClusterConfig
Input
Input
Filter
Filter
Output
Output
ClusterParser
ClusterParser
Parser
Parser
ClusterOutput
ClusterOutput
Viewer does not support full SVG 1.1
\ No newline at end of file +
Fluentd : Log Aggregation & Forwarding
Fluentd : Log Aggregation & Forwarding
Fluentd CRDs
Fluentd CRDs
FluentBit: Log Collection
FluentBit: Log Collection
Sinks
Sinks
Fluent Operator : Build Cloud Native Log Processing Pipeline
Fluent Operator : Build Cloud Native Log Processing Pipeline
Fluentd
Fluentd
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
FluentdConfig%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%26lt%3Bfont%20color%3D%26quot%3B%23000000%26quot%3B%26gt%3BFluend%20CRD%26lt%3B%2Ffont%26gt%3B%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BfontSize%3D11%3BfontColor%3D%239E9E9E%3Balign%3Dcenter%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22280%22%20y%3D%22600%22%20width%3D%2290%22%20height%3D%2240%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
FluentdConfig%3CmxGr...
FluentdClusterConfig
FluentdClusterConfig
Output
Output
ClusterOutput
ClusterOutput
mount secret 
mount secret 
ES
ES
         FluentBit
         FluentBit
         FluentBit
         FluentBit
         FluentBit
         FluentBit
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
Kafka
Kafka
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
Loki
Loki
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
S3
S3
%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22strokeColor%3D%23dddddd%3BfillColor%3D%23ffffff%3Bshadow%3D1%3BstrokeWidth%3D1%3Brounded%3D1%3BabsoluteArcSize%3D1%3BarcSize%3D2%3BlabelPosition%3Dcenter%3BverticalLabelPosition%3Dmiddle%3Balign%3Dcenter%3BverticalAlign%3Dbottom%3BspacingLeft%3D0%3BfontColor%3D%23999999%3BfontSize%3D12%3BwhiteSpace%3Dwrap%3BspacingBottom%3D2%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22920%22%20y%3D%22310%22%20width%3D%2280%22%20height%3D%2270%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
%3CmxGraph...
...
...
Log Producers
Log Producers
Forward
Forward
Apps
Apps
mount
mount
mount
mount
secret
secret
Forward
Forward
Forward
Forward
      Fluend
HTTP
HTTP
Forward
Forward
mount
mount
Syslog
Syslog
Syslog
FluentBit CRDs
FluentBit CRDs
FluentBit
FluentBit
FluentBitClusterConfig
FluentBitClusterConfig
ClusterInput
ClusterInput
ClusterFilter
ClusterFilter
ClusterOutput
ClusterOutput
ClusterParser
ClusterParser
Filter
Filter
ClusterFilter
ClusterFilter
Send
Send
Text is not SVG - cannot display
\ No newline at end of file diff --git a/docs/proposals/fluent_operator_20210907.md b/docs/proposals/fluent_operator_20210907.md index 054fd14a3..f37f70ec2 100644 --- a/docs/proposals/fluent_operator_20210907.md +++ b/docs/proposals/fluent_operator_20210907.md @@ -2,29 +2,50 @@ ## Table of contents -- [Introduction](#Introduction) - - [Why adding Fluentd CRDs to Fluent Operator? ](#Why-adding-Fluentd-CRDs-to-Fluent-Operator? ) - - [How to integrate fluentd ?](#How-to-integrate-fluentd) - - [Introduction to the types of CRD](#Introduction-to-the-types-of-CRD-and-workflow) - - [Support for namespaces isolation and cloud native selectors](#Support-for-namespaces-isolation-and-cloud-native-selectors) - - [Support for strict configuration validation](#Support-for-strict-configuration-validation) - - [Select popular Fluentd plugins to add CRD support](#Select-popular-fluentd-plugins-to-add-crd-support) - - [Support for custom plugins installation](#Support-for-custom-plugins-installation) - - [Enable filesystem pvc for buffer](#Enable-filesystem-pvc-for-buffer) - - [Add ClusterDirectives CRD to support directives predefined](#Add-ClusterDirectives-CRD-to-support-directives-predefined) -- [Fluentd CRDs Design](#Fluentd-CRDs-Design) - - [Fluentd CRD](#Fluentd-CRD) - - [FluentdClusterConfig CRD Or FluentdConfig CRD](#FluentdClusterConfig-CRD-Or-FluentdConfig-CRD) - - [Output CRD Or ClusterOutput CRD](#Output-CRD-Or-ClusterOutput-CRD) - - [Configuration Supplement](#Configuration-Supplement) - - [Common Directive Configuration Supplement](#Common-Directive-Configuration-Supplement) - - [Buffer Directive Configuration Supplement](#Buffer-Directive-Configuration-Supplement) - - [Format Directive Configuration Supplement](#Format-Directive-Configuration-Supplement) - - [Inject Directive Configuration Supplement](#Inject-Directive-Configuration-Supplement) - - [Transport Directive Configuration Supplement](#Transport-Directive-Configuration-Supplement) - - [Storage Directive Configuration Supplement](#Storage-Directive-Configuration-Supplement) - - [Service Discovery Directive Configuration Supplement](#Service-Discovery-Directive-Configuration-Supplement) -- [Parts that need to be discussed or added](#Parts-that-need-to-be-discussed-or-added) +- [Proposal: Add Fluentd to Fluent Operator as an optional log aggregation and forwarding layer](#proposal-add-fluentd-to-fluent-operator-as-an-optional-log-aggregation-and-forwarding-layer) + - [Table of contents](#table-of-contents) + - [Introduction](#introduction) + - [FluentBit CRDs refactoring](#fluentbit-crds-refactoring) + - [Why adding Fluentd CRDs to Fluent Operator?](#why-adding-fluentd-crds-to-fluent-operator) + - [How to integrate fluentd?](#how-to-integrate-fluentd) + - [Introduction to the types of CRD and workflow:](#introduction-to-the-types-of-crd-and-workflow) + - [Support for namespaces isolation and cloud native selectors](#support-for-namespaces-isolation-and-cloud-native-selectors) + - [Support for strict configuration validation](#support-for-strict-configuration-validation) + - [Select popular Fluentd plugins to add CRD support](#select-popular-fluentd-plugins-to-add-crd-support) + - [Support for custom plugins installation](#support-for-custom-plugins-installation) + - [Enable filesystem pvc for buffer](#enable-filesystem-pvc-for-buffer) + - [Add ClusterDirectives CRD to support directives predefined](#add-clusterdirectives-crd-to-support-directives-predefined) + - [Fluentd CRDs Design](#fluentd-crds-design) + - [Fluentd CRD](#fluentd-crd) + - [FluentdClusterConfig CRD Or FluentdConfig CRD](#fluentdclusterconfig-crd-or-fluentdconfig-crd) + - [Input Section Of FluentdClusterConfig CRD or FluentdConfig CRD](#input-section-of-fluentdclusterconfig-crd-or-fluentdconfig-crd) + - [forward](#forward) + - [http](#http) + - [Filters Section Of the FluentdClusterConfig Crd or FluentdConfig Crd](#filters-section-of-the-fluentdclusterconfig-crd-or-fluentdconfig-crd) + - [record_transformer](#record_transformer) + - [grep](#grep) + - [parser](#parser) + - [stdout](#stdout) + - [Output CRD Or ClusterOutput CRD](#output-crd-or-clusteroutput-crd) + - [forward](#forward-1) + - [http](#http-1) + - [stdout](#stdout-1) + - [kafka](#kafka) + - [elasticsearch](#elasticsearch) + - [s3](#s3) + - [Configuration Supplement](#configuration-supplement) + - [Common Directive Configuration Supplement](#common-directive-configuration-supplement) + - [type](#type) + - [id](#id) + - [log_level](#log_level) + - [label](#label) + - [Buffer Directive Configuration Supplement](#buffer-directive-configuration-supplement) + - [Format Directive Configuration Supplement](#format-directive-configuration-supplement) + - [Inject Directive Configuration Supplement](#inject-directive-configuration-supplement) + - [Transport Directive Configuration Supplement](#transport-directive-configuration-supplement) + - [Storage Directive Configuration Supplement](#storage-directive-configuration-supplement) + - [Service Discovery Directive Configuration Supplement](#service-discovery-directive-configuration-supplement) + - [Parts that need to be discussed or added](#parts-that-need-to-be-discussed-or-added) ## Introduction @@ -43,7 +64,7 @@ Now all FluentBit CRDs are namespace level. We should use cluster level CRDs for The following table from [Fluent Bit documentation](https://docs.fluentbit.io/manual/about/fluentd-and-fluent-bit) describes a comparison in different areas of the two projects: -![image-20210618114950689](https://raw.githubusercontent.com/zhu733756/bedpic/main/images/image-20210618114950689.png) +![image-20210618114950689](https://raw.githubusercontent.com/fluent/fluentbit-operator/master/docs/images/fluent-operator.svg) The rich plugins is one of the reasons, which helps to do better handling of logging in some complex scenarios.