flowlogs-pipeline integration

README.md

@@ -8,7 +8,7 @@ A Grafana dashboard is also provided.
 
 It is also possible to use without OpenShift:
 - Using the upstream [ovn-kubernetes](https://github.com/ovn-org/ovn-kubernetes/) with any supported Kubernetes flavour ([see below](#ovnk-config) for enabling IPFIX exports on ovn-kubernetes).
-- If you don't use ovn-kubernetes but still can manage having IPFIX exports by a different mean, you're more on your own, but still should be able to use this operator. You will need to configure the IPFIX export to push flows to the `goflow-kube` service deployed by this operator. You could also consider using [goflow-kube](https://github.com/netobserv/goflow2-kube-enricher) directly.
+- If you don't use ovn-kubernetes but still can manage having IPFIX exports by a different mean, you're more on your own, but still should be able to use this operator. You will need to configure the IPFIX export to push flows to the `flowlogs-pipeline` component deployed by this operator. You could also consider using [flowlogs-pipeline](https://github.com/netobserv/flowlogs-pipeline) directly.
 
 The operator itself is deployed in the namespace "network-observability", whereas managed components are deployed in a namespace configured via a Custom Resource (see [FlowCollector custom resource](#flowcollector-custom-resource) section below).
 
@@ -144,7 +144,7 @@ If you use OpenShift 4.10, you don't have anything to do: the operator will conf
 ### With upstream ovn-kubernetes (e.g. using KIND)
 
 ```bash
-GF_IP=`kubectl get svc goflow-kube -n network-observability -ojsonpath='{.spec.clusterIP}'` && echo $GF_IP
+GF_IP=`kubectl get svc flowlogs-pipeline -n network-observability -ojsonpath='{.spec.clusterIP}'` && echo $GF_IP
 kubectl set env daemonset/ovnkube-node -c ovnkube-node -n ovn-kubernetes OVN_IPFIX_TARGETS="$GF_IP:2055"
 ```
 
@@ -153,15 +153,15 @@ kubectl set env daemonset/ovnkube-node -c ovnkube-node -n ovn-kubernetes OVN_IPF
 In OpenShift, a difference with the upstream `ovn-kubernetes` is that the flows export config is managed by the `ClusterNetworkOperator`.
 
 ```bash
-GF_IP=`oc get svc goflow-kube -n network-observability -ojsonpath='{.spec.clusterIP}'` && echo $GF_IP
+GF_IP=`oc get svc flowlogs-pipeline -n network-observability -ojsonpath='{.spec.clusterIP}'` && echo $GF_IP
 oc patch networks.operator.openshift.io cluster --type='json' -p "[{'op': 'add', 'path': '/spec', 'value': {'exportNetworkFlows': {'ipfix': { 'collectors': ['$GF_IP:2055']}}}}]"
 ```
 
 ## Installing Loki
 
 Loki is used to store the flows, however its installation is not managed directly by the operator. There are several options to install Loki, like using the `loki-operator` or the helm charts. Get some help about it on [this page](https://github.com/netobserv/documents/blob/main/hack_loki.md).
 
-Once Loki is setup, you may have to update the `flowcollector` CR to update the Loki URL (use an URL that is accessible in-cluster by the `goflow-kube` pods; default is `http://loki:3100/`).
+Once Loki is setup, you may have to update the `flowcollector` CR to update the Loki URL (use an URL that is accessible in-cluster by the `flowlogs-pipeline` pods; default is `http://loki:3100/`).
 
 ## Enabling the console plugin
 

api/v1alpha1/flowcollector_types.go

@@ -28,24 +28,24 @@ type FlowCollectorSpec struct {
 	// Important: Run "make generate" to regenerate code after modifying this file
 
 	//+kubebuilder:default:=""
-	// Namespace where console plugin and goflowkube pods are going to be deployed.
+	// Namespace where console plugin and collector pods are going to be deployed.
 	// If empty, the namespace of the operator is going to be used
 	Namespace string `json:"namespace,omitempty"`
 
 	// IPFIX contains IPFIX-related settings for the flow reporter
 	IPFIX FlowCollectorIPFIX `json:"ipfix,omitempty"`
 
-	// GoflowKube contains settings related to goflow-kube
-	GoflowKube FlowCollectorGoflowKube `json:"goflowkube,omitempty"`
+	// FlowlogsPipeline contains settings related to the flowlogs-pipeline component
+	FlowlogsPipeline FlowCollectorFLP `json:"flowlogsPipeline,omitempty"`
 
 	// Loki contains settings related to the loki client
 	Loki FlowCollectorLoki `json:"loki,omitempty"`
 
 	// ConsolePlugin contains settings related to the console dynamic plugin
 	ConsolePlugin FlowCollectorConsolePlugin `json:"consolePlugin,omitempty"`
 
-	// CNO contains settings related to the cluster network operator
-	CNO ClusterNetworkOperator `json:"cno,omitempty"`
+	// ClusterNetworkOperator contains settings related to the cluster network operator
+	ClusterNetworkOperator ClusterNetworkOperator `json:"clusterNetworkOperator,omitempty"`
 }
 
 // FlowCollectorIPFIX defines the desired IPFIX state of FlowCollector
@@ -68,8 +68,8 @@ type FlowCollectorIPFIX struct {
 	Sampling int32 `json:"sampling,omitempty" mapstructure:"sampling,omitempty"`
 }
 
-// FlowCollectorGoflowKube defines the desired goflow-kube state of FlowCollector
-type FlowCollectorGoflowKube struct {
+// FlowCollectorFLP defines the desired flowlogs-pipeline state of FlowCollector
+type FlowCollectorFLP struct {
 	// Important: Run "make generate" to regenerate code after modifying this file
 
 	//+kubebuilder:validation:Enum=DaemonSet;Deployment
@@ -100,7 +100,7 @@ type FlowCollectorGoflowKube struct {
 	// HealthPort is a collector HTTP port in the Pod that exposes the health check API
 	HealthPort int32 `json:"healthPort,omitempty"`
 
-	//+kubebuilder:default:="quay.io/netobserv/goflow2-kube:main"
+	//+kubebuilder:default:="quay.io/netobserv/flowlogs-pipeline:main"
 	// Image is the collector image (including domain and tag)
 	Image string `json:"image,omitempty"`
 
@@ -120,8 +120,12 @@ type FlowCollectorGoflowKube struct {
 	// +optional
 	Resources corev1.ResourceRequirements `json:"resources,omitempty" protobuf:"bytes,8,opt,name=resources"`
 
+	//+kubebuilder:default:=true
+	// EnableKubeProbes is a flag to enable or disable Kubernetes liveness/readiness probes
+	EnableKubeProbes bool `json:"enableKubeProbes,omitempty"`
+
 	//+kubebuilder:default:=false
-	// PrintOutput is a debug flag to print flows exported in kube-enricher logs
+	// PrintOutput is a debug flag to print flows exported in flowlogs-pipeline stdout
 	PrintOutput bool `json:"printOutput,omitempty"`
 }
 
@@ -236,7 +240,7 @@ type ClusterNetworkOperator struct {
 type FlowCollectorStatus struct {
 	// Important: Run "make" to regenerate code after modifying this file
 
-	// Namespace where console plugin and goflowkube have been deployed.
+	// Namespace where console plugin and flowlogs-pipeline have been deployed.
 	Namespace string `json:"namespace,omitempty"`
 }
 

bundle/manifests/flows.netobserv.io_flowcollectors.yaml

@@ -39,9 +39,9 @@ spec:
           spec:
             description: FlowCollectorSpec defines the desired state of FlowCollector
             properties:
-              cno:
-                description: CNO contains settings related to the cluster network
-                  operator
+              clusterNetworkOperator:
+                description: ClusterNetworkOperator contains settings related to the
+                  cluster network operator
                 properties:
                   namespace:
                     default: openshift-network-operator
@@ -107,9 +107,15 @@ spec:
                         type: object
                     type: object
                 type: object
-              goflowkube:
-                description: GoflowKube contains settings related to goflow-kube
+              flowlogsPipeline:
+                description: FlowlogsPipeline contains settings related to the flowlogs-pipeline
+                  component
                 properties:
+                  enableKubeProbes:
+                    default: true
+                    description: EnableKubeProbes is a flag to enable or disable Kubernetes
+                      liveness/readiness probes
+                    type: boolean
                   healthPort:
                     default: 8080
                     description: HealthPort is a collector HTTP port in the Pod that
@@ -146,7 +152,7 @@ spec:
                     - maxReplicas
                     type: object
                   image:
-                    default: quay.io/netobserv/goflow2-kube:v0.1.1
+                    default: quay.io/netobserv/flowlogs-pipeline:v0.1.1
                     description: Image is the collector image (including domain and
                       tag)
                     type: string
@@ -192,7 +198,7 @@ spec:
                   printOutput:
                     default: false
                     description: PrintOutput is a debug flag to print flows exported
-                      in kube-enricher logs
+                      in flowlogs-pipeline stdout
                     type: boolean
                   replicas:
                     default: 1
@@ -316,7 +322,7 @@ spec:
                 type: object
               namespace:
                 default: ""
-                description: Namespace where console plugin and goflowkube pods are
+                description: Namespace where console plugin and collector pods are
                   going to be deployed. If empty, the namespace of the operator is
                   going to be used
                 type: string
@@ -325,8 +331,8 @@ spec:
             description: FlowCollectorStatus defines the observed state of FlowCollector
             properties:
               namespace:
-                description: Namespace where console plugin and goflowkube have been
-                  deployed.
+                description: Namespace where console plugin and flowlogs-pipeline
+                  have been deployed.
                 type: string
             type: object
         type: object

bundle/manifests/netobserv-operator.clusterserviceversion.yaml

@@ -11,17 +11,18 @@ metadata:
             "name": "cluster"
           },
           "spec": {
-            "cno": {
+            "clusterNetworkOperator": {
               "namespace": "openshift-network-operator"
             },
             "consolePlugin": {
               "image": "quay.io/netobserv/network-observability-console-plugin:v0.1.1",
               "imagePullPolicy": "IfNotPresent",
               "port": 9001
             },
-            "goflowkube": {
+            "flowlogsPipeline": {
+              "enableKubeProbes": true,
               "healthPort": 8080,
-              "image": "quay.io/netobserv/goflow2-kube:v0.1.1",
+              "image": "quay.io/netobserv/flowlogs-pipeline:v0.1.1",
               "imagePullPolicy": "IfNotPresent",
               "kind": "Deployment",
               "logLevel": "info",
@@ -52,13 +53,13 @@ metadata:
       ]
     capabilities: Basic Install
     categories: Monitoring
-    containerImage: quay.io/netobserv/network-observability-operator:0.1.1
-    createdAt: "2022-03-07T09:18:29Z"
+    containerImage: quay.io/netobserv/network-observability-operator:0.1.1-rc0
+    createdAt: "2022-03-01T14:33:59Z"
     description: A network observability operator based on netflows (IPFIX) for OVN
     operators.operatorframework.io/builder: operator-sdk-v1.16.0+git
     operators.operatorframework.io/project_layout: go.kubebuilder.io/v3
     repository: https://github.com/netobserv/network-observability-operator
-  name: netobserv-operator.v0.1.1
+  name: netobserv-operator.v0.1.1-rc0
   namespace: placeholder
 spec:
   apiservicedefinitions: {}
@@ -268,14 +269,14 @@ spec:
             matchLabels:
               app: network-observability-operator
               control-plane: controller-manager
-              version: 0.1.1
+              version: 0.1.1-rc0
           strategy: {}
           template:
             metadata:
               labels:
                 app: network-observability-operator
                 control-plane: controller-manager
-                version: 0.1.1
+                version: 0.1.1-rc0
             spec:
               containers:
               - args:
@@ -296,7 +297,7 @@ spec:
                 - --leader-elect
                 command:
                 - /manager
-                image: quay.io/netobserv/network-observability-operator:0.1.1
+                image: quay.io/netobserv/network-observability-operator:0.1.1-rc0
                 imagePullPolicy: IfNotPresent
                 livenessProbe:
                   httpGet:
@@ -395,4 +396,4 @@ spec:
   provider:
     name: Red Hat
     url: https://www.redhat.com
-  version: 0.1.1
+  version: 0.1.1-rc0

config/crd/bases/flows.netobserv.io_flowcollectors.yaml

@@ -37,9 +37,9 @@ spec:
           spec:
             description: FlowCollectorSpec defines the desired state of FlowCollector
             properties:
-              cno:
-                description: CNO contains settings related to the cluster network
-                  operator
+              clusterNetworkOperator:
+                description: ClusterNetworkOperator contains settings related to the
+                  cluster network operator
                 properties:
                   namespace:
                     default: openshift-network-operator
@@ -624,9 +624,15 @@ spec:
                         type: object
                     type: object
                 type: object
-              goflowkube:
-                description: GoflowKube contains settings related to goflow-kube
+              flowlogsPipeline:
+                description: FlowlogsPipeline contains settings related to the flowlogs-pipeline
+                  component
                 properties:
+                  enableKubeProbes:
+                    default: true
+                    description: EnableKubeProbes is a flag to enable or disable Kubernetes
+                      liveness/readiness probes
+                    type: boolean
                   healthPort:
                     default: 8080
                     description: HealthPort is a collector HTTP port in the Pod that
@@ -1155,7 +1161,7 @@ spec:
                     - maxReplicas
                     type: object
                   image:
-                    default: quay.io/netobserv/goflow2-kube:main
+                    default: quay.io/netobserv/flowlogs-pipeline:main
                     description: Image is the collector image (including domain and
                       tag)
                     type: string
@@ -1201,7 +1207,7 @@ spec:
                   printOutput:
                     default: false
                     description: PrintOutput is a debug flag to print flows exported
-                      in kube-enricher logs
+                      in flowlogs-pipeline stdout
                     type: boolean
                   replicas:
                     default: 1
@@ -1325,7 +1331,7 @@ spec:
                 type: object
               namespace:
                 default: ""
-                description: Namespace where console plugin and goflowkube pods are
+                description: Namespace where console plugin and collector pods are
                   going to be deployed. If empty, the namespace of the operator is
                   going to be used
                 type: string
@@ -1334,8 +1340,8 @@ spec:
             description: FlowCollectorStatus defines the observed state of FlowCollector
             properties:
               namespace:
-                description: Namespace where console plugin and goflowkube have been
-                  deployed.
+                description: Namespace where console plugin and flowlogs-pipeline
+                  have been deployed.
                 type: string
             type: object
         type: object

config/crd/patches/version_in_flowcollectors.yaml

@@ -3,5 +3,5 @@
   path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/consolePlugin/properties/image/default
   value: "quay.io/netobserv/network-observability-console-plugin:v0.1.1"
 - op: add
-  path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/goflowkube/properties/image/default
-  value: "quay.io/netobserv/goflow2-kube:v0.1.1"
+  path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/flowlogsPipeline/properties/image/default
+  value: "quay.io/netobserv/flowlogs-pipeline:v0.1.1"

config/crd/patches/version_in_flowcollectors_envtpl.yaml

@@ -3,5 +3,5 @@
   path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/consolePlugin/properties/image/default
   value: "quay.io/netobserv/network-observability-console-plugin:v$VERSION"
 - op: add
-  path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/goflowkube/properties/image/default
-  value: "quay.io/netobserv/goflow2-kube:v$VERSION"
+  path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/flowlogsPipeline/properties/image/default
+  value: "quay.io/netobserv/flowlogs-pipeline:v$VERSION"

config/manager/manager.yaml

@@ -31,7 +31,7 @@ spec:
         - --leader-elect
         image: controller:latest
         name: manager
-        imagePullPolicy: IfNotPresent
+        imagePullPolicy: Always
         securityContext:
           allowPrivilegeEscalation: false
         livenessProbe: