diff --git a/README.md b/README.md index b54b789..1fa3b9e 100644 --- a/README.md +++ b/README.md @@ -2,22 +2,19 @@ This repo contains various documents related to network observability (a.k.a. netobserv): ### Repositories -* [network-observability-operator](https://github.com/netobserv/network-observability-operator) +* [netobserv-operator](https://github.com/netobserv/network-observability-operator) OpenShift / Kubernetes operator for network observability. This operator will deloy the following components: - * [goflow2-kube-enricher](https://github.com/netobserv/goflow2-kube-enricher) - This component implements the following libraries: - * [loki-client-go](https://github.com/netobserv/loki-client-go) - An HTTP client to send logs to Loki server - * [goflow2-loki-exporter](https://github.com/netobserv/goflow2-loki-exporter) - A Loki exporter plugin - * [network-observability-console-plugin](https://github.com/netobserv/network-observability-console-plugin) - Network Observability plugin for the Openshift Console - * [nflow-generator](https://github.com/netobserv/nflow-generator) - A fake legacy v5 netflow generator + * [eBPF Agent](https://github.com/netobserv/netobserv-ebpf-agent): An eBPF agent that captures and exports network flows. + * [flowlogs-pipeline](https://github.com/netobserv/flowlogs-pipeline): A configurable flow collector, enricher and metrics producer. + * [netobserv-plugin](https://github.com/netobserv/network-observability-console-plugin): + The NetObserv plugin for the Openshift Console. +* [nflow-generator](https://github.com/netobserv/nflow-generator): +A fake legacy v5 netflow generator +* etc. ### Get started -The easiest way to get started is to use the [Network Observability Operator](https://github.com/netobserv/network-observability-operator) that will deploy all the components for you. +The easiest way to get started is to use the [NetObserv Operator](https://github.com/netobserv/network-observability-operator) that will deploy all the components for you. It is available via [OLM](https://operatorhub.io/operator/netobserv-operator). You can either: - [Deploy an existing image](https://github.com/netobserv/network-observability-operator#deploy-an-existing-image) @@ -25,11 +22,10 @@ You can either: ### Development -You will need [Kind](./kind.md) or [OpenShift](./openshift.md) with ovn-kubernetes configured to get network flows. -For quick development on OCP use [this guide](./noo_dev_ocp.md) +You will need a Kubernetes cluster, such as [Kind](./kind.md) or [OpenShift](./openshift.md). If you don't need the entire stack, you can just [check components](#repositories) above. -### Sample applications +### Other links -[sample_apps.md](./sample_apps.md): a few suggestions to generate traffic, useful for testing NetObserv. +- [Sample apps](./sample_apps.md): a few suggestions to generate traffic, useful for testing NetObserv. diff --git a/examples/distributed-loki/1-prerequisites/secret.yaml b/examples/distributed-loki/1-prerequisites/secret.yaml index 6438c0d..c18588e 100644 --- a/examples/distributed-loki/1-prerequisites/secret.yaml +++ b/examples/distributed-loki/1-prerequisites/secret.yaml @@ -3,7 +3,7 @@ apiVersion: v1 kind: Secret metadata: name: loki-distributed - namespace: network-observability + namespace: netobserv type: Opaque stringData: ACCESS_KEY_ID: XXXXXXXXXXXXXXXXXXXX diff --git a/examples/kowl.yaml b/examples/kowl.yaml index 0ffdc12..4eb5ec3 100644 --- a/examples/kowl.yaml +++ b/examples/kowl.yaml @@ -8,7 +8,7 @@ kowl: kafka: brokers: # Set your bootstrap url. Kowl does domain name validation so be sure to pick a valid url which is covered in the subject alternative name of the certificate of the bootsrap endpoint (e.g. no .svc.cluster.local address). - - kafka-cluster-kafka-bootstrap.network-observability + - kafka-cluster-kafka-bootstrap.netobserv tls: enabled: true caFilepath: /etc/strimzi/ca/ca.crt diff --git a/examples/metrics/monitoring.yaml b/examples/metrics/monitoring.yaml index e6789b3..bda6570 100644 --- a/examples/metrics/monitoring.yaml +++ b/examples/metrics/monitoring.yaml @@ -13,7 +13,7 @@ metadata: labels: app: flowlogs-pipeline name: flowlogs-pipeline-metrics - namespace: network-observability + namespace: netobserv spec: ports: - port: 9102 @@ -28,7 +28,7 @@ apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: flowlogs-pipeline - namespace: network-observability + namespace: netobserv spec: endpoints: - interval: 15s @@ -36,7 +36,7 @@ spec: scheme: http namespaceSelector: matchNames: - - network-observability + - netobserv selector: matchLabels: app: flowlogs-pipeline @@ -45,7 +45,7 @@ apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: netobserv-console-plugin - namespace: network-observability + namespace: netobserv spec: endpoints: - interval: 15s @@ -55,7 +55,7 @@ spec: insecureSkipVerify: true namespaceSelector: matchNames: - - network-observability + - netobserv selector: matchLabels: - app: network-observability-plugin + app: netobserv-plugin diff --git a/kafka.md b/kafka.md index f01083d..94e2b86 100644 --- a/kafka.md +++ b/kafka.md @@ -9,7 +9,7 @@ Simply install it and create a "Kafka" instance in `default` namespace. You can use the following command to deploy the strimzi operator : ``` -export NAMESPACE=network-observability +export NAMESPACE=netobserv kubectl create -f "https://strimzi.io/install/latest?namespace=$NAMESPACE" -n $NAMESPACE ``` @@ -68,8 +68,8 @@ You can use [kfk](https://github.com/systemcraftsman/strimzi-kafka-cli), a CLI f To create a new user with public/private keys for mTLS, and get its secrets: ```bash -kfk users --create --user flp-kafka --authentication-type tls -n network-observability -c kafka-cluster -kubectl describe secret/flp-kafka -n network-observability +kfk users --create --user flp-kafka --authentication-type tls -n netobserv -c kafka-cluster +kubectl describe secret/flp-kafka -n netobserv ``` To create a terminal-based producer and consumer with mTLS, refer to [this page](https://github.com/systemcraftsman/strimzi-kafka-cli/tree/main/examples/2_tls_authentication). diff --git a/loki_distributed.md b/loki_distributed.md index ec94922..df0128c 100644 --- a/loki_distributed.md +++ b/loki_distributed.md @@ -68,8 +68,8 @@ Check [official examples](https://grafana.com/docs/loki/latest/storage/#examples ## Deploy Ensure [storage configuration](#storage-configuration) is up to date. -Run the following command to apply all yamls at once in `network-observability` namespace: -`kubectl apply -f ./examples/distributed-loki/ -n network-observability --recursive` +Run the following command to apply all yamls at once in `netobserv` namespace: +`kubectl apply -f ./examples/distributed-loki/ -n netobserv --recursive` This will create all the components and services described above. @@ -93,10 +93,10 @@ The first argument is the bucket name, second is the AWS region. Example: ## Network Observability Operator -To connect NOO with this config, you will have to update the `loki` section with the following urls assuming you used `network-observability` namespace: +To connect NOO with this config, you will have to update the `loki` section with the following urls assuming you used `netobserv` namespace: ```yaml loki: - url: 'http://loki-distributed-distributor.network-observability.svc.cluster.local:3100/' - querierUrl: 'http://loki-distributed-query-frontend.network-observability.svc.cluster.local:3100/' + url: 'http://loki-distributed-distributor.netobserv.svc.cluster.local:3100/' + querierUrl: 'http://loki-distributed-query-frontend.netobserv.svc.cluster.local:3100/' ``` \ No newline at end of file diff --git a/noo_dev_ocp.md b/noo_dev_ocp.md deleted file mode 100644 index ee63a38..0000000 --- a/noo_dev_ocp.md +++ /dev/null @@ -1,58 +0,0 @@ -# Developing network Operator (on OCP) - -The following instructions are useful for local speedy development of the operator. - -## prerequisites - -1. Local linux machine -2. OCP cluster - -## Deployment process - -1. Connect to the OCP cluster using `oc login ...` -2. execute `kubectl create namespace network-observability` - this creates the `network-observability` namespace -3. execute `kubectl config set-context --current --namespace network-observability` - this changes the `kubectl` context to `network-observability` namespace -4. Execute `make deploy-loki` - this deploys a simple loki instance into the cluster -5. Execute `make deploy-grafana` - this deploys a simple grafana instance into the cluster (user: admin password: admin) -6. Execute `oc patch console.operator.openshift.io cluster --type='json' -p '[{"op": "add", "path": "/spec/plugins", "value": ["network-observability-plugin"]}]'` - this enables the console plugin - - note: the console plug-in itself is deployed as part of `make run` by the operator. This patch can be applied also post deployment of the console plug-in -7. Execute `make install` - this installs the operator CRDs into the cluster -8. Execute `make deploy-sample-cr` - this installs an example of *flows.netobserv.io* instance into the cluster -9. Execute `make run` - this starts running the operator process locally (on the laptop, not inside the cluster) -10. Use `kubectl get pods` to see that the pods are deployed as expected -11. Once the `goflow-kube` pod is ready, on OCP 4.9 and earlier, enable IPFIX collection into `goflow-kube` using: -```bash -GF_IP=`oc get svc goflow-kube -n network-observability -ojsonpath='{.spec.clusterIP}'` && echo $GF_IP -oc patch network.operator cluster --type='json' --patch='[{"op": "add","path": "/spec","value": {"exportNetworkFlows": {"ipfix": { "collectors": ["'"$GF_IP"':2055"]}}}}]' -``` - -## Notes - -> Deployment is done into `network-observability` namespace - -> Once executed operator log are emitted to stdout - -## Code update process - -1. Execute `make undeploy-sample-cr` - this deletes the *flows.netobserv.io* instance. The operator will delete all deployed resources from the cluster -2. Stop the running operator process - after making sure that the resources are removed by the operator in the previous step -3. **Update the operator code** :-) -4. Rerun steps from [Deployment process](#deployment-process) section starting with `make create-sample-cr` - -## Troubleshooting - -- use `kubectl get pods` to observe the list of pods. Expect to see output such as -```bash -$ kubectl get pods -NAME READY STATUS RESTARTS AGE -goflow-kube-577d8c89b5-sll57 1/1 Running 0 77s -loki 1/1 Running 0 92m -network-observability-plugin-676b6f8c8f-f62qz 1/1 Running 0 77s -``` - -> Note: expect pod names with different random IDs - -- use `kubectl exec -it goflow-kube-757dcf5c74-8b24w bash` to start shell inside the `goflow-kube` pod -- use `curl http://loki:3100/ready` to validate that Loki is healthy -- use `curl http://localhost:8080/metrics` to validate that goflow-kube is healthy -- diff --git a/openshift.md b/openshift.md index d679d71..9590b13 100644 --- a/openshift.md +++ b/openshift.md @@ -1,8 +1,5 @@ # Openshift -* [Deploy using OpenShift Container Platform](#deploy-using-openshift-container-platform) -* [Deploy using Cluster Bot](#deploy-using-cluster-bot) - ## Deploy using OpenShift Container Platform The easiest way to get Openshift installed is using [OpenShift Container Platform](https://docs.openshift.com/container-platform/4.8/installing/index.html) @@ -26,9 +23,6 @@ Example for GCP: This allow you to get more ressources in order to deploy Loki for example. -### Set OVN-Kubernetes as default CNI network provider on Openshift -This is mandatory for kube-enricher used in network-observability-operator. - #### Using OCP You can [customize your OCP installation](https://docs.openshift.com/container-platform/4.8/installing/installing_aws/installing-aws-network-customizations.html#installation-initializing_installing-aws-network-customizations) using `install-config.yaml` file. @@ -37,49 +31,6 @@ Generate the configuration using: openshift-install create install-config --dir= ``` -Add [network configuration parameters](https://docs.openshift.com/container-platform/4.8/installing/installing_aws/installing-aws-network-customizations.html#installation-configuration-parameters-network_installing-aws-network-customizations) - -For example: -```yaml -networking: - networkType: OVNKubernetes -``` - -#### Migrate running cluster -Follow the [official guide](https://docs.openshift.com/container-platform/4.8/networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.html#nw-ovn-kubernetes-migration_migrate-from-openshift-sdn) for OVNK networkType migration according to your Openshift version. - -This can take some time and you will need ssh access or infrastructure provider management portal to restart each nodes so you should consider [using OCP install-config.yaml](####using-ocp) - -## Deploy using Cluster Bot - -Cluster Bot is a Slack bot that allows easily deploying short-lived clusters (auto-removed in ~2 -hours) with concrete in-development branches of several Openshift components. - -To use it, just add the `cluster-bot` app in your Slack (usually, in the Apps dropdown from the -Slack left panel) and type `help` in the message Window. It will show you several options. - -The deployment takes around 30 minutes. When it finishes, `cluster-bot` provides you the contents -of the `KUBECONFIG` file as well as the `kubeadmin` password. - -### Examples - -Launch the last 4.x stable version with OVNKubernetes CNI: -`launch 4-stable ovn` - -Launch the latest stable version of OpenShift, but replacing the bundled Console and CNO by the -contents of Pull Requests `#9953` and `#1231`, respectively: - -``` -launch openshift/console#9953,openshift/cluster-network-operator#1231 -``` - -Launch an in-development 4.10-ci version of OpenShift, replacing the `master` branch of the -Console by the Pull Request `#9953`: - -``` -launch 4.10-ci,openshift/console#9953 -``` - ## Metrics NetObserv comes with a bunch of metrics, however they are not scraped by default by OpenShift Cluster Monitoring (the OpenShift cluster Prometheus that is used for infra monitoring).