docs: fluentd for orc8r minikube deployment documentation

Adds the documentation for using fluentd, elasticsearch, and kibana for
the orc8r minikube deployment.

Signed-off-by: Sebastian Wolf <sebastian.wolf@tngtech.com>

docs/readmes/orc8r/dev_build.md

@@ -64,11 +64,19 @@ git checkout -b v1.6 origin/v1.6  # or desired Git target, e.g. master
 ```
 
 Build and publish Orchestrator images
+(Optionally, build and publish fluentd images for the Orchestrator by uncommenting the line `docker-compose.logging.yaml` in `build.py`.)
+
 
 ```bash
 cd ${MAGMA_ROOT}/orc8r/cloud/docker
 ./build.py --all
-for image in controller nginx ; do ${PUBLISH} -r ${REGISTRY} -i ${image} -v ${MAGMA_TAG} ; done
+for image in controller nginx ; do ${PUBLISH} -r ${REGISTRY} -i ${image} -v ${MAGMA_TAG} -u '' -p '' ; done
+```
+
+Optional: build and publish fluentd images
+
+```bash
+for image in fluentd-forward fluentd-daemon ; do ${PUBLISH} -r ${REGISTRY} -i ${image} -v ${MAGMA_TAG} -u '' -p '' ; done
 ```
 
 Build and publish NMS images

docs/readmes/orc8r/dev_minikube.md

@@ -74,6 +74,17 @@ export CERTS_DIR=${MAGMA_ROOT}/.cache/test_certs
 cd ${MAGMA_ROOT}/orc8r/cloud/docker && ./build.py && ./run.py && sleep 30 && docker-compose down && ls -l ${CERTS_DIR} && cd -
 ```
 
+To use fluentd on the minikube deployment, create the fluentd certs with:
+
+```bash
+openssl genrsa -out dp_fluentd.key 2048
+openssl req -new -key dp_fluentd.key -out dp_fluentd.csr -subj "/C=US/CN=dp-fluentd.$domain"
+openssl x509 -req -in dp_fluentd.csr -CA certifier.pem -CAkey certifier.key -CAcreateserial -out dp_fluentd.pem -days 3650 -sha256
+openssl genrsa -out fluentd.key 2048
+openssl req -new -key fluentd.key -out fluentd.csr -subj "/C=US/CN=fluentd.$domain"
+openssl x509 -req -in fluentd.csr -CA certifier.pem -CAkey certifier.key -CAcreateserial -out fluentd.pem -days 3650 -sha256
+```
+
 ### Apply secrets
 
 Create the K8s secrets
@@ -105,6 +116,13 @@ helm template orc8r charts/secrets \
   kubectl apply -f -
 ```
 
+Include the following two lines in the `helm template` command to apply the fluentd secrets:
+
+```bash
+--set-file 'secret.certs.files.fluentd\.pem'=${CERTS_DIR}/fluentd.pem \
+--set-file 'secret.certs.files.fluentd\.key'=${CERTS_DIR}/fluentd.key \
+```
+
 ### Create values file
 
 A minimal values file is at `${MAGMA_ROOT}/orc8r/cloud/helm/orc8r/examples/minikube.values.yaml`
@@ -120,6 +138,26 @@ This section describes how to install based on local charts. However, you can al
 - Stable: <https://artifactory.magmacore.org/artifactory/helm/>
 - Test: <https://artifactory.magmacore.org/artifactory/helm-test/>
 
+Optional: install the fluentd charts by editing the file at `${MAGMA_ROOT}/orc8r/cloud/helm/orc8r/charts/logging/values.yaml`
+
+```yaml
+fluentd_daemon:
+  create: true
+
+  image:
+    repository: IMAGE_REGISTRY_URL
+    tag: IMAGE_TAG
+    pullPolicy: IfNotPresent
+
+  env:
+    elastic_host: "host.minikube.internal"
+    elastic_port: "9200"
+    elastic_scheme: "http"
+```
+
+and equivalent edits for `fluentd_forward`.
+Again, replace `IMAGE_REGISTRY_URL` with your registry and `IMAGE_TAG` with your tag.
+
 Install base `orc8r` chart
 
 ```bash
@@ -156,6 +194,27 @@ orc8r-user-grafana-6498bb6959-rchx5              1/1     Running   0          2m
 postgresql-0                                     1/1     Running   4          6d23h
 ```
 
+If using fluentd, there should be a pod `orc8r-fluentd-forward` running as well. Additionally,
+
+```bash
+kubectl --namespace kube-system get pods
+```
+
+should show a pod `orc8r-fluentd-daemon` running.
+
+Optionally, start the elasticsearch and kibana containers which handle the logs aggregated by fluentd
+
+```bash
+cd ${MAGMA_ROOT}/cloud/docker
+./run.py
+```
+
+### Access logs through Kibana
+
+The Orchestrator logs aggregated by fluentd can be accessed via Kibana in the web browser under `http://localhost:5601/`.
+There should be two index patterns: `fluentd` (from fluentd-forward) and `logstash` (from fluentd-daemon).
+The `logstash` index pattern collects all logs from the Orchestrator, and `fluentd` the logs from connected gateways.
+
 ## Configure
 
 ### Access Orc8r