SumoLogic · samjsong · Jun 5, 2019 · Jun 4, 2019 · Jun 4, 2019 · Jun 4, 2019
diff --git a/deploy/README.md b/deploy/README.md
@@ -65,41 +65,44 @@ This approach requires access to the Sumo Logic Collector API. It will create a
 
 ```sh
 curl -s https://raw.githubusercontent.com/SumoLogic/sumologic-kubernetes-collection/master/deploy/kubernetes/setup.sh \
-  | bash -s <api-endpoint> <access-id> <access-key> [collector-name]
+  | bash -s [-c collector-name] [-k cluster-name] <api-endpoint> <access-id> <access-key>
 ```
 
 __NOTE__ This script will be executed in bash and requires [jq command-line JSON parser](https://stedolan.github.io/jq/download/) to be installed.
 
 #### Parameters
 
+* __-c collector-name__ - optional. Name of Sumo collector that will be created. If not specified, it will be named as `kubernetes-<timestamp>`
+* __-k cluster-name__ - optional. Name of the Kubernetes cluster that will be attached to logs and events as metadata. If not specified, it will be named as `kubernetes-<timestamp>`. For metrics, specify the cluster name in the `overrides.yaml` provided for the prometheus operator; further details in [step 2](#step-2-configure-prometheus).
 * __api-endpoint__ - required. The API endpoint from [this page](https://help.sumologic.com/APIs/General-API-Information/Sumo-Logic-Endpoints-and-Firewall-Security).
 * __access-id__ - required. Sumo [access id](https://help.sumologic.com/Manage/Security/Access-Keys)
 * __access-key__ - required. Sumo [access key](https://help.sumologic.com/Manage/Security/Access-Keys)
-* __collector-name__ - optional. Name of Sumo collector that will be created. If not specified, it will be named as `kubernetes-<timestamp>`
 
 ### Manual Source Creation and Setup
 
 This is a manual alternative approach to the automatic script if you don't have API access or need customized configuration, such as reusing an existing collector.
 
 #### 1.1 Create a hosted collector and an HTTP source
 
-In this step you create a Sumo Logic hosted collector with a set of HTTP sources to receive your Kubernetes metrics.
+In this step you create a Sumo Logic hosted collector with a set of HTTP sources to receive your Kubernetes data.
 
 Create a hosted collector, following the instructions on [Configure a Hosted Collector](https://help.sumologic.com/03Send-Data/Hosted-Collectors/Configure-a-Hosted-Collector) in Sumo help. If you already have a Sumo hosted collector that you want to use, skip this step.
 
-Create seven HTTP sources under the collector you created in the previous step, one for each of the Kubernetes components that report metrics in this solution:
+Create nine HTTP sources under the collector you created in the previous step, one for each of the Kubernetes components that report metrics in this solution, one for logs, and one for events:
 
-* api-server
-* kubelet
-* controller-manager
-* scheduler
+* api-server-metrics
+* kubelet-metrics
+* controller-manager-metrics
+* scheduler-metrics
 * kube-state-metrics
-* node-exporter
-* default
+* node-exporter-metrics
+* default-metrics
+* logs
+* events
 
 Follow the instructions on [HTTP Logs and Metrics Source](https://help.sumologic.com/03Send-Data/Sources/02Sources-for-Hosted-Collectors/HTTP-Source) to create the sources, with the following additions:
 
-* **Naming the sources.** You can assign any name you like to the sources, but it’s a good idea to assign a name to each source that reflects the Kubernetes component from which it receives metrics. For example, you might name the source that receives API Server metrics “api-server”.
+* **Naming the sources.** You can assign any name you like to the sources, but it’s a good idea to assign a name to each source that reflects the Kubernetes component from which it receives metrics. For example, you might name the source that receives API Server metrics “api-server-metrics”.
 * **HTTP Source URLs.** When you configure each HTTP source, Sumo will display the URL of the HTTP endpoint. Make a note of the URL. You will use it when you configure the Kubernetes service secrets to send data to Sumo.
 
 #### 1.2 Deploy Fluentd
@@ -112,17 +115,19 @@ Run the following command to create namespace `sumologic`
 kubectl create namespace sumologic
 ```
 
-Run the following command to create a Kubernetes secret containing the 7 HTTP source URLs previously created.
+Run the following command to create a Kubernetes secret containing the 9 HTTP source URLs previously created.
 
 ```sh
-kubectl -n sumologic create secret generic metric-endpoints \
+kubectl -n sumologic create secret generic sumologic \
   --from-literal=endpoint-metrics=$ENDPOINT_METRICS \
   --from-literal=endpoint-metrics-apiserver=$ENDPOINT_METRICS_APISERVER \
   --from-literal=endpoint-metrics-kube-controller-manager=$ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER \
   --from-literal=endpoint-metrics-kube-scheduler=$ENDPOINT_METRICS_KUBE_SCHEDULER \
   --from-literal=endpoint-metrics-kube-state=$ENDPOINT_METRICS_KUBE_STATE \
   --from-literal=endpoint-metrics-kubelet=$ENDPOINT_METRICS_KUBELET \
-  --from-literal=endpoint-metrics-node-exporter=$ENDPOINT_METRICS_NODE_EXPORTER
+  --from-literal=endpoint-metrics-node-exporter=$ENDPOINT_METRICS_NODE_EXPORTER \
+  --from-literal=endpoint-logs=$ENDPOINT_LOGS \
+  --from-literal=endpoint-events=$ENDPOINT_EVENTS
 ```
 
 Apply `fluentd-sumologic.yaml` manifest with following command:
@@ -249,8 +254,31 @@ This filter will:
 * Rename the label/metadata `container_name` to `container`, and `pod_name` to `pod`.
 * Only apply to metrics with the `kube-system` namespace
 
+## Step 3: Deploy FluentBit
+
+In this step, you will deploy FluentBit to forward logs to Fluentd.
+
+Download the FluentBit `overrides.yaml` from GitHub:
+
+```sh
+curl -LJO https://raw.githubusercontent.com/SumoLogic/sumologic-kubernetes-collection/master/deploy/fluent-bit/overrides.yaml
+```
+
+Install `fluent-bit` using Helm:
+
+```sh
+helm repo update \
+   && helm install stable/fluent-bit --name fluent-bit --namespace sumologic -f overrides.yaml
+```
+
 ## Tear down
 
+To delete `fluent-bit` from the Kubernetes cluster:
+
+```sh
+helm del --purge fluent-bit
+```
+
 To delete `prometheus-operator` from the Kubernetes cluster:
 
 ```sh
@@ -265,10 +293,10 @@ To delete the `fluentd-sumologic` app:
 kubectl delete -f ./fluentd-sumologic.yaml
 ```
 
-To delete the `metric-endpoints` secrets (for recreating collector/sources):
+To delete the `sumologic` secret (for recreating collector/sources):
 
 ```sh
-kubectl -n sumologic delete secret metric-endpoints
+kubectl -n sumologic delete secret sumologic
 ```
 
 To delete the `sumologic` namespace and all resources under it:

diff --git a/deploy/kubernetes/fluentd-sumologic.yaml b/deploy/kubernetes/fluentd-sumologic.yaml
@@ -328,37 +328,37 @@ spec:
         - name: SUMO_ENDPOINT_METRICS
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics
         - name: SUMO_ENDPOINT_METRICS_APISERVER
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics-apiserver
         - name: SUMO_ENDPOINT_METRICS_KUBELET
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics-kubelet
         - name: SUMO_ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics-kube-controller-manager
         - name: SUMO_ENDPOINT_METRICS_KUBE_SCHEDULER
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics-kube-scheduler
         - name: SUMO_ENDPOINT_METRICS_KUBE_STATE
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics-kube-state
         - name: SUMO_ENDPOINT_METRICS_NODE_EXPORTER
           valueFrom:
             secretKeyRef:
-              name: metric-endpoints
+              name: sumologic
               key: endpoint-metrics-node-exporter
         - name: SUMO_ENDPOINT_LOGS
           valueFrom:

diff --git a/deploy/kubernetes/setup.sh b/deploy/kubernetes/setup.sh
@@ -4,7 +4,7 @@ set -e
 usage() {
   echo
   echo 'Usage:'
-  echo '  setup.sh <endpoint> <access-id> <access-key> [collector-name]'
+  echo '  setup.sh [-c collector-name] [-k cluster-name] <endpoint> <access-id> <access-key>'
   echo
 }
 
@@ -15,8 +15,9 @@ timestamp() {
 create_host_collector()
 {
   _P='{"collector":{"collectorType":"Hosted","name":"'
-  _S='"}}'
-  JSON="$_P$1$_S"
+  _M='","fields":{"cluster":"'
+  _S='"}}}'
+  JSON="$_P$1$_M$2$_S"
   COMMAND="curl -s -u $ACC_ID:$ACC_KEY -X POST -H Content-Type:application/json -d $JSON $SUMO_ENDPOINT/collectors"
   RESULT=$($COMMAND)
   set +e
@@ -51,6 +52,26 @@ create_http_source()
   echo "Source was created(id=$SOURCE_ID, name=$SOURCE_NAME)."
 }
 
+while getopts c:k: option
+do
+ case "${option}"
+ in
+ c) COLLECTOR_NAME=${OPTARG};;
+ k) CLUSTER_NAME=${OPTARG};;
+ esac
+done
+shift "$(($OPTIND -1))"
+
+TIME=`timestamp`;
+
+if [ -z $COLLECTOR_NAME ]; then
+  COLLECTOR_NAME="kubernetes-$TIME";
+fi
+
+if [ -z $CLUSTER_NAME ]; then
+  CLUSTER_NAME="kubernetes-$TIME";
+fi
+
 if [ -n "$1" ]; then
   SUMO_ENDPOINT=${1%/};
 else
@@ -75,13 +96,6 @@ else
   exit -1;
 fi
 
-if [ -n "$4" ]; then
-  COLLECTOR_NAME=$4;
-else
-  TIME=`timestamp`;
-  COLLECTOR_NAME="kubernetes-$TIME";
-fi
-
 set +e
 kubectl describe namespace sumologic &>/dev/null
 retVal=$?
@@ -94,50 +108,58 @@ else
 fi
 
 set +e
-echo "Creating secret 'metric-endpoints'..."
-kubectl -n sumologic describe secret metric-endpoints &>/dev/null
+echo "Creating secret 'sumologic'..."
+kubectl -n sumologic describe secret sumologic &>/dev/null
 retVal=$?
 set -e
 if [ $retVal -eq 0 ]; then
-  echo "Secret 'sumologic::metric-endpoints' exists, abort."
+  echo "Secret 'sumologic::sumologic' exists, abort."
   exit -2;
 fi
 
-echo "Creating collector '$COLLECTOR_NAME'..."
+echo "Creating collector '$COLLECTOR_NAME' for cluster $CLUSTER_NAME..."
 COLLECTOR_ID=
-create_host_collector $COLLECTOR_NAME
+create_host_collector $COLLECTOR_NAME $CLUSTER_NAME
 
 echo "Creating sources in '$COLLECTOR_NAME'..."
 SOURCE_URL=
-create_http_source '(default)' $COLLECTOR_ID
+create_http_source '(default-metrics)' $COLLECTOR_ID
 ENDPOINT_METRICS="$SOURCE_URL"
 SOURCE_URL=
-create_http_source apiserver $COLLECTOR_ID
+create_http_source apiserver-metrics $COLLECTOR_ID
 ENDPOINT_METRICS_APISERVER="$SOURCE_URL"
 SOURCE_URL=
-create_http_source kube-controller-manager $COLLECTOR_ID
+create_http_source kube-controller-manager-metrics $COLLECTOR_ID
 ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER="$SOURCE_URL"
 SOURCE_URL=
-create_http_source kube-scheduler $COLLECTOR_ID
+create_http_source kube-scheduler-metrics $COLLECTOR_ID
 ENDPOINT_METRICS_KUBE_SCHEDULER="$SOURCE_URL"
 SOURCE_URL=
-create_http_source kube-state $COLLECTOR_ID
+create_http_source kube-state-metrics $COLLECTOR_ID
 ENDPOINT_METRICS_KUBE_STATE="$SOURCE_URL"
 SOURCE_URL=
-create_http_source kubelet $COLLECTOR_ID
+create_http_source kubelet-metrics $COLLECTOR_ID
 ENDPOINT_METRICS_KUBELET="$SOURCE_URL"
 SOURCE_URL=
-create_http_source node-exporter $COLLECTOR_ID
+create_http_source node-exporter-metrics $COLLECTOR_ID
 ENDPOINT_METRICS_NODE_EXPORTER="$SOURCE_URL"
+SOURCE_URL=
+create_http_source logs $COLLECTOR_ID
+ENDPOINT_LOGS="$SOURCE_URL"
+SOURCE_URL=
+create_http_source events $COLLECTOR_ID
+ENDPOINT_EVENTS="$SOURCE_URL"
 
-kubectl -n sumologic create secret generic metric-endpoints \
+kubectl -n sumologic create secret generic sumologic \
   --from-literal=endpoint-metrics=$ENDPOINT_METRICS \
   --from-literal=endpoint-metrics-apiserver=$ENDPOINT_METRICS_APISERVER \
   --from-literal=endpoint-metrics-kube-controller-manager=$ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER \
   --from-literal=endpoint-metrics-kube-scheduler=$ENDPOINT_METRICS_KUBE_SCHEDULER \
   --from-literal=endpoint-metrics-kube-state=$ENDPOINT_METRICS_KUBE_STATE \
   --from-literal=endpoint-metrics-kubelet=$ENDPOINT_METRICS_KUBELET \
-  --from-literal=endpoint-metrics-node-exporter=$ENDPOINT_METRICS_NODE_EXPORTER
+  --from-literal=endpoint-metrics-node-exporter=$ENDPOINT_METRICS_NODE_EXPORTER \
+  --from-literal=endpoint-logs=$ENDPOINT_LOGS \
+  --from-literal=endpoint-events=$ENDPOINT_EVENTS
 
 echo "Applying deployment 'fluentd'..."
 kubectl apply -f https://raw.githubusercontent.com/SumoLogic/sumologic-kubernetes-collection/master/deploy/kubernetes/fluentd-sumologic.yaml