Forward Kubernetes logs and system metrics to Falcon LogScale with the fluent-bit log shipper.
This integration provides instructions on how to configure the Fluent-Bit log shipper to forward Kubernetes cluster logs and node metrics to Falcon LogScale for ingestion.
This integration was created as a companion to the LogScale Kubernetes Logging Package which is available on the CrowdStrike Lgscale Community Content GitHub repository.
note: The Kubernetes Logging package dashboards rely on specific enrichments added by this integration.
Fluent-Bit is a fast and lightweight end-to-end log processing pipeline that collects, filters, and forwards logs and metrics. It is an open source sub-project of the CNCF.
Fluent-bit is written in C. It provides a variety of plugins to input, filter, and output logging telemetry. It can be run as system level service or as a DaemonSet within Kubernetes.
This integration employs a split horizon setup of fluent-bit in order to capture a complete view of Kubernetes cluster logging telemetry: that is, fluent-bit is run simultaneously as a system service outside of Kubernetes, and as a DaemonSet within Kubernetes.
As a DaemonSet, fluent-bit monitors and forwards well known Kubernetes log sources: container, docker, audit.
As a system service, fluent-bit collects and forwards a variety of node metrics (cpu, network, disk, memory), and Docker events.
The split horizon implementation reflects the current state of fluent-bit namespace contraints among the various fluent-bit input plugins: metrics scraping is currently only possible from the system level where the fluent-bit metrics [INPUT] plugins have access to special device files in /proc and /sys.
Metrics capture is not currently possible when running fluent-bit from within a Kubernetes container.
Logs collected by the DaemonSet are automatic enriched with a set of Kubernetes properties: i.e. "kubernetes": { ... }
{
"@timestamp": "2023-XX-XXTXX:XX:XX.XXXZ",
"log": "[2023/XX/XX XX:XX:XX] [ info] [engine] flush chunk '1-1111111111.111111111.flb' succeeded at retry 2: task_id=7, input=tail.1 > output=es.0 (out_id=0)\n",
"stream": "stderr",
"time": "2023-XX-XXTXX:XX:XX.000000000Z",
"source": "container",
"hostname": "host.name.com",
"kubernetes": {
"pod_name": "fluent-bit-tsv4f",
"namespace_name": "logging",
"pod_id": "nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn",
"labels": {
"app.kubernetes.io/instance": "fluent-bit",
"app.kubernetes.io/name": "fluent-bit",
"controller-revision-hash": "1111111111",
"pod-template-generation": "2"
},
"annotations": {
"checksum/config": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"checksum/luascripts": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"cni.projectcalico.org/containerID": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"cni.projectcalico.org/podIP": "10.10.10.11/32",
"cni.projectcalico.org/podIPs": "10.10.10.11/32",
"kubectl.kubernetes.io/restartedAt": "2023-XX-XXTXX:XX:XX-XX:XX"
},
"host": "host.name.com",
"container_name": "fluent-bit",
"docker_id": "nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"container_hash": "cr.fluentbit.io/fluent/fluent-bit@sha256:nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn",
"container_image": "cr.fluentbit.io/fluent/fluent-bit:2.0.8"
}
}Kubernetes Logging Package
note: the fluentbit parser is included in the package
note: this integration assumes Ubuntu Linux as the target platform
Apt Install Fluent-Bit
sudo apt-get install fluent-bit
Prepare the /etc/fluent-bit/fluent-bit.conf file.
Use the fluent-bit.conf template file included with this integration to source the [INPUT], [FILTER], and [OUTPUT] stanzas.
Update the [OUTPUT] stanza with your LogScale ingest credentials.
note: a fluent-bit timestamp is added to some metrics events that lack native timestamping.
sudo systemctl start fluent-bit
Verify that events are being ingested to the repository.
Use the upstream fluentbit opensource distribution to install a daemonset through a helm chart.
kubectl create namespace logging
Follow the instructions in the guide to installing Fluent-Bit via Helm to install the base Fluent-Bit DaemonSet from the public helm chart https://github.com/fluent/helm-charts/
Use the included fluent-bit.conf template file to provide the stanzas for [INPUT], [FILTER], and [OUTPUT].
Update the [OUTPUT] stanza with your LogScale ingest credentials.
Add a toleration to the DaemonSet, to allow Fluent-Bit to be scheduled on control-plane master nodes:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
Update the fluent-bit ClusterRole yaml to allow access to events resources:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
. . .
. . .
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
[[- events]] add this resource <-------
verbs:
- get
- list
- watch
note: without events resource access, the following error will occur:
{"log":"[2023/XX/XX XX:XX:XX] [error] [input:kubernetes_events:kubernetes_events.4] http_status=403:\n","stream":"stderr","time":"2023-XX-XXTXX:XX:XX.XXXXXXXXXZ"}
{"log":"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"events is forbidden: User \\\"system:serviceaccount:logging:fluent-bit\\\" cannot list resource \\\"events\\\" in API group \\\"\\\" at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"events\"},\"code\":403}\n","stream":"stderr","time":"2023-XX-XXTXX:XX:XX.XXXXXXXXZ"}
helm install fluent-bit .
Once fluent-bit is enabled and the DaemonSet is running on all nodes, events should begin to appear in LogScale.
