Skip to content

Commit

Permalink
Client recommendations documentation tweaks (#1124)
Browse files Browse the repository at this point in the history
* consistent use of proper noun for Fluent Bit

* add notes about metrics extraction to client selection docs
  • Loading branch information
geowa4 authored and cyriltovena committed Oct 9, 2019
1 parent 8a95972 commit 9a6b5db
Show file tree
Hide file tree
Showing 6 changed files with 50 additions and 37 deletions.
18 changes: 9 additions & 9 deletions cmd/fluent-bit/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# fluent-bit output plugin
# Fluent Bit output plugin

[Fluent-bit](https://fluentbit.io/) is a Fast and Lightweight Data Forwarder, it can be configured with the [Loki output plugin](https://fluentbit.io/documentation/0.12/output/) to ship logs to Loki. You can define which log files you want to collect using the [`Tail`](https://fluentbit.io/documentation/0.12/input/tail.html) [input plugin](https://fluentbit.io/documentation/0.12/getting_started/input.html). Additionally fluent-bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc..) to structure and alter log lines.
[Fluent Bit](https://fluentbit.io/) is a Fast and Lightweight Data Forwarder, it can be configured with the [Loki output plugin](https://fluentbit.io/documentation/0.12/output/) to ship logs to Loki. You can define which log files you want to collect using the [`Tail`](https://fluentbit.io/documentation/0.12/input/tail.html) [input plugin](https://fluentbit.io/documentation/0.12/getting_started/input.html). Additionally Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc..) to structure and alter log lines.

This plugin is implemented with [fluent-bit's go plugin](https://github.com/fluent/fluent-bit-go) interface. It pushes logs to Loki using a GRPC connection.
This plugin is implemented with [Fluent Bit's Go plugin](https://github.com/fluent/fluent-bit-go) interface. It pushes logs to Loki using a GRPC connection.

> syslog and systemd input plugin have not been tested yet, feedback appreciated.
Expand All @@ -29,7 +29,7 @@ You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the o

### LabelMapPath

When using the `Parser` and `Filter` plugins fluent-bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures.
When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures.
You can pass a json file that defines how to extract [labels](../../docs/overview/README.md#overview-of-loki) from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names.

Considering the record below :
Expand Down Expand Up @@ -103,15 +103,15 @@ make fluent-bit-plugin

## Local

If you have fluent-bit installed in your `$PATH` you can run the plugin using:
If you have Fluent Bit installed in your `$PATH` you can run the plugin using:

```bash
fluent-bit -e /path/to/built/out_loki.so -c fluent-bit.conf
```

## Docker

You can run a fluent-bit container with Loki output plugin pre-installed using our [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki) image:
You can run a Fluent Bit container with Loki output plugin pre-installed using our [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki) image:

```bash
docker run -v /var/log:/var/log \
Expand All @@ -121,9 +121,9 @@ docker run -v /var/log:/var/log \

## Kubernetes

You can run fluent-bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs.
You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs.

To do so you can use our [fluent-bit helm chart](../../production/helm/fluent-bit/README.md):
To do so you can use our [Fluent Bit helm chart](../../production/helm/fluent-bit/README.md):

> Make sure [tiller](https://helm.sh/docs/install/) is installed correctly in your cluster
Expand All @@ -136,7 +136,7 @@ helm upgrade --install fluent-bit loki/fluent-bit \

By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc..).

Alternatively you can install the Loki and fluent-bit all together using:
Alternatively you can install the Loki and Fluent Bit all together using:

```bash
helm upgrade --install loki-stack loki/loki-stack \
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ simplifies the operation and significantly lowers the cost of Loki.
5. [Troubleshooting](clients/promtail/troubleshooting.md)
2. [Docker Driver](clients/docker-driver/README.md)
1. [Configuration](clients/docker-driver/configuration.md)
4. [Fluent-bit](../cmd/fluent-bit/README.md)
4. [Fluent Bit](../cmd/fluent-bit/README.md)
3. [Fluentd](clients/fluentd.md)
6. [LogQL](logql.md)
7. [Operations](operations/README.md)
Expand Down
25 changes: 19 additions & 6 deletions docs/clients/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,40 @@ Loki supports the following official clients for sending logs:

1. [Promtail](./promtail/README.md)
2. [Docker Driver](./docker-driver/README.md)
4. [Fluent-bit](../../cmd/fluent-bit/README.md)
4. [Fluent Bit](../../cmd/fluent-bit/README.md)
3. [Fluentd](./fluentd.md)

## Picking a Client

While all clients can be used simultaneously to cover multiple use cases, which
client is initially picked to send logs depends on your use case.

### Promtail

Promtail is the client of choice when you're running Kubernetes, as you can
configure it to automatically scrape logs from pods running on the same node
that Promtail runs on. Promtail and Prometheus running together in Kubernetes
enables powerful debugging: if Prometheus and Promtail use the same labels,
users can use tools like Grafana to switch between metrics and logs based on the
label set.

Promtail is also the client of choice on bare-metal: since it can be configured
to tail logs from all files given a host path, it is the easiest way to send
Promtail is also the client of choice on bare-metal since it can be configured
to tail logs from all files given a host path. It is the easiest way to send
logs to Loki from plain-text files (e.g., things that log to `/var/log/*.log`).

When using Docker and not Kubernetes, the Docker Logging driver should be used,
as it automatically adds labels appropriate to the running container.
Lastly, Promtail works well if you want to extract metrics from logs such as
counting the occurrences of a particular message.

### Docker Logging Driver

When using Docker and not Kubernetes, the Docker logging driver for Loki should
be used as it automatically adds labels appropriate to the running container.

The Fluentd and fluent-bit plugins are ideal when you already have Fluentd deployed
### Fluentd and Fluent Bit

The Fluentd and Fluent Bit plugins are ideal when you already have Fluentd deployed
and you already have configured `Parser` and `Filter` plugins.

Fluentd also works well for extracting metrics from logs when using its
Prometheus plugin.

2 changes: 1 addition & 1 deletion docs/installation/helm.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ $ helm upgrade --install loki loki/loki --set "key1=val1,key2=val2,..."
$ helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false
```

### Deploy Loki Stack (Loki, fluent-bit, Grafana, Prometheus)
### Deploy Loki Stack (Loki, Fluent Bit, Grafana, Prometheus)

```bash
$ helm upgrade --install loki loki/loki-stack \
Expand Down
4 changes: 2 additions & 2 deletions production/helm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,14 @@ $ helm upgrade --install loki loki/loki
$ helm upgrade --install promtail loki/promtail --set "loki.serviceName=loki"
```

## Deploy Loki and fluent-bit to your cluster
## Deploy Loki and Fluent Bit to your cluster

```bash
$ helm upgrade --install loki loki/loki-stack \
--set fluent-bit.enabled=true,promtail.enabled=false
```

## Deploy fluent-bit only
## Deploy Fluent Bit only

```bash
$ helm upgrade --install fluent-bit loki/fluent-bit \
Expand Down
36 changes: 18 additions & 18 deletions production/helm/fluent-bit/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Fluent-bit Loki chart
# Fluent Bit Loki chart

This chart install the fluent-bit application to ship logs to Loki. It defines daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.
This chart install the Fluent Bit application to ship logs to Loki. It defines daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.

## Installing the Chart

Expand All @@ -13,7 +13,7 @@ helm upgrade --install my-release loki/fluent-bit \
--set loki.serviceName=loki.svc.cluster.local
```

The command deploys fluent-bit on the Kubernetes cluster with the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
The command deploys Fluent Bit on the Kubernetes cluster with the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.

To configure the chart to send to [Grafana Cloud](https://grafana.com/products/cloud) use:

Expand All @@ -34,7 +34,7 @@ helm upgrade --install my-release loki/fluent-bit \

The full list of available tags on [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki).

Alternatively you can install the full [Loki stack](../loki-stack) (Loki + fluent-bit) using:
Alternatively you can install the full [Loki stack](../loki-stack) (Loki + Fluent Bit) using:

```bash
helm upgrade --install my-release loki/loki-stack \
Expand All @@ -45,9 +45,9 @@ This will automatically configured the `loki.serviceName` configuration field to

## RBAC

By default, `rbac.create` is set to true. This enable RBAC support in fluent-bit and must be true if RBAC is enabled in your cluster.
By default, `rbac.create` is set to true. This enable RBAC support in Fluent Bit and must be true if RBAC is enabled in your cluster.

The chart will take care of creating the required service accounts and roles for fluent-bit.
The chart will take care of creating the required service accounts and roles for Fluent Bit.

If you have RBAC disabled, or to put it another way, ABAC enabled, you should set this value to `false`.

Expand All @@ -63,9 +63,9 @@ The command removes all the Kubernetes components associated with the chart and

## Configuration

The following tables lists the configurable parameters of the fluent-bit chart and their default values.
The following tables lists the configurable parameters of the Fluent Bit chart and their default values.

For more details, read the [fluent-bit documentation](../../../cmd/fluent-bit/README.md)
For more details, read the [Fluent Bit documentation](../../../cmd/fluent-bit/README.md)

| Parameter | Description | Default |
|--------------------------|----------------------------------------------------------------------------------------------------|----------------------------------|
Expand All @@ -74,27 +74,27 @@ For more details, read the [fluent-bit documentation](../../../cmd/fluent-bit/RE
| `loki.serviceScheme` | The scheme of the Loki service. | `http` |
| `loki.user` | The http basic auth username to access the Loki service. | |
| `loki.password` | The http basic auth password to access the Loki service. | |
| `config.port` | the fluent-bit port to listen. (This is mainly used to serve metrics) | `2020` |
| `config.loglevel` | the fluent-bit log level (debug,info,warn,error). | `warn` |
| `config.port` | the Fluent Bit port to listen. (This is mainly used to serve metrics) | `2020` |
| `config.loglevel` | the Fluent Bit log level (debug,info,warn,error). | `warn` |
| `config.lineFormat` | The line format to use to send a record (json/key_value) | `json` |
| `config.removeKeys` | The list of key to remove from each record | `[removeKeys,stream]` |
| `config.labels` | A set of labels to send for every log | `'{job="fluent-bit"}'` |
| `config.labelMap` | Mapping of labels from a record. See [fluent-bit documentation](../../../cmd/fluent-bit/README.md) | |
| `config.labelMap` | Mapping of labels from a record. See [Fluent Bit documentation](../../../cmd/fluent-bit/README.md) | |
| `affinity` | [affinity][affinity] settings for pod assignment | `{}` |
| `annotations` | Annotations to add to Kubernetes resources. | `{}` |
| `deploymentStrategy` | The deployment strategy to use with the daemonset | `RollingUpdate` |
| `image.repository` | The fluent-bit docker image repository | `grafana/fluent-bit-plugin-loki` |
| `image.tag` | The fluent-bit docker image tag | `0.1` |
| `image.pullPolicy` | The fluent-bit docker image pull policy | `IfNotPresent` |
| `nodeSelector` | fluent-bit [node labels][nodeSelector] for pod assignment | `{}` |
| `podLabels` | additional fluent-bit pod labels | `{}` |
| `podAnnotations` | additional fluent-bit pod annotations | `Prometheus discovery` |
| `image.repository` | The Fluent Bit docker image repository | `grafana/fluent-bit-plugin-loki` |
| `image.tag` | The Fluent Bit docker image tag | `0.1` |
| `image.pullPolicy` | The Fluent Bit docker image pull policy | `IfNotPresent` |
| `nodeSelector` | Fluent Bit [node labels][nodeSelector] for pod assignment | `{}` |
| `podLabels` | additional Fluent Bit pod labels | `{}` |
| `podAnnotations` | additional Fluent Bit pod annotations | `Prometheus discovery` |
| `rbac.create` | Activate support for RBAC | `true` |
| `resources` | Resource requests/limit | |
| `tolerations` | [Toleration][toleration] labels for pod assignment | `no schedule on master nodes` |
| `volumes` | [Volume]([volumes]) to mount | `host containers log` |
| `volumeMounts` | Volume mount mapping | |
| `serviceMonitor.enabled` | Create a [Prometheus Operator](operator) serviceMonitor resource for fluent-bit | `false` |
| `serviceMonitor.enabled` | Create a [Prometheus Operator](operator) serviceMonitor resource for Fluent Bit | `false` |


[toleration]: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
Expand Down

0 comments on commit 9a6b5db

Please sign in to comment.