Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client recommendations documentation tweaks #1124

Merged
merged 2 commits into from
Oct 9, 2019
Merged

Client recommendations documentation tweaks #1124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7633906
consistent use of proper noun for Fluent Bit
geowa4 Oct 7, 2019
58b0942
add notes about metrics extraction to client selection docs
geowa4 Oct 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions cmd/fluent-bit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# fluent-bit output plugin
# Fluent Bit output plugin

[Fluent-bit](https://fluentbit.io/) is a Fast and Lightweight Data Forwarder, it can be configured with the [Loki output plugin](https://fluentbit.io/documentation/0.12/output/) to ship logs to Loki. You can define which log files you want to collect using the [`Tail`](https://fluentbit.io/documentation/0.12/input/tail.html) [input plugin](https://fluentbit.io/documentation/0.12/getting_started/input.html). Additionally fluent-bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc..) to structure and alter log lines.
[Fluent Bit](https://fluentbit.io/) is a Fast and Lightweight Data Forwarder, it can be configured with the [Loki output plugin](https://fluentbit.io/documentation/0.12/output/) to ship logs to Loki. You can define which log files you want to collect using the [`Tail`](https://fluentbit.io/documentation/0.12/input/tail.html) [input plugin](https://fluentbit.io/documentation/0.12/getting_started/input.html). Additionally Fluent Bit supports multiple `Filter` and `Parser` plugins (`Kubernetes`, `JSON`, etc..) to structure and alter log lines.

This plugin is implemented with [fluent-bit's go plugin](https://github.com/fluent/fluent-bit-go) interface. It pushes logs to Loki using a GRPC connection.
This plugin is implemented with [Fluent Bit's Go plugin](https://github.com/fluent/fluent-bit-go) interface. It pushes logs to Loki using a GRPC connection.

> syslog and systemd input plugin have not been tested yet, feedback appreciated.

Expand All @@ -29,7 +29,7 @@ You can use `Labels`, `RemoveKeys` , `LabelKeys` and `LabelMapPath` to how the o

### LabelMapPath

When using the `Parser` and `Filter` plugins fluent-bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures.
When using the `Parser` and `Filter` plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures.
You can pass a json file that defines how to extract [labels](../../docs/overview/README.md#overview-of-loki) from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names.

Considering the record below :
Expand Down Expand Up @@ -103,15 +103,15 @@ make fluent-bit-plugin

## Local

If you have fluent-bit installed in your `$PATH` you can run the plugin using:
If you have Fluent Bit installed in your `$PATH` you can run the plugin using:

```bash
fluent-bit -e /path/to/built/out_loki.so -c fluent-bit.conf
```

## Docker

You can run a fluent-bit container with Loki output plugin pre-installed using our [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki) image:
You can run a Fluent Bit container with Loki output plugin pre-installed using our [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki) image:

```bash
docker run -v /var/log:/var/log \
Expand All @@ -121,9 +121,9 @@ docker run -v /var/log:/var/log \

## Kubernetes

You can run fluent-bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs.
You can run Fluent Bit as a [Daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) to collect all your Kubernetes workload logs.

To do so you can use our [fluent-bit helm chart](../../production/helm/fluent-bit/README.md):
To do so you can use our [Fluent Bit helm chart](../../production/helm/fluent-bit/README.md):

> Make sure [tiller](https://helm.sh/docs/install/) is installed correctly in your cluster

Expand All @@ -136,7 +136,7 @@ helm upgrade --install fluent-bit loki/fluent-bit \

By default it will collect all containers logs and extract labels from Kubernetes API (`container_name`, `namespace`, etc..).

Alternatively you can install the Loki and fluent-bit all together using:
Alternatively you can install the Loki and Fluent Bit all together using:

```bash
helm upgrade --install loki-stack loki/loki-stack \
Expand Down
2 changes: 1 addition & 1 deletion docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ simplifies the operation and significantly lowers the cost of Loki.
5. [Troubleshooting](clients/promtail/troubleshooting.md)
2. [Docker Driver](clients/docker-driver/README.md)
1. [Configuration](clients/docker-driver/configuration.md)
4. [Fluent-bit](../cmd/fluent-bit/README.md)
4. [Fluent Bit](../cmd/fluent-bit/README.md)
3. [Fluentd](clients/fluentd.md)
6. [LogQL](logql.md)
7. [Operations](operations/README.md)
Expand Down
25 changes: 19 additions & 6 deletions docs/clients/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,40 @@ Loki supports the following official clients for sending logs:

1. [Promtail](./promtail/README.md)
2. [Docker Driver](./docker-driver/README.md)
4. [Fluent-bit](../../cmd/fluent-bit/README.md)
4. [Fluent Bit](../../cmd/fluent-bit/README.md)
3. [Fluentd](./fluentd.md)

## Picking a Client

While all clients can be used simultaneously to cover multiple use cases, which
client is initially picked to send logs depends on your use case.

### Promtail

Promtail is the client of choice when you're running Kubernetes, as you can
configure it to automatically scrape logs from pods running on the same node
that Promtail runs on. Promtail and Prometheus running together in Kubernetes
enables powerful debugging: if Prometheus and Promtail use the same labels,
users can use tools like Grafana to switch between metrics and logs based on the
label set.

Promtail is also the client of choice on bare-metal: since it can be configured
to tail logs from all files given a host path, it is the easiest way to send
Promtail is also the client of choice on bare-metal since it can be configured
to tail logs from all files given a host path. It is the easiest way to send
logs to Loki from plain-text files (e.g., things that log to `/var/log/*.log`).

When using Docker and not Kubernetes, the Docker Logging driver should be used,
as it automatically adds labels appropriate to the running container.
Lastly, Promtail works well if you want to extract metrics from logs such as
counting the occurrences of a particular message.

### Docker Logging Driver

When using Docker and not Kubernetes, the Docker logging driver for Loki should
be used as it automatically adds labels appropriate to the running container.

The Fluentd and fluent-bit plugins are ideal when you already have Fluentd deployed
### Fluentd and Fluent Bit

The Fluentd and Fluent Bit plugins are ideal when you already have Fluentd deployed
and you already have configured `Parser` and `Filter` plugins.

Fluentd also works well for extracting metrics from logs when using its
Prometheus plugin.

2 changes: 1 addition & 1 deletion docs/installation/helm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ $ helm upgrade --install loki loki/loki --set "key1=val1,key2=val2,..."
$ helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=false
```

### Deploy Loki Stack (Loki, fluent-bit, Grafana, Prometheus)
### Deploy Loki Stack (Loki, Fluent Bit, Grafana, Prometheus)

```bash
$ helm upgrade --install loki loki/loki-stack \
Expand Down
4 changes: 2 additions & 2 deletions production/helm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,14 @@ $ helm upgrade --install loki loki/loki
$ helm upgrade --install promtail loki/promtail --set "loki.serviceName=loki"
```

## Deploy Loki and fluent-bit to your cluster
## Deploy Loki and Fluent Bit to your cluster

```bash
$ helm upgrade --install loki loki/loki-stack \
--set fluent-bit.enabled=true,promtail.enabled=false
```

## Deploy fluent-bit only
## Deploy Fluent Bit only

```bash
$ helm upgrade --install fluent-bit loki/fluent-bit \
Expand Down
36 changes: 18 additions & 18 deletions production/helm/fluent-bit/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Fluent-bit Loki chart
# Fluent Bit Loki chart

This chart install the fluent-bit application to ship logs to Loki. It defines daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.
This chart install the Fluent Bit application to ship logs to Loki. It defines daemonset on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.

## Installing the Chart

Expand All @@ -13,7 +13,7 @@ helm upgrade --install my-release loki/fluent-bit \
--set loki.serviceName=loki.svc.cluster.local
```

The command deploys fluent-bit on the Kubernetes cluster with the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
The command deploys Fluent Bit on the Kubernetes cluster with the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.

To configure the chart to send to [Grafana Cloud](https://grafana.com/products/cloud) use:

Expand All @@ -34,7 +34,7 @@ helm upgrade --install my-release loki/fluent-bit \

The full list of available tags on [docker hub](https://cloud.docker.com/u/grafana/repository/docker/grafana/fluent-bit-plugin-loki).

Alternatively you can install the full [Loki stack](../loki-stack) (Loki + fluent-bit) using:
Alternatively you can install the full [Loki stack](../loki-stack) (Loki + Fluent Bit) using:

```bash
helm upgrade --install my-release loki/loki-stack \
Expand All @@ -45,9 +45,9 @@ This will automatically configured the `loki.serviceName` configuration field to

## RBAC

By default, `rbac.create` is set to true. This enable RBAC support in fluent-bit and must be true if RBAC is enabled in your cluster.
By default, `rbac.create` is set to true. This enable RBAC support in Fluent Bit and must be true if RBAC is enabled in your cluster.

The chart will take care of creating the required service accounts and roles for fluent-bit.
The chart will take care of creating the required service accounts and roles for Fluent Bit.

If you have RBAC disabled, or to put it another way, ABAC enabled, you should set this value to `false`.

Expand All @@ -63,9 +63,9 @@ The command removes all the Kubernetes components associated with the chart and

## Configuration

The following tables lists the configurable parameters of the fluent-bit chart and their default values.
The following tables lists the configurable parameters of the Fluent Bit chart and their default values.

For more details, read the [fluent-bit documentation](../../../cmd/fluent-bit/README.md)
For more details, read the [Fluent Bit documentation](../../../cmd/fluent-bit/README.md)

| Parameter | Description | Default |
|--------------------------|----------------------------------------------------------------------------------------------------|----------------------------------|
Expand All @@ -74,27 +74,27 @@ For more details, read the [fluent-bit documentation](../../../cmd/fluent-bit/RE
| `loki.serviceScheme` | The scheme of the Loki service. | `http` |
| `loki.user` | The http basic auth username to access the Loki service. | |
| `loki.password` | The http basic auth password to access the Loki service. | |
| `config.port` | the fluent-bit port to listen. (This is mainly used to serve metrics) | `2020` |
| `config.loglevel` | the fluent-bit log level (debug,info,warn,error). | `warn` |
| `config.port` | the Fluent Bit port to listen. (This is mainly used to serve metrics) | `2020` |
| `config.loglevel` | the Fluent Bit log level (debug,info,warn,error). | `warn` |
| `config.lineFormat` | The line format to use to send a record (json/key_value) | `json` |
| `config.removeKeys` | The list of key to remove from each record | `[removeKeys,stream]` |
| `config.labels` | A set of labels to send for every log | `'{job="fluent-bit"}'` |
| `config.labelMap` | Mapping of labels from a record. See [fluent-bit documentation](../../../cmd/fluent-bit/README.md) | |
| `config.labelMap` | Mapping of labels from a record. See [Fluent Bit documentation](../../../cmd/fluent-bit/README.md) | |
| `affinity` | [affinity][affinity] settings for pod assignment | `{}` |
| `annotations` | Annotations to add to Kubernetes resources. | `{}` |
| `deploymentStrategy` | The deployment strategy to use with the daemonset | `RollingUpdate` |
| `image.repository` | The fluent-bit docker image repository | `grafana/fluent-bit-plugin-loki` |
| `image.tag` | The fluent-bit docker image tag | `0.1` |
| `image.pullPolicy` | The fluent-bit docker image pull policy | `IfNotPresent` |
| `nodeSelector` | fluent-bit [node labels][nodeSelector] for pod assignment | `{}` |
| `podLabels` | additional fluent-bit pod labels | `{}` |
| `podAnnotations` | additional fluent-bit pod annotations | `Prometheus discovery` |
| `image.repository` | The Fluent Bit docker image repository | `grafana/fluent-bit-plugin-loki` |
| `image.tag` | The Fluent Bit docker image tag | `0.1` |
| `image.pullPolicy` | The Fluent Bit docker image pull policy | `IfNotPresent` |
| `nodeSelector` | Fluent Bit [node labels][nodeSelector] for pod assignment | `{}` |
| `podLabels` | additional Fluent Bit pod labels | `{}` |
| `podAnnotations` | additional Fluent Bit pod annotations | `Prometheus discovery` |
| `rbac.create` | Activate support for RBAC | `true` |
| `resources` | Resource requests/limit | |
| `tolerations` | [Toleration][toleration] labels for pod assignment | `no schedule on master nodes` |
| `volumes` | [Volume]([volumes]) to mount | `host containers log` |
| `volumeMounts` | Volume mount mapping | |
| `serviceMonitor.enabled` | Create a [Prometheus Operator](operator) serviceMonitor resource for fluent-bit | `false` |
| `serviceMonitor.enabled` | Create a [Prometheus Operator](operator) serviceMonitor resource for Fluent Bit | `false` |


[toleration]: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
Expand Down