You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Fluent Bit](http://fluentbit.io) is a lightweight and optimized events data collector for Linux, Embedded Linux, OSX and BSD family operating systems. It has been made with a strong focus on IoT to allow the collection of events from different sources like network protocols, hardware radios and IoT frameworks.
3
+
[Fluent Bit](http://fluentbit.io) is a Fast and Lightweight Data Forwarder for Linux, OSX and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.
4
4
5
-
[Fluent Bit](http://fluentbit.io) is part of the [Fluentd](http://fluentd.org) project ecosystem, it's licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
6
-
7
-
This project is made and sponsored by [Treasure Data](https://www.treasuredata.com).
5
+
[Fluent Bit](http://fluentbit.io) is part of the [Fluentd](http://fluentd.org) project ecosystem, it's licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0). This project is made and sponsored by [Treasure Data](https://www.treasuredata.com).
[Fluent Bit](http://fluentbit.io) is an open source and multi-platform data collection tool which aims to be a generic Swiss knife for events collection on IoT environments and related hardware devices.
3
+
[Fluent Bit](http://fluentbit.io) is an open source and multi-platform data forwarder tool which aims to be a generic Swiss knife for log collection and distribution.
4
4
5
-
We, [Treasure Data](http://treasuredata.com), as a Big Data company, provide an analytics infrastructure in the Cloud where we provide an end-to-end solution to collect, store and do analytics over the data. [Fluent Bit](http://fluentbit.io) is an integral part of this pipeline where it solve the data collection needs where the __source__ information may come from Embedded and IoT environments mostly.
5
+
We, [Treasure Data](http://treasuredata.com), as a Big Data company, provide an analytics infrastructure in the Cloud where we provide an end-to-end solution to collect, store and do analytics over the data. [Fluent Bit](http://fluentbit.io) is an integral part of this pipeline where it solve the data collection needs.
Copy file name to clipboardExpand all lines: about/fluentd_and_fluentbit.md
+12-7
Original file line number
Diff line number
Diff line change
@@ -2,18 +2,23 @@
2
2
3
3
Data collection matters and nowadays the scenarios from where the information can _come from_ are very variable. For hence to be more flexible on certain markets needs, we may need different options. On this page we will describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects.
4
4
5
-
[Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) projects are both created and sponsored by [Treasure Data](http://treasuredata.com) and they aim to solve the data collection needs but on different scenarios and environments, the following table describe a comparisson on different areas of the projects:
5
+
[Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) projects are both created and sponsored by [Treasure Data](http://treasuredata.com) and they aim to solve the _Data Collection_ and _Data Forwarding_ needs as a whole.
6
+
7
+
From an architecture usability perspective:
8
+
9
+
- Fluentd is an Aggregator.
10
+
- Fluent Bit is a Data Forwarder.
11
+
12
+
The following table describe a comparisson on different areas of the projects:
As described in the table, if the target environment is a server with common capacity, [Fluentd](http://fluentd.org) is a great option due to it flexibility and availability of plugins (more than 300 extensions!) but if the _data collection_ will happen in an Embedded environment or an IoT device where the system capacity is restricted, [Fluent Bit](http://fluentbit.io) is the solution to use.
18
-
19
-
Both tools are not mutual exclusive, [Fluent Bit](http://fluentbit.io) provides and _output_ plugin to flush the information to a [Fluentd](http://fluentd.org) instance, so they can work together in your architecture or as independent services.
24
+
Considering Fluentd as a main Aggregator and Fluent Bit as a Data Forwarder, we can see both projects complement each other providing a full reliable solution.
Copy file name to clipboardExpand all lines: about/why.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
Data collection is hard.
4
4
5
-
Nowadays the number of sources of information that we get in our environments often increase. From an _Embedded/IoT_ perspective the scenario is more complex, devices also provide their own data which can come from sensors, custom applications or the system in general. Considering that these environments may be restrictive in terms of capacity, we need a fast and optimized solution that can deal with:
5
+
Nowadays the number of sources of information that we get in our environments often increase. Handling data collection at scale is complex, the ways to collect data from and it formats requires a specialized tool that can deal with:
Copy file name to clipboardExpand all lines: development/library_api.md
+23-25
Original file line number
Diff line number
Diff line change
@@ -7,23 +7,21 @@
7
7
[Fluent Bit](http://fluentbit.io) runs as a service, meaning that the API exposed for developers provide interfaces to create and manage a context, specify inputs/outputs, set configuration parameters and set routing paths for the event/records. A typical usage of the library involves:
8
8
9
9
- Create library instance/context and set properties.
10
-
- Enable _input_ plugin(s) instances and set properties.
11
-
- Enable _output_ plugin(s) instances and set properties.
10
+
- Enable _input_ plugin(s) and set properties.
11
+
- Enable _output_ plugin(s) and set properties.
12
12
- Start the library runtime.
13
13
- Optionally ingest records manually.
14
14
- Stop the library runtime.
15
15
- Destroy library instance/context.
16
16
17
17
## Data Types
18
18
19
-
There are three main data types exposed by the library. All of them including further functions are prefixed with __flb\___. The following table describes them:
19
+
Starting from Fluent Bit v0.9, there is only one data type exposed by the library, by convention prefixed with __flb\___.
20
20
21
21
22
22
| Type | Description |
23
23
|--------------|----------------------|
24
24
| flb_ctx_t | Main library context. It aims to reference the context returned by _flb\_create();_|
25
-
| flb_input_t | Reference an enabled _input_ plugin instance. Used to store the output of _flb\_input(...);_ function. |
26
-
| flb_output_t | Reference an enabled _output_ plugin instance. Used to store the output of _flb\_output(...);_ function. |
27
25
28
26
## API Reference
29
27
@@ -90,7 +88,7 @@ When built, [Fluent Bit](http://fluentbit.io) library contains a certain number
The argument __ctx__ represents the library context created by __flb_create()__, then __name__ is the name of the input plugin that is required to enable.
@@ -99,14 +97,14 @@ The third argument __data__ can be used to pass a custom reference to the plugin
99
97
100
98
__Return Value__
101
99
102
-
On success, __flb_input()__ returns the input plugin instance; on error, it returns NULL.
100
+
On success, __flb_input()__ returns an integer value >= zero (similar to a file descriptor); on error, it returns a negative number.
103
101
104
102
__Usage__
105
103
106
104
```C
107
-
flb_input_t *in;
105
+
int in_ffd;
108
106
109
-
in = flb_input(ctx, "cpu", NULL);
107
+
in_ffd = flb_input(ctx, "cpu", NULL);
110
108
```
111
109
112
110
### Set Input Plugin Properties
@@ -116,7 +114,7 @@ A plugin instance created through __flb_input()__, may provide some configuratio
116
114
__Prototype__
117
115
118
116
```C
119
-
intflb_input_set(flb_input_t *in, ...);
117
+
intflb_input_set(flb_ctx_t *ctx, int in_ffd, ...);
120
118
```
121
119
122
120
__Return Value__
@@ -130,13 +128,13 @@ The __flb_input_set()__ allows to set one or more properties in a key/value stri
130
128
```C
131
129
int ret;
132
130
133
-
ret = flb_input_set(in,
131
+
ret = flb_input_set(ctx, in_ffd,
134
132
"tag", "my_records",
135
133
"ssl", "false",
136
134
NULL);
137
135
```
138
136
139
-
The above example specified the values for the properties __tag__ and __ssl__, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
137
+
The argument __ctx__ represents the library context created by __flb_create()__. The above example specified the values for the properties __tag__ and __ssl__, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
140
138
141
139
The properties allowed per input plugin are specified on each specific plugin documentation.
142
140
@@ -149,7 +147,7 @@ When built, [Fluent Bit](http://fluentbit.io) library contains a certain number
The argument __ctx__ represents the library context created by __flb_create()__, then __name__ is the name of the output plugin that is required to enable.
@@ -158,14 +156,14 @@ The third argument __data__ can be used to pass a custom reference to the plugin
158
156
159
157
__Return Value__
160
158
161
-
On success, __flb_output()__ returns the output plugin instance; on error, it returns NULL.
159
+
On success, __flb_output()__ returns the output plugin instance; on error, it returns a negative number.
162
160
163
161
__Usage__
164
162
165
163
```C
166
-
flb_output_t *out;
164
+
int out_ffd;
167
165
168
-
out = flb_output(ctx, "stdout", NULL);
166
+
out_ffd = flb_output(ctx, "stdout", NULL);
169
167
```
170
168
171
169
### Set Output Plugin Properties
@@ -175,12 +173,12 @@ A plugin instance created through __flb_output()__, may provide some configurati
175
173
__Prototype__
176
174
177
175
```C
178
-
intflb_output_set(flb_output_t *in, ...);
176
+
intflb_output_set(flb_ctx_t *ctx, int out_ffd, ...);
179
177
```
180
178
181
179
__Return Value__
182
180
183
-
On success it returns 0; on error it returns a negative number.
181
+
On success it returns an integer value >= zero (similar to a file descriptor); on error it returns a negative number.
184
182
185
183
__Usage__
186
184
@@ -189,13 +187,13 @@ The __flb_output_set()__ allows to set one or more properties in a key/value str
189
187
```C
190
188
int ret;
191
189
192
-
ret = flb_output_set(in,
193
-
"tag", "my_records",
194
-
"ssl", "false",
195
-
NULL);
190
+
ret = flb_output_set(ctx, out_ffd,
191
+
"tag", "my_records",
192
+
"ssl", "false",
193
+
NULL);
196
194
```
197
195
198
-
The above example specified the values for the properties __tag__ and __ssl__, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
196
+
The argument __ctx__ represents the library context created by __flb_create()__. The above example specified the values for the properties __tag__ and __ssl__, note that the value is always a string (char *) and once there is no more parameters a NULL argument must be added at the end of the list.
199
197
200
198
The properties allowed per output plugin are specified on each specific plugin documentation.
201
199
@@ -278,10 +276,10 @@ There are some cases where the caller application may want to ingest data into F
intflb_lib_push(flb_ctx_t *ctx, int in_ffd, void *data, size_t len);
282
280
```
283
281
284
-
The first argument __input__ is a reference to an input instance of the __lib__ plugin, __data__ is a reference to the message to be ingested and __len__ the number of bytes to take from it.
282
+
The first argument is the context created previously through __flb_create()__. __in_ffd__ is the numeric reference of the input plugin (for this case it should be an input of plugin __lib__ type), __data__ is a reference to the message to be ingested and __len__ the number of bytes to take from it.
Fluent Bit provides integrated support for _Transport Layer Security_\(TLS\) and it predecessor _Secure Sockets Layer_\(SSL\) respectively. In this section we will refer as TLS only for both implementations.
4
+
5
+
Each output plugin that requires to perform Network I\/O can optionally enable TLS and configure the behavior. The following table describe the properties available:
6
+
7
+
| Property | Description | Default |
8
+
| --- | --- | --- |
9
+
| tls | enable or disable TLS support | Off |
10
+
| tls.verify | force certificate validation | On |
11
+
| tls.ca\_file | absolute path to CA certificate file ||
12
+
| tls.crt\_file | absolute path to Certificate file ||
13
+
| tls.key\_file | absolute path to private Key file ||
14
+
| tls.key\_passwd | optional password for tls.key\_file file ||
15
+
16
+
The listed properties can be enabled in the configuration file, specifically on each output plugin section or directly through the command line. The following **output** plugins can take advante of the TLS feature:
17
+
18
+
*[HTTP](../output/http.md)
19
+
*[Elasticsearch](../output/elasticsearch.md)
20
+
21
+
## Example: enable TLS on HTTP output
22
+
23
+
By default HTTP output plugin uses plain TCP, enabling TLS from the command line can be done with:
24
+
25
+
```
26
+
$ fluent-bit -i cpu -t cpu -o http://192.168.2.3:80/something \
27
+
-p tls=on \
28
+
-p tls.verify=off \
29
+
-m '*'
30
+
31
+
```
32
+
33
+
In the command line above, the two properties _tls_ and _tls.verify_ where enabled for demonstration purposes \(we strongly suggest always keep verification ON\).
34
+
35
+
The same behavior can be accomplished using a configuration file:
0 commit comments