Skip to content

Commit 1586683

Browse files
nokute78nokute78
authored
restore in_forward.md (#295)
* pipeline: inputs: forward: copy from previous version (#284) Signed-off-by: Takahiro YAMASHITA <nokute78@gmail.com> * pipeline: inputs: forward: fix broken links Signed-off-by: Takahiro YAMASHITA <nokute78@gmail.com>
1 parent 09e5551 commit 1586683

File tree

1 file changed

+36
-191
lines changed

1 file changed

+36
-191
lines changed

‎pipeline/inputs/forward.md‎

Lines changed: 36 additions & 191 deletions
Original file line numberDiff line numberDiff line change
@@ -1,226 +1,71 @@
11
# Forward
22

3-
_Forward_ is the protocol used by [Fluentd](http://www.fluentd.org) to route messages between peers. The **forward** output plugin allows to provide interoperability between [Fluent Bit](http://fluentbit.io) and [Fluentd](http://fluentd.org). There are not configuration steps required besides to specify where [Fluentd](http://fluentd.org) is located, it can be in the local host or a in a remote machine.
4-
5-
This plugin offers two different transports and modes:
6-
7-
* Forward \(TCP\): It uses a plain TCP connection.
8-
* Secure Forward \(TLS\): when TLS is enabled, the plugin switch to Secure Forward mode.
3+
_Forward_ is the protocol used by [Fluent Bit](http://fluentbit.io) and [Fluentd](http://www.fluentd.org) to route messages between peers. This plugin implements the input service to listen for Forward messages.
94

105
## Configuration Parameters
116

12-
The following parameters are mandatory for either Forward for Secure Forward modes:
13-
14-
| Key | Description | Default |
15-
| :--- | :--- | :--- |
16-
| Host | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | 127.0.0.1 |
17-
| Port | TCP Port of the target service. | 24224 |
18-
| Time\_as\_Integer | Set timestamps in integer format, it enable compatibility mode for Fluentd v0.12 series. | False |
19-
| Upstream | If Forward will connect to an _Upstream_ instead of a simple host, this property defines the absolute path for the Upstream configuration file, for more details about this refer to the [Upstream Servers](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/configuration/upstream_servers.md) documentation section. | |
20-
| Send\_options | Always send options \(with "size"=count of messages\) | False |
21-
| Require\_ack\_response | Send "chunk"-option and wait for "ack" response from server. Enables at-least-once and receiving server can control rate of traffic. \(Requires Fluentd v0.14.0+ server\) | False |
22-
23-
## Secure Forward Mode Configuration Parameters
24-
25-
When using Secure Forward mode, the [TLS](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/configuration/tls_ssl.md) mode requires to be enabled. The following additional configuration parameters are available:
7+
The plugin supports the following configuration parameters:
268

279
| Key | Description | Default |
2810
| :--- | :--- | :--- |
29-
| Shared\_Key | A key string known by the remote Fluentd used for authorization. | |
30-
| Empty\_Shared\_Key | Use this option to connect to Fluentd with a zero-length secret. | False |
31-
| Username | Specify the username to present to a Fluentd server that enables `user_auth`. | |
32-
| Password | Specify the password corresponding to the username. | |
33-
| Self\_Hostname | Default value of the auto-generated certificate common name \(CN\). | |
34-
| tls | Enable or disable TLS support | Off |
35-
| tls.verify | Force certificate validation | On |
36-
| tls.debug | Set TLS debug verbosity level. It accept the following values: 0 \(No debug\), 1 \(Error\), 2 \(State change\), 3 \(Informational\) and 4 Verbose | 1 |
37-
| tls.ca\_file | Absolute path to CA certificate file | |
38-
| tls.crt\_file | Absolute path to Certificate file. | |
39-
| tls.key\_file | Absolute path to private Key file. | |
40-
| tls.key\_passwd | Optional password for tls.key\_file file. | |
41-
42-
## Forward Setup
43-
44-
Before proceeding, make sure that [Fluentd](http://fluentd.org) is installed in your system, if it's not the case please refer to the following [Fluentd Installation](http://docs.fluentd.org/v0.12/categories/installation) document and go ahead with that.
45-
46-
Once [Fluentd](http://fluentd.org) is installed, create the following configuration file example that will allow us to stream data into it:
47-
48-
```text
49-
<source>
50-
type forward
51-
bind 0.0.0.0
52-
port 24224
53-
</source>
54-
55-
<match fluent_bit>
56-
type stdout
57-
</match>
58-
```
11+
| Listen | Listener network interface. | 0.0.0.0 |
12+
| Port | TCP port to listen for incoming connections. | 24224 |
13+
| Buffer\_Max\_Size | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](../administration/configuring-fluent-bit/unit-sizes.md) specification. | _Buffer\_Chunk\_Size_ |
14+
| Buffer\_Chunk\_Size | By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_. The value must be according to the [Unit Size](../administration/configuring-fluent-bit/unit-sizes.md) specification. | 32KB |
5915

60-
That configuration file specifies that it will listen for _TCP_ connections on the port _24224_ through the **forward** input type. Then for every message with a _fluent\_bit_ **TAG**, will print the message to the standard output.
16+
## Getting Started
6117

62-
In one terminal launch [Fluentd](http://fluentd.org) specifying the new configuration file created \(in\_fluent-bit.conf\):
63-
64-
```bash
65-
$ fluentd -c test.conf
66-
2017-03-23 11:50:43 -0600 [info]: reading config file path="test.conf"
67-
2017-03-23 11:50:43 -0600 [info]: starting fluentd-0.12.33
68-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
69-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-docker' version '0.1.0'
70-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-elasticsearch' version '1.4.0'
71-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flatten-hash' version '0.2.0'
72-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.0.4'
73-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-influxdb' version '0.2.8'
74-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-json-in-json' version '0.1.4'
75-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
76-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-out-http' version '0.1.3'
77-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-parser' version '0.6.0'
78-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-record-reformer' version '0.7.0'
79-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.1'
80-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-stdin' version '0.1.1'
81-
2017-03-23 11:50:43 -0600 [info]: gem 'fluent-plugin-td' version '0.10.27'
82-
2017-03-23 11:50:43 -0600 [info]: adding match pattern="fluent_bit" type="stdout"
83-
2017-03-23 11:50:43 -0600 [info]: adding source type="forward"
84-
2017-03-23 11:50:43 -0600 [info]: using configuration file: <ROOT>
85-
<source>
86-
type forward
87-
bind 0.0.0.0
88-
port 24224
89-
</source>
90-
<match fluent_bit>
91-
type stdout
92-
</match>
93-
</ROOT>
94-
2017-03-23 11:50:43 -0600 [info]: listening fluent socket on 0.0.0.0:24224
95-
```
96-
97-
## Fluent Bit + Forward Setup <a id="forward_setup"></a>
98-
99-
Now that [Fluentd](http://fluentd.org) is ready to receive messages, we need to specify where the **forward** output plugin will flush the information using the following format:
100-
101-
```text
102-
bin/fluent-bit -i INPUT -o forward://HOST:PORT
103-
```
18+
In order to receive Forward messages, you can run the plugin from the command line or through the configuration file:
10419

105-
If the **TAG** parameter is not set, the plugin will set the tag as _fluent\_bit_. Keep in mind that **TAG** is important for routing rules inside [Fluentd](http://fluentd.org).
20+
### Command Line
10621

107-
Using the [CPU](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/input/cpu.md) input plugin as an example we will flush CPU metrics to [Fluentd](http://fluentd.org):
22+
From the command line you can let Fluent Bit listen for _Forward_ messages with the following options:
10823

10924
```bash
110-
$ bin/fluent-bit -i cpu -t fluent_bit -o forward://127.0.0.1:24224
25+
$ fluent-bit -i forward -o stdout
11126
```
11227

113-
Now on the [Fluentd](http://fluentd.org) side, you will see the CPU metrics gathered in the last seconds:
28+
By default the service will listen an all interfaces \(0.0.0.0\) through TCP port 24224, optionally you can change this directly, e.g:
11429

11530
```bash
116-
2017-03-23 11:53:06 -0600 fluent_bit: {"cpu_p":0.0,"user_p":0.0,"system_p":0.0,"cpu0.p_cpu":0.0,"cpu0.p_user":0.0,"cpu0.p_system":0.0,"cpu1.p_cpu":0.0,"cpu1.p_user":0.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0}
117-
2017-03-23 11:53:07 -0600 fluent_bit: {"cpu_p":2.25,"user_p":2.0,"system_p":0.25,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":1.0,"cpu1.p_user":1.0,"cpu1.p_system":0.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":3.0,"cpu3.p_user":2.0,"cpu3.p_system":1.0}
118-
2017-03-23 11:53:08 -0600 fluent_bit: {"cpu_p":1.75,"user_p":1.0,"system_p":0.75,"cpu0.p_cpu":2.0,"cpu0.p_user":1.0,"cpu0.p_system":1.0,"cpu1.p_cpu":3.0,"cpu1.p_user":1.0,"cpu1.p_system":2.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0}
119-
2017-03-23 11:53:09 -0600 fluent_bit: {"cpu_p":4.75,"user_p":3.5,"system_p":1.25,"cpu0.p_cpu":4.0,"cpu0.p_user":3.0,"cpu0.p_system":1.0,"cpu1.p_cpu":5.0,"cpu1.p_user":4.0,"cpu1.p_system":1.0,"cpu2.p_cpu":3.0,"cpu2.p_user":2.0,"cpu2.p_system":1.0,"cpu3.p_cpu":5.0,"cpu3.p_user":4.0,"cpu3.p_system":1.0}
31+
$ fluent-bit -i forward://192.168.3.2:9090 -o stdout
12032
```
12133

122-
So we gathered [CPU](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/input/cpu.md) metrics and flushed them out to [Fluentd](http://fluentd.org) properly.
34+
In the example the Forward messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090.
12335

124-
## Fluent Bit + Secure Forward Setup <a id="secure_forward_setup"></a>
36+
### Configuration File
12537

126-
> DISCLAIMER: the following example do not consider the generation of certificates for a proper usage of production environments.
127-
128-
Secure Forward aims to provide a secure channel of communication with the remote Fluentd service using [TLS](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/configuration/tls_ssl.md). Above there is a minimalist configuration for testing purposes.
129-
130-
### Fluent Bit
131-
132-
Paste this content in a file called _flb.conf_:
133-
134-
```text
135-
[SERVICE]
136-
Flush 5
137-
Daemon off
138-
Log_Level info
38+
In your main configuration file append the following _Input_ & _Output_ sections:
13939

40+
```python
14041
[INPUT]
141-
Name cpu
142-
Tag cpu_usage
42+
Name forward
43+
Listen 0.0.0.0
44+
Port 24224
45+
Buffer_Chunk_Size 32KB
46+
Buffer_Max_Size 64KB
14347

14448
[OUTPUT]
145-
Name forward
146-
Match *
147-
Host 127.0.0.1
148-
Port 24284
149-
Shared_Key secret
150-
Self_Hostname flb.local
151-
tls on
152-
tls.verify off
49+
Name stdout
50+
Match *
15351
```
15452

155-
### Fluentd
53+
## Testing
15654

157-
Paste this content in a file called _fld.conf_:
55+
Once Fluent Bit is running, you can send some messages using the _fluent-cat_ tool \(this tool is provided by [Fluentd](http://www.fluentd.org):
15856

159-
```text
160-
<source>
161-
@type secure_forward
162-
self_hostname myserver.local
163-
shared_key secret
164-
secure no
165-
</source>
166-
167-
<match **>
168-
@type stdout
169-
</match>
170-
```
171-
172-
If you're using Fluentd v1, set up it as below:
173-
174-
```text
175-
<source>
176-
@type forward
177-
<transport tls>
178-
cert_path /etc/td-agent/certs/fluentd.crt
179-
private_key_path /etc/td-agent/certs/fluentd.key
180-
private_key_passphrase password
181-
</transport>
182-
<security>
183-
self_hostname myserver.local
184-
shared_key secret
185-
</security>
186-
</source>
187-
188-
<match **>
189-
@type stdout
190-
</match>
191-
```
192-
193-
### Test Communication
194-
195-
Start Fluentd:
196-
197-
```text
198-
$ fluentd -c fld.conf
57+
```bash
58+
$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | fluent-cat my_tag
19959
```
20060

201-
Start Fluent Bit:
61+
In [Fluent Bit](http://fluentbit.io) we should see the following output:
20262

203-
```text
204-
$ fluent-bit -c flb.conf
205-
```
63+
```bash
64+
$ bin/fluent-bit -i forward -o stdout
65+
Fluent-Bit v0.9.0
66+
Copyright (C) Treasure Data
20667

207-
After five seconds, Fluent Bit will write the records to Fluentd. In Fluentd output you will see a message like this:
208-
209-
```text
210-
2017-03-23 13:34:40 -0600 [info]: using configuration file: <ROOT>
211-
<source>
212-
@type secure_forward
213-
self_hostname myserver.local
214-
shared_key xxxxxx
215-
secure no
216-
</source>
217-
<match **>
218-
@type stdout
219-
</match>
220-
</ROOT>
221-
2017-03-23 13:34:41 -0600 cpu_usage: {"cpu_p":1.0,"user_p":0.75,"system_p":0.25,"cpu0.p_cpu":1.0,"cpu0.p_user":1.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":1.0,"cpu1.p_system":1.0,"cpu2.p_cpu":1.0,"cpu2.p_user":1.0,"cpu2.p_system":0.0,"cpu3.p_cpu":2.0,"cpu3.p_user":1.0,"cpu3.p_system":1.0}
222-
2017-03-23 13:34:42 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.75,"system_p":0.0,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":1.0,"cpu3.p_system":0.0}
223-
2017-03-23 13:34:43 -0600 cpu_usage: {"cpu_p":1.75,"user_p":1.25,"system_p":0.5,"cpu0.p_cpu":3.0,"cpu0.p_user":3.0,"cpu0.p_system":0.0,"cpu1.p_cpu":2.0,"cpu1.p_user":2.0,"cpu1.p_system":0.0,"cpu2.p_cpu":0.0,"cpu2.p_user":0.0,"cpu2.p_system":0.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0}
224-
2017-03-23 13:34:44 -0600 cpu_usage: {"cpu_p":5.0,"user_p":3.25,"system_p":1.75,"cpu0.p_cpu":4.0,"cpu0.p_user":2.0,"cpu0.p_system":2.0,"cpu1.p_cpu":8.0,"cpu1.p_user":5.0,"cpu1.p_system":3.0,"cpu2.p_cpu":4.0,"cpu2.p_user":3.0,"cpu2.p_system":1.0,"cpu3.p_cpu":4.0,"cpu3.p_user":2.0,"cpu3.p_system":2.0}
68+
[2016/10/07 21:49:40] [ info] [engine] started
69+
[2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224
70+
[0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}]
22571
```
226-

0 commit comments

Comments
 (0)