Skip to content

Commit 18263cd

Browse files
ganmacsganmacs
committed
Add service discovery document
Signed-off-by: Yuta Iwama <ganmacs@gmail.com>
1 parent 71e0299 commit 18263cd

File tree

14 files changed

+359
-15
lines changed

14 files changed

+359
-15
lines changed

‎SUMMARY.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
* [Config: Inject Section](/configuration/inject-section.md)
3030
* [Config: Transport Section](/configuration/transport-section.md)
3131
* [Config: Storage Section](/configuration/storage-section.md)
32+
* [Config: Service Discovery Section](/configuration/service_discovery-section.md)
3233

3334
* [Deployment]()
3435
* [System Configuration](/deployment/system-config.md)
@@ -123,6 +124,10 @@
123124
* [Storage Plugins](/plugins/storage/README.md)
124125
* [local](/plugins/storage/local.md)
125126

127+
* [Service Discovery Plugins](/plugins/service_discovery/README.md)
128+
* [static](/plugins/service_discovery/static.md)
129+
* [file](/plugins/service_discovery/file.md)
130+
126131
* [How-to Guides]()
127132
* [Send Apache Logs to S3](/guides/apache-to-s3.md)
128133
* [Send Apache Logs to Minio](/guides/apache-to-minio.md)
@@ -157,6 +162,7 @@
157162
* [How to Write Output Plugin](/developer/api-plugin-output.md)
158163
* [How to Write Parser Plugin](/developer/api-plugin-parser.md)
159164
* [How to Write Storage Plugin](/developer/api-plugin-storage.md)
165+
* [How to Write Service Discovery Plugin](/developer/api-plugin-service_discovery.md)
160166
* [How to Write Tests for Plugin](/developer/plugin-test-code.md)
161167
* [Configuration Parameter Types](/developer/api-config-types.md)
162168
* [Upgrade Plugin from v0.12](/developer/plugin-update-from-v0.12.md)

‎configuration/service_discovery-section.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Service Discovery section configurations
2+
3+
Some of Fluentd's plugins support the `<service_discovery>` section to set target nodes dynamiclly
4+
5+
## Service Discovery section overview
6+
7+
Service discovery section can be used in `<match>`.
8+
9+
```
10+
<match tag.*>
11+
@type forward
12+
# parameters for output plugin
13+
<service_discovery>
14+
# service_discovery section parameters
15+
</service_discovery>
16+
</match>
17+
```
18+
19+
## Service Discovery plugin type
20+
21+
`<service_discovery>` section requires `@type` parameter to specify the type of
22+
service_discovery plugin. Fluentd core bundles [some useful service discovery plugins](/plugins/service_discovery/README.md).
23+
24+
```
25+
<service_discovery>
26+
@type file
27+
</service_discovery>
28+
```
29+
30+
These are the list of built-in service discovery plugins.
31+
32+
- [static](/plugins/service_discovery/static.md)
33+
- [file](/plugins/service_discovery/file.md)
34+
35+
For more details, see plugins documentation.
36+
37+
### @type
38+
39+
`@type` key is to specify the type of service discovery plugin.
40+
41+
```
42+
<service_discovery>
43+
@type static
44+
# ...
45+
</service_discovery>
46+
```
47+
48+
------------------------------------------------------------------------
49+
50+
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open).
51+
[Fluentd](http://www.fluentd.org/) is a open source project under [Cloud Native Computing Foundation (CNCF)](https://cncf.io/). All components are available under the Apache 2 License.

‎developer/api-plugin-service_discovery.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Writing Service Discovery
2+
3+
4+
TODO

‎developer/plugin-development.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Alternatively, you can create a Ruby Gem package that includes a
1616
- `formatter` for formatter plugins
1717
- `storage` for storage plugins
1818
- `buf` for buffer plugins
19+
- `sd` for service discovery
1920

2021
For example, an email Output plugin would have the path:
2122
`lib/fluent/plugin/out_mail.rb`. The packaged gem can be distributed and

‎plugins/buffer/README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Buffer Plugin Overview
22

3-
Fluentd has 7 types of plugins: [Input](/plugins/input/README.md),
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
44
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
55
[Output](/plugins/output/README.md),
66
[Formatter](/plugins/formatter/README.md),
7-
[Storage](/plugins/storage/README.md) and [Buffer](/plugins/buffer/README.md).
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
89
This article gives an overview of Buffer Plugin.
910

1011

‎plugins/filter/README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Filter Plugin Overview
22

3-
Fluentd has 7 types of plugins: [Input](/plugins/input/README.md),
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
44
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
55
[Output](/plugins/output/README.md),
66
[Formatter](/plugins/formatter/README.md),
7-
[Storage](/plugins/storage/README.md) and [Buffer](/plugins/buffer/README.md).
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
89
This article gives an overview of Filter Plugin.
910

1011

‎plugins/formatter/README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Formatter Overview
22

3-
Fluentd has 7 types of plugins: [Input](/plugins/input/README.md),
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
44
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
55
[Output](/plugins/output/README.md),
66
[Formatter](/plugins/formatter/README.md),
7-
[Storage](/plugins/storage/README.md) and [Buffer](/plugins/buffer/README.md).
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
89
This article gives an overview of Formatter Plugin.
910

1011

‎plugins/input/README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Input Plugin Overview
22

3-
Fluentd has 7 types of plugins: [Input](/plugins/input/README.md),
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
44
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
55
[Output](/plugins/output/README.md),
66
[Formatter](/plugins/formatter/README.md),
7-
[Storage](/plugins/storage/README.md) and [Buffer](/plugins/buffer/README.md).
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
89
This article gives an overview of Input Plugin.
910

1011

‎plugins/output/README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Output Plugin Overview
22

3-
Fluentd has 7 types of plugins: [Input](/plugins/input/README.md),
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
44
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
55
[Output](/plugins/output/README.md),
66
[Formatter](/plugins/formatter/README.md),
7-
[Storage](/plugins/storage/README.md) and [Buffer](/plugins/buffer/README.md).
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
89
This article gives an overview of Output Plugin.
910

1011

‎plugins/parser/README.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
# Parser Plugin Overview
22

3-
Fluentd has 7 types of plugins: [Input](/plugins/input/README.md),
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
44
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
55
[Output](/plugins/output/README.md),
66
[Formatter](/plugins/formatter/README.md),
7-
[Storage](/plugins/storage/README.md) and [Buffer](/plugins/buffer/README.md).
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
89
This article gives an overview of Parser Plugin.
910

1011

‎plugins/service_discovery/README.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Service Discovery Plugin Overview
2+
3+
Fluentd has 8 types of plugins: [Input](/plugins/input/README.md),
4+
[Parser](/plugins/parser/README.md), [Filter](/plugins/filter/README.md),
5+
[Output](/plugins/output/README.md),
6+
[Formatter](/plugins/formatter/README.md),
7+
[Storage](/plugins/storage/README.md),
8+
[Service Discovery](/plugins/service_discovery/README.md) and [Buffer](/plugins/buffer/README.md).
9+
This article gives an overview of Service Discovery Plugin.
10+
11+
## Overview
12+
13+
Some of the plugins has the `<service_discovery>` (ex: [out\_forward](/plugins/output/forward.md)).
14+
Sometimes, the service discovery for an output plugin doesn't meet one's
15+
needs. Fluentd has a pluggable system called Service Discovery that lets the
16+
user extend and re-use custom output service discovery.
17+
18+
19+
## How To Use
20+
21+
Here is a simple example to update target by reading file(`/etc/fluentd/sd.yaml`) with `out_forward` and
22+
`service_discovery`:
23+
24+
```
25+
<source>
26+
@type forward
27+
28+
<service_discovery>
29+
@type file
30+
path "/etc/fluentd/sd.yaml"
31+
</service_discovery>
32+
</source>
33+
```
34+
35+
36+
## List of Built-in Service Discovery
37+
38+
- [static](/plugins/service_discovery/static.md)
39+
- [file](/plugins/service_discovery/file.md)
40+
41+
## List of Core Input Plugins with Parser support
42+
43+
with `<service_discovery>` directive.
44+
45+
- [out\_forward](/plugins/output/forward.md)
46+
47+
------------------------------------------------------------------------
48+
49+
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open).
50+
[Fluentd](http://www.fluentd.org/) is a open source project under [Cloud Native Computing Foundation (CNCF)](https://cncf.io/). All components are available under the Apache 2 License.

‎plugins/service_discovery/file.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# File Service Discovery Plugin
2+
3+
The `file` serivce discovery plugin updates targets by reading local file.
4+
YAML and JSON are allowed as the file formats.
5+
6+
## Example Configuration
7+
8+
This is the exmple with out_forward.
9+
It udpates targets to send data by out_forward.
10+
11+
```
12+
<source>
13+
@type forward
14+
15+
<service_discovery>
16+
@type file
17+
path "/etc/fluentd/sd.yaml"
18+
</service_discovery>
19+
</source>
20+
```
21+
22+
Here is the example of target list file(`/etc/fluentd/sd.yaml`).
23+
24+
```
25+
- 'host': 127.0.0.1
26+
'port': 24224
27+
'weight': 1
28+
'name': server1
29+
- 'host': 127.0.0.1
30+
'port': 24225
31+
'weight': 1
32+
'name': server2
33+
```
34+
35+
## Parameters
36+
37+
### @type
38+
39+
The value must be `file`.
40+
41+
### path
42+
43+
| type | default | version |
44+
|:-------|:------------------------|:--------|
45+
| string | `'/etc/fluent/sd.yaml'` | 0.17.0 |
46+
47+
The path of target list.
48+
49+
### conf_encoding
50+
51+
| type | default | version |
52+
|:-------|:----------|:--------|
53+
| string | `'utf-8'` | 0.17.0 |
54+
55+
The encoding of config file.
56+
57+
### Parameters in target list file
58+
59+
Each target has following parameters.
60+
61+
- host
62+
- name
63+
- port
64+
- shared\_key
65+
- username
66+
- password
67+
- standby
68+
- weight
69+
70+
#### host
71+
72+
| type | default | version |
73+
|:-------|:-------------------|:--------|
74+
| string | required parameter | 1.7.0 |
75+
76+
The IP address or host name of the server.
77+
78+
#### port
79+
80+
| type | default | version |
81+
|:--------|:-------------------|:--------|
82+
| integer | required parameter | 1.7.0 |
83+
84+
The port number of the host.
85+
86+
#### name
87+
88+
| type | default | version |
89+
|:-------|:--------|:--------|
90+
| string | nil | 1.7.0 |
91+
92+
The name of the server.
93+
94+
#### shared\_key
95+
96+
| type | default | version |
97+
|:-------|:--------|:--------|
98+
| string | nil | 1.7.0 |
99+
100+
The shared key per server.
101+
102+
#### username
103+
104+
| type | default | version |
105+
|:-------|:---------|:--------|
106+
| string | nil | 1.7.0 |
107+
108+
The username for authentication.
109+
110+
#### password
111+
112+
| type | default | version |
113+
|:-------|:--------|:--------|
114+
| string | nil | 1.7.0 |
115+
116+
The password for authentication.
117+
118+
#### standby
119+
120+
| type | default | version |
121+
|:-----|:--------|:--------|
122+
| bool | nil | 1.7.0 |
123+
124+
#### weight
125+
126+
| type | default | version |
127+
|:--------|:--------|:--------|
128+
| integer | 60 | 1.7.0 |
129+
130+
131+
------------------------------------------------------------------------
132+
133+
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open).
134+
[Fluentd](http://www.fluentd.org/) is a open source project under [Cloud Native Computing Foundation (CNCF)](https://cncf.io/). All components are available under the Apache 2 License.

0 commit comments

Comments
 (0)