-
Notifications
You must be signed in to change notification settings - Fork 151
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update deb/rpm and installer script for fluentd (#52)
- Loading branch information
Showing
17 changed files
with
564 additions
and
129 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
46 changes: 46 additions & 0 deletions
46
internal/buildscripts/packaging/fpm/etc/otel/collector/fluentd/README
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
Custom TD Agent Fluentd Configuration for the Splunk OpenTelemetry Collector | ||
|
||
This directory contains a custom fluentd configuration to forward log events | ||
to the Splunk OpenTelemetry Collector. By default, the collector will listen | ||
on 127.0.0.1:8006 for log events forwarded from fluentd. See the | ||
"fluentforward" receiver in the default collector config at | ||
/etc/otel/collector/splunk_config_linux.yaml for details or to make any changes | ||
to the collector. | ||
|
||
Directory contents: | ||
|
||
- splunk-otel-collector.conf: Drop-in file for the fluentd service. As an | ||
alternative to overwriting the default fluentd config file | ||
(/etc/td-agent/td-agent.conf), copy this file to | ||
/etc/systemd/system/td-agent.service.d/splunk-otel-collector.conf to | ||
override the default fluentd config path in favor of the custom | ||
fluentd config file in this directory (see fluent.conf below), and run the | ||
following commands to apply the changes: | ||
|
||
systemctl daemon-reload | ||
systemctl restart td-agent | ||
|
||
- fluent.conf: The main fluentd configuration file to forward events to the | ||
collector. By default, this file will configure fluentd to include custom | ||
fluentd sources from the conf.d sub-directory (see conf.d below) and forward | ||
all log events with the @SPLUNK label to the collector. If changes are made | ||
to this file, run the following command to apply the changes: | ||
|
||
systemctl restart td-agent | ||
|
||
- conf.d: Directory for custom fluentd configuration files. The main fluentd | ||
configuration (see fluent.conf above) will automatically include all files | ||
ending in .conf from the conf.d directory. New fluentd sources should | ||
include the @SPLUNK label for all log events intended to be forwarded to the | ||
collector (see the sample file in conf.d for details). After adding new | ||
config files to the conf.d directory, run the following command to apply the | ||
changes: | ||
|
||
systemctl restart td-agent | ||
|
||
*Important*: By default, the fluentd service runs as the "td-agent" user. | ||
When adding new fluentd source configurations, ensure that the "td-agent" | ||
user has permissions to access the paths defined in these sources. | ||
|
||
See https://docs.fluentd.org/configuration for general fluentd configuration | ||
details. |
55 changes: 55 additions & 0 deletions
55
internal/buildscripts/packaging/fpm/etc/otel/collector/fluentd/conf.d/journald.conf.example
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
# Sample fluentd source config to read from journald and forward all log events | ||
# to the Splunk OpenTelemetry Collector. | ||
# | ||
# To enable this source, perform the following steps: | ||
# | ||
# 1. Ensure the systemd fluentd plugin is installed by running: | ||
# | ||
# td-agent-gem list fluent-plugin-systemd | ||
# | ||
# If necessary, install the plugin by running: | ||
# | ||
# td-agent-gem install fluent-plugin-systemd | ||
# | ||
# Note: Additional dependencies may be required to be installed/upgraded in | ||
# order to successfully install the plugin. See | ||
# https://github.com/fluent-plugin-systemd/fluent-plugin-systemd for | ||
# installation and configuration details. | ||
# | ||
# 2. Ensure that the fluentd service is configured to forward events to the | ||
# collector by checking if the drop-in file | ||
# "/etc/systemd/system/td-agent.service.d/splunk-otel-collector.conf" | ||
# exists. If not, copy "/etc/otel/collector/splunk-otel-collector.conf" to | ||
# "/etc/systemd/system/td-agent.service.d/splunk-otel-collector.conf" and | ||
# run: | ||
# | ||
# systemctl daemon-reload | ||
# | ||
# 3. Make any necessary changes to the configuration below and copy/rename this | ||
# file to "/etc/otel/collector/fluentd/conf.d/journald.conf". See | ||
# https://github.com/fluent-plugin-systemd/fluent-plugin-systemd for | ||
# details and other options. | ||
# | ||
# 4. Since the fluentd service runs as the "td-agent" user, ensure that this | ||
# user has permissions to access the path configured below. For example, if | ||
# the "/run/log/journal" path is only readable by the "systemd-journal" | ||
# group, it may be necessary to add the "td-agent" user to the group by | ||
# running the command: | ||
# | ||
# usermod -a -G systemd-journal td-agent | ||
# | ||
# 5. Restart the fluentd service to apply the changes by running: | ||
# | ||
# systemctl restart td-agent | ||
# | ||
|
||
<source> | ||
@type systemd | ||
@label @SPLUNK | ||
tag "journald" | ||
path "/run/log/journal" | ||
<entry> | ||
fields_strip_underscores true | ||
fields_lowercase true | ||
</entry> | ||
</source> |
26 changes: 26 additions & 0 deletions
26
internal/buildscripts/packaging/fpm/etc/otel/collector/fluentd/fluent.conf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
@include conf.d/*.conf | ||
|
||
<label @SPLUNK> | ||
<match **> | ||
@type forward | ||
heartbeat_type udp | ||
<server> | ||
host 127.0.0.1 | ||
port 8006 | ||
</server> | ||
<buffer> | ||
@type memory | ||
total_limit_size 600m | ||
chunk_limit_size 1m | ||
chunk_limit_records 100000 | ||
flush_interval 5s | ||
flush_thread_count 1 | ||
overflow_action block | ||
retry_max_times 3 | ||
</buffer> | ||
</match> | ||
</label> | ||
|
||
<system> | ||
log_level info | ||
</system> |
11 changes: 11 additions & 0 deletions
11
internal/buildscripts/packaging/fpm/etc/otel/collector/fluentd/splunk-otel-collector.conf
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Drop-in file for the fluentd service | ||
|
||
# Copy this file to /etc/systemd/system/td-agent.service.d/ in order to override | ||
# the default fluentd config path in favor of the custom config defined below. | ||
|
||
# Then run the following commands to apply the changes: | ||
# systemctl daemon-reload | ||
# systemctl restart td-agent | ||
|
||
[Service] | ||
Environment=FLUENT_CONF=/etc/otel/collector/fluentd/fluent.conf |
Oops, something went wrong.