Note
Graylog 3.0 comes with a new Sidecar implementation.
We still support the old Collector Sidecars, which can be found in the System / Collectors (legacy)
menu entry.
In case you need to configure legacy Collector Sidecar please refer to the Graylog Collector Sidecar documentation.
We encourage users to migrate to the new Sidecar, which is covered by this document.
Graylog Sidecar is a lightweight configuration management system for different log collectors, also called Backends. The Graylog node(s) act as a centralized hub containing the configurations of log collectors. On supported message-producing devices/hosts, Sidecar can run as a service (Windows host) or daemon (Linux host).
The log collector configurations are centrally managed through the Graylog web interface. Periodically, the Sidecar daemon will fetch all relevant configurations for the target, using the :doc:`REST API <configuration/rest_api>`. On its first run, or when a configuration change has been detected, Sidecar will generate (render) relevant backend configuration files. Then it will start, or restart, those reconfigured log collectors.
You can get .deb and .rpm packages for Graylog Sidecar in our package repository. For Windows, you can download the installer from here.
Please follow the version matrix to pick the right package:
Sidecar version | Graylog server version |
---|---|
1.1.x | 3.2.5 or higher |
1.0.x | 3.0 or higher |
0.1.x | 2.2.x,2.3.x,2.4.x,2.5.x,3.0.x |
0.0.9 | 2.1.x |
All following commands should be executed on the remote machine where you want to collect log data from.
Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:
$ wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb $ sudo dpkg -i graylog-sidecar-repository_1-2_all.deb $ sudo apt-get update && sudo apt-get install graylog-sidecar
Edit the configuration (see :ref:`Configuration <sidecar-configuration>`) and activate the Sidecar as a system service:
$ vi /etc/graylog/sidecar/sidecar.yml $ sudo graylog-sidecar -service install [Ubuntu 14.04 with Upstart] $ sudo start graylog-sidecar [Ubuntu 16.04 and later with Systemd] $ sudo systemctl start graylog-sidecar
Install the Graylog Sidecar repository configuration and Graylog Sidecar itself with the following commands:
$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-sidecar-repository-1-2.noarch.rpm $ sudo yum update && sudo yum install graylog-sidecar
Edit the configuration (see :ref:`Configuration <sidecar-configuration>`) and activate the Sidecar as a system service:
$ vi /etc/graylog/sidecar/sidecar.yml $ sudo graylog-sidecar -service install $ sudo systemctl start graylog-sidecar
Use the Windows installer, it can be run interactively:
$ graylog_sidecar_installer_1.0.0-1.exe
Or in silent mode with:
$ graylog_sidecar_installer_1.0.0-1.exe /S -SERVERURL=http://10.0.2.2:9000/api -APITOKEN=yourapitoken
Optionally edit the configuration (see :ref:`Configuration <sidecar-configuration>`) and register the system service:
notepad.exe C:\Program Files\Graylog\sidecar\sidecar.yml & "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install & "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start
Next up, you can decide which collectors you want to use with your Sidecar and install them as well. We only cover the installation of the most common ones here, but you are free to use other collectors as well. Graylog contains default collector configurations for Filebeat, Winlogbeat and NXLog. But since you're able to define your own collector backends, there is nothing stopping you from running e.g. sysmon, auditd or packetbeat.
Install Filebeat or another Beats package by following the instructions on the official Filebeat download page.
The Windows Sidecar package already includes Filebeat and Winlogbeat. For other Beats packages follow the instructions on the official Beats download page.
Install the NXLog package from the official NXLog download page. Because the Sidecar takes control of stopping and starting NXlog it is necessary to stop all running instances of NXlog and unconfigure the default system service:
$ sudo /etc/init.d/nxlog stop $ sudo update-rc.d -f nxlog remove $ sudo gpasswd -a nxlog adm $ sudo chown -R nxlog.nxlog /var/spool/nxlog
The same on a RedHat based system:
$ sudo service nxlog stop $ sudo chkconfig --del nxlog $ sudo gpasswd -a nxlog root $ sudo chown -R nxlog.nxlog /var/spool/nxlog
Install the NXLog package from the official download page and deactivate the system service. We just need the binaries installed on the system:
$ C:\Program Files (x86)\nxlog\nxlog -u
On the command line you can provide a path to the configuration file with the -c
switch.
The default configuration path on Linux systems is /etc/graylog/sidecar/sidecar.yml
and C:\Program Files\Graylog\sidecar\sidecar.yml
on Windows.
Most configuration parameters come with built-in defaults.
The only parameters that need adjustment are server_url
and server_api_token
.
You can get your API token by following the link on the :ref:`Sidecars Overview <sidecar_overview>` page.
Parameter | Description |
---|---|
server_url | URL to the Graylog API, e.g. http://192.168.1.1:9000/api/ |
server_api_token | The API token to use to authenticate against the Graylog server API. e.g 1jq26cssvc6rj4qac4bt9oeeh0p4vt5u5kal9jocl1g9mdi4og3n The token is mandatory and needs to be configured. |
node_id | The node ID of the sidecar. This can be a path to a file or an ID string. Example file path: file:/etc/graylog/sidecar/node-id Example ID string: 6033137e-d56b-47fc-9762-cd699c11a5a9 ATTENTION: Every sidecar instance needs a unique ID! Default: file:/etc/graylog/sidecar/node-id |
node_name | Name of the Sidecar instance, will also show up in the web interface. The hostname will be used if not set. |
update_interval | The interval in seconds the sidecar will fetch new configurations from the Graylog server Default: 10 |
tls_skip_verify | This configures if the sidecar should skip the verification of TLS connections. Default: false |
send_status | This controls the transmission of detailed sidecar information like collector status, metrics and log file lists. It can be disabled to reduce load on the Graylog server if needed. Default: true |
list_log_files | Send a directory listing to Graylog and display it on the host status page, e.g. /var/log . This can also be a list of directories. Default: [] |
cache_path | The directory where the sidecar stores internal data. Default: /var/cache/graylog-sidecar |
collector_configuration_directory | The directory where the sidecar generates configurations for collectors. Default: /var/lib/graylog-sidecar/generated |
log_path | The directory where the sidecar stores its logs. Default: /var/log/graylog-sidecar |
log_rotate_max_file_size | The maximum size of the log file before it gets rotated. Default: 10MiB |
log_rotate_keep_files | The maximum number of old log files to retain. |
collector_binaries_whitelist | A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the white list feature. Default: /usr/bin/filebeat, /usr/bin/packetbeat, /usr/bin/metricbeat, /usr/bin/heartbeat, /usr/bin/auditbeat, /usr/bin/journalbeat, /usr/share/filebeat/bin/filebeat, /usr/share/packetbeat/bin/packetbeat, /usr/share/metricbeat/bin/metricbeat, /usr/share/heartbeat/bin/heartbeat, /usr/share/auditbeat/bin/auditbeat, /usr/share/journalbeat/bin/journalbeat, /usr/bin/nxlog, /opt/nxlog/bin/nxlog |
Once you installed the Sidecar package and started the service for the first time, you can verify that it shows up in the :ref:`Sidecars Overview <sidecar_overview>` page. A new sidecar instance will not have any configurations assigned yet. Take the :ref:`sidecar_step-by-step` to create your first configuration.
When the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the
collector_configuration_directory
directory for each collector backend. E.g. if you assigned a Filebeat collector you will find a
filebeat.yml
file in that directory. All changes have to be made in the Graylog web interface.
Every time the Sidecar detects an update to its configuration it will
rewrite the corresponding collector configuration file. Manually editing these files is not recommended.
Every time a collector configuration file is changed the collector process is restarted. The Sidecar takes care of the collector processes and reports the status back to the web interface
Each Sidecar instance is able to send status information back to Graylog. By enabling the option send_status
metrics like load or the IP address of the host Sidecar is running on
are sent. Also metrics that are relevant for a stable operation e.g. disk volumes over 75% utilization are included. Additionally with the list_log_files
option a directory listing is displayed in
the Graylog web interface. In that way an administrator can see which files are available for collecting. The list is periodically updated and files with write access are highlighted for easy identification.
After enabling send_status
or send_status
+ list_log_files
go to the collector overview and click on one of them, a status page with the configured information will be displayed.
We have prepared an example on how to configure the Sidecar using the Graylog web interface. The assumption is that we want to collect Apache logfiles and ship them with a Filebeat collector to a Beats input that is listening on Port 5044 on your Graylog Server.
- The first step is to create a Beats input where collectors can send data to. Click on
System / Inputs
and start a global Beats input on the listening address 0.0.0.0 and port 5044.
- Navigate to the Sidecars overview. In your Graylog web interface click on
System / Sidecars
.
- Navigate to the Sidecar
Configuration
page.
- Next we create a new configuration: We give the configuration a name and select
filebeat on Linux
as collector. (This collector definition is shipped with Graylog, and comes with a default configuration template). Most of the configuration defaults should work for you. However you need to change thehosts:
setting and point it to your Beats input. You also might want to change thepaths:
to the location of your Apache logs. When done clickCreate
to save your configuration.
- Next we need to assign our newly created configuration (and therefore the Filebeat collector) to our sidecar.
Go to the
Collector Administration
page.
- You will see a list of sidecars and underneath them a list of collectors that could be assigned to them.
Please note that collectors are assigned to sidecars by means of applying a collector configuration to the sidecar.
Therefore, we first select the
filebeat
collector and then click on theConfigure
menu, where we can select thefilebeat-conf
configuration we created earlier.
- Confirming the assignment, will directly push this configuration to your sidecar which will go and start the Filebeat collector with this configuration.
- If everything went fine, you should see the status
running
on the administration page.
- Congratulations your collector setup is working now!
You can go back to the Sidecars overview and click on the
Show messages
button to search for logs that have been collected via your sidecar.
Graylog comes with a few predefined log collectors which can be easily extended and changed to your needs. Let's assume you want your sidecar to run rsyslogd(8) for you.
- Navigate to the Sidecars overview. In your Graylog web interface click on
System / Sidecars
.
- Navigate to the Sidecar
Configuration
page.
- After we click on
Create Log Collector
, we are presented with the following page, where we have to fill out some fields for our new collector. We give the collector a unique name and selectLinux
andForeground Execution
. Given that you installed rsyslogd(8) under/usr/sbin/rsyslogd
we configure the executable path accordingly. If you are usingForeground Execution
make sure that the collector you are running does not daemonize itself. Otherwise the sidecar has no way of controlling the collector once it has forked off into the background. For rsyslogd we therefore provide-n
as Execute Parameter. If your collector supports configuration validation, it is advised to use it. This acts as a pre-check, so that sidecar won't restart a collector with a broken configuration. For rsyslogd the option to do a configuration check is-N 1
. Optionally you can provide a Default Template which will be proposed once you create a configuration for this collector.
- Next up you can use your newly created collector by creating a configuration for it and assign it to a Sidecar. Please follow the :ref:`sidecar_step-by-step` accordingly.
- Note: Your Sidecar might refuse to start your collector, because it needs
to be added to the
collector_binaries_whitelist
first. Please edit your :ref:`Configuration <sidecar-configuration>` and restart your Sidecar.
Configuration variables can contain arbitrary strings like the IP address of your Graylog server or the port of an input. The variables can then be used in multiple collector configurations, which avoids duplication and simplifies management.
To create a configuration variable go any Collector Configuration
page:
On the right you'll find a box Collector Configuration Reference
which
contains Runtime Variables and Variables.
Click on Variables
and then Create Variable
to receive the following
modal:
In this example we replace the hard coded IP and Port from our
Beats input with a new variable named ${user.BeatsInput}
:
We can now use this variable in all our configurations. If we ever need to change the IP/port of our input, we just change the variable.
Runtime variables contain runtime informations from each Sidecar that
is requesting this configuration.
An important example is the ${sidecar.nodeId}
variable.
The collector configuration should contain an instruction to fill
that variable in an extra field gl2_source_collector.
This allows Graylog to relate messages to the Sidecar that produced
them. (This is what makes the Show messages
button on the Sidecars overview page work)
The Communication between Sidecar and Graylog will be secured if your API :ref:`uses SSL <https_setup>`.
To secure the communication between the Collector and Graylog you just need to mark Enable TLS
in your Beats Input. Without giving additional Information, Graylog will now create a self-signed certificate for this Input.
Now in the Sidecar Beats Output Configuration you just mark Enable TLS Support
and Insecure TLS connection
. After this is saved, the communication between Beats and Graylog will use TLS.
If you want Graylog to only accept data from authenticated Collectors please follow the steps at :ref:`Secured Graylog and Beats input <sec_graylog_beats>`
The default is that the Sidecar is started with the root user to allow access to all log files. But this is not mandatory. If you like to start it with a daemon user, proceed like the following:
- Create a daemon user e.g.
sidecar
The Sidecar itself is accessing the following files and directories:
sidecar.yml
- /etc/graylog/sidecar/sidecar.ymlcollector_configuration_directory
- /var/lib/graylog-sidecar/generated/node_id
- /etc/graylog/sidecar/node-idcache_path
- /var/cache/graylog-sidecar/log_path
- /var/log/graylog-sidecar/
So to make these directories readable for the sidecar
user, use:
chown -R sidecar /etc/graylog/sidecar
chown -R sidecar /var/cache/graylog-sidecar
chown -R sidecar /var/lib/graylog-sidecar
chown -R sidecar /var/log/graylog-sidecar
You can change all paths to different places in the file system. If you prefer to store all Sidecar data in the home directory of the sidecar
user, just change the paths accordingly.
Now systemd
needs to know that the Sidecar should be started with a non-root user. Open /etc/systemd/system/collector-sidecar.service
with an editor and navigate to the [Service]
section, add:
User=sidecar Group=sidecar
To make use of these settings reload systemd:
$ sudo systemctl daemon-reload $ sudo systemctl restart graylog-sidecar
Check the log files in /var/log/graylog-sidecar
for any errors. Understand that not only the Sidecar but also all backends, like filebeat
, will be started as sidecar
user after these changes.
So all log files that the backend should observe also need to be readable by the sidecar
user. Depending on the Linux distribution there is usually an administrator group which has access to most log files.
By adding the sidecar
user to that group you can grant access fairly easy. For example on Debian/Ubuntu systems this group is called adm
(see System Groups in Debian Wiki or Security/Privileges - Monitor system logs in Ubuntu wiki).
This guide describes how you can perform an upgrade from the deprecated Collector Sidecars (0.1.x) to the new Sidecars (1.x).
One major difference between the old and the new Sidecars, is that we replaced the UI based collector configuration approach with one where you can manage the plain text configuration of the collectors directly. This might seem like an inconvenience at first, but gives you the flexibility to configure any collector backend you want.
Additionally, the new Sidecars don't assign configurations based on tags anymore. Instead you have to assign configurations explicitly (see :ref:`Step-by-Step guide <sidecar_assign_config_sbs>`).
The new Sidecar has different paths and executable names, so it can coexist with the old one. Install the new Sidecar by following the :ref:`Installation instructions <sidecar_installation>` and have your Sidecar running as described in :ref:`First Start <sidecar_first_start>`.
Note: In case you were using filebeat on Linux, please make sure to also install the official collector package, since the filebeat binary is not part of the Sidecar package anymore.
Next, we need to migrate the configuration that was previously rendered on each host by the Collector Sidecar, to a new Collector Configuration.
We recommend to use the :ref:`Sidecar Configuration Migrator <config_migrator>`.
However, retrieving the old configuration can also be done manually by fetching it from
your host at the /etc/graylog/collector-sidecar/generated/
directory.
There are a few things that might need attention after an upgrade:
Use :ref:`Runtime variables <sidecar_runtime_variables>` for static fields
The imported configuration contains instructions that add static fields which allows Graylog to relate messages to a Sidecar. You should replace the hardcoded values of
gl2_source_collector
andcollector_node_id
with runtime variables.In case of a Beats collector this would be:
fields.gl2_source_collector: ${sidecar.nodeId} fields.collector_node_id: ${sidecar.nodeName}
Migrate to the new Beats input
Graylog 3.0 comes with a new Beats input. The former one was renamed to
Beats (deprecated)
. The new input handles fields a little different. Therefore you should definefields_under_root: true
for the new input to get the Graylog fields work.
Once you're done creating a new configuration, you can assign it to your Sidecar (see :ref:`Step-by-Step guide <sidecar_assign_config_sbs>`). If everything works as expected, make sure to uninstall the old Collector Sidecar to avoid collecting your logs twice.
The task of the Sidecar configuration migrator is to extract the configuration from existing Collector Sidecars and convert it into new Sidecar configurations.
This feature needs a Collector Sidecar with version 0.1.8 or greater. Please upgrade the instance you want to import configurations from, if necessary.
- Navigate to the Collectors (legacy) overview. In your Graylog web interface click on
System / Collectors (legacy)
.
- Click on the name of the Collector you want to import configurations from
- Click the
Import Configuration
button on a backend to import a configuration. If the import was successful, follow the link to create a new Sidecar configuration:
- After clicking on
Create Configuration
use theMigrate
button underneath the configuration editor:
- A window opens up and lets you pick already imported configurations.
Clicking
Apply
will paste the configuration into the editor. Afterwards you can edit and save the configuration as usual.
To understand the different parts of the Graylog Sidecar they are explained in the following section.
A configuration is the representation of a log collector configuration file in the Graylog web interface. A configuration can be assigned to Sidecars, which also assigns the corresponding collector. You can have multiple configurations for a single log collector. However, you can not assign the same collector twice to a Sidecar.
Inputs are the way how collectors ingest data. An input can be a log file that the collector should continuously read or a connection to the Windows event system that emits log events. An input is connected to an output, otherwise there would be no way of sending the data to the next hop. So first create an output and then associate one or many inputs with it.
The Sidecar is writing log files to the directory configured in log_path
. One file for each backend, there you can check for general issues like
file permissions or log transmission problems. The Sidecar itself is writing to sidecar.log
. Problems like failed connection to the Graylog API can
be found there.
You can also start the Sidecar in foreground and monitor the output of the process:
$ graylog-sidecar -debug
On Linux just uninstall the package, to perform an uninstall on Windows run:
& "C:\Program Files\Graylog\graylog-sidecar.exe" -service stop & "C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall
Currently we know of two problems with NXLog:
- Since version 2.9.17 timestamps are transmitted without millisecond precision
- On Windows machines NXlog is not able to store its collector state so features like file tailing don't work correctly in combination with Sidecar. Use Sidecar version 0.1.0-alpha.1 or newer.
Known issue if you use a loadbalancer or firewall in front of Graylog's API:
- The Sidecar is using a persistent connection for API requests. Therefore it logs
408 Request Time-out
if the loadbalancer session or http timeout is lower than the configuredupdate_interval
.