Cannot successful use more than one winlogbeat configuration

[drewmiranda-gl]

With Graylog 5, a new feature has been introduced to allow for more than one [collector] configuration of the same collector type to be assigned to a sidecar. However, when assigning more than one winlogbeat configuration, only one of them can successfully the start. All others will fail to start. The service won't start. Attempting to run the command the service attempts to run indicates the problem:

C:\Windows\system32>"C:\Program Files\Graylog\sidecar\winlogbeat.exe" -c "C:\Program Files\Graylog\sidecar\generated\639b9ead625ade58bf5e46cd\winlogbeat.conf"
2022-12-15 16:26:00.0762068 -0600 CST m=+0.144902701 write error: failed to rotate backups: failed to rotate backups: rename C:\Program Files\Graylog\sidecar\logs\winlogbeat C:\Program Files\Graylog\sidecar\logs\winlogbeat.1: The process cannot access the file because it is being used by another process.
2022-12-15 16:26:00.080628 -0600 CST m=+0.149323901 write error: failed to rotate backups: failed to rotate backups: rename C:\Program Files\Graylog\sidecar\logs\winlogbeat C:\Program Files\Graylog\sidecar\logs\winlogbeat.1: The process cannot access the file because it is being used by another process.
2022-12-15 16:26:00.0831382 -0600 CST m=+0.151834101 write error: failed to rotate backups: failed to rotate backups: rename C:\Program Files\Graylog\sidecar\logs\winlogbeat C:\Program Files\Graylog\sidecar\logs\winlogbeat.1: The process cannot access the file because it is being used by another process.
2022-12-15 16:26:00.0859584 -0600 CST m=+0.154654301 write error: failed to rotate backups: failed to rotate backups: rename C:\Program Files\Graylog\sidecar\logs\winlogbeat C:\Program Files\Graylog\sidecar\logs\winlogbeat.1: The process cannot access the file because it is being used by another process.
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

Each instance of winlogbeat MUST have a unique working directory or path to its data and logs directorires.

Expected Behavior

Can assign more than one winlogbeat config, OR graylog prevents this since it doesn't work.

Current Behavior

Graylog allows assigning more than one winlogbeat configuration even though only one can successfully start.

Possible Solution

Specify unique paths for data, home, and logs using the winlogbeat arguments:

--path.data string            Data path
--path.home string            Home path
--path.logs string            Logs path

Steps to Reproduce (for bugs)

1. Assign more than one winlogbeat config to a sidecar

Context

It can be very useful to apply more than one winlogbeat configuration, for example to have a number of general purpose configurations that can later be combined to mix and match what configuration is applied. Currently only a single winlogbeat collector can be applied.

Your Environment

• Graylog Version: 5.0.1
• Java Version: 17.0.5
• Elasticsearch Version: OpenSearch 1.3.4
• MongoDB Version: 5.0.14
• Operating System: Ubuntu Server 20.04 LTS
• Browser version: Chrome 108.0.5359.124

[mpfz0r]

@drewmiranda-gl

That's likely because you've been updating with an existing config.
The new templates on a fresh installation look like this, and shouldn't have this problem:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["192.168.1.1:5044"]
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\winlogbeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
   - name: System
   - name: Security

[drewmiranda-gl]

Forgot to reply, confirmed the above suggestion worked. However, those changes won't be present for anyone who has upgraded from a previous version and may not know they need to change these configuration settings.

Ideally graylog-server could inspect the config and alert the user if an applied config is missing this (where applicable).

[drewmiranda-gl]

Any ideas on how to handle this for people who upgrade from older versions and won't have the updated templates?

[zyxep]

I don't believe the sidecar collector updates the paths from the config file.
when i look in ps aux i don't see updated paths.. they are "hardcoded" to /var/lib/filebeat or something like that.

[mpfz0r]

@zyxep are you sure the collector was started by the sidecar itself? It sounds like you are running it via systemd or so.
That's not the idea, the sidecar is a process and configuration manager for collectors.
You install the beats package, but dont' register the service. The sidecar is gonna start / restart / configure it for you.

[mpfz0r]

Any ideas on how to handle this for people who upgrade from older versions and won't have the updated templates?

@drewmiranda-gl
We could adjust the migrations to not only create missing collectors (and their templates) but also modify existing ones.
We'd have to be smart so that it doesn't overwrite any user modifications of the template.

[zyxep]

@mpfz0r
It's installed through apt on ubuntu so yes it's systemd.
filebeat is installed by it's own repo since it's not delivered in the sidecar anymore :)

the sidecar requests configuration files from the graylog server just fine and it places it in /var/lib/graylog-sidecar/generated/<id of the configuration> in that folder there is a filebeat.conf file.. no logs data folder though.

my filebeat configuration does have this configured:

path:
  data: /var/lib/graylog-sidecar/generated/6489831245c6352ac3bb1c99/data
  logs: /var/lib/graylog-sidecar/generated/6489831245c6352ac3bb1c99/log

[mpfz0r]

See #17040
Fixed with #17246

[bloodhunterd]

What is the solution to the problem?

I had installed Graylog with version 5.2.0 and then upgraded to 5.2.1 where I first noticed the problem. I am currently using version 5.2.2 where the problem still exists.

As with @zyxep, neither the data nor the log folder is in the generated directories, only the filebeat.conf.

ls -ahl /var/lib/graylog-sidecar/generated/6576d8ba79911916e1e42457/
-rw------- 1 root root  521 Dec 11 12:59 filebeat.conf

ls -ahl /var/lib/graylog-sidecar/generated/6576d9fe79911916e1e424f7/
-rw------- 1 root root  523 Dec 11 12:59 filebeat.conf

The first configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
  - type: filestream
    paths:
      - /var/log/nginx/*-access.log

output.logstash:
  hosts: ["${user.graylog_host}:5044"]
  ssl.enable: true
  ssl.verification_mode: none

path:
  data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
  logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

tags:
  - nginx
  - nginx-access

The second configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
  - type: filestream
    paths:
      - /var/log/nginx/*-error.log

output.logstash:
  hosts: ["${user.graylog_host}:5044"]
  ssl.enable: true
  ssl.verification_mode: none

path:
  data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
  logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

tags:
  - nginx
  - nginx-error

When I create a new configuration and select filebeat on Linux, I get the following default configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}


output.logstash:
   hosts: ["${user.graylog_host}:5044"]
data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log
filebeat.inputs:

- type: filestream
  id: snort-filestream
  enabled: true
  paths:
    - /var/log/snort/alert_json.txt
    - /var/log/snort/appid-output.json
  parsers:
    - ndjson:
        target: "snort3"
        add_error_key: true
        overwrite_keys: true
  fields:
    event_source_product: snort3

- type: filestream
  id: zeek-filestream
  enabled: true
  paths:
    - /opt/zeek/logs/current
  parsers:
    - ndjson:
        target: "zeek"
        add_error_key: true
        overwrite_keys: true
  fields:
    event_source_product: zeek
- type: filestream
  id: apache-filestream
  enabled: true
  paths:
  - /var/log/apache2/access.log
  - /var/log/apache2/error.log
  - /var/log/httpd/access_log
  - /var/log/httpd/error_log

  fields_under_root: true
  fields:
      event_source_product: apache_httpd

There, data and log are specified without path, but it makes no difference if I adjust the configurations in this way.

Exiting: /usr/lib/graylog-sidecar/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)

[mpfz0r]

There, data and log are specified without path.

This is a known bug. It will be fixed with the next release.
So something like

path:
  data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
  logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

is the right config format.
The only reason I can think of, why the spoolDir isn't expanded is that you are running an older version of the sidecar.
Which one are you using?

[bloodhunterd]

I use Sidecar version 1.5.0.

So I will have to wait for Graylog version 5.2.3 to come out, before I can continue my work.

[thll]

So I will have to wait for Graylog version 5.2.3 to come out, before I can continue my work.

You can already manually fix the configuration that is applied to the sidecar in the Graylog UI at /system/sidecars/configuration by making sure that the format is correct.

There, data and log are specified without path, but it makes no difference if I adjust the configurations in this way.

To verify that the config has been applied correctly, check the generated config files at /var/lib/graylog-sidecar/generated/*/filebeat.conf on the machine running the sidecar. If the files are still wrong, maybe the collector configuration is not assigned correctly to the sidecar in the Graylog UI.

[bloodhunterd]

I don't quite follow you. In my opinion, the format is correct and @mpfz0r also writes that it is correct, or where is the mistake?

I have checked the Filebeat configurations on the server and they match the configuration in the graylog system. So they are applied correctly.

FYI: I am still using version 5.2.1 after all, as Graylog tells me that an update to version 5.2.2 is available, but that there is currently no package available for my system.

[starislv]

Hi!
On Graylog 5.2.2 with sidecar 1.5 on device, for me worked "path:"
path:
data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

default configuration doesn't have " path:" in there as You mentioned before. Adding path: before data: helped me.
So many hours lost because of this.

In my case I got generated folders with filebeat configuration file and data folder where 2 json files were located. They both were empty. Now they are not empty anymore :)
Thanks @mpfz0r mpfz0r

[bloodhunterd]

Many thanks @starislv for this tip! That was the solution to the problem!

I did have it written above it, but as a plural, paths instead of path. Now it works wonderfully!