Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[all beats] Remove BEAT_LOG_OPTS from the systemd unit file #12024

Open
magnuslarsen opened this issue May 2, 2019 · 10 comments

Comments

Projects
None yet
7 participants
@magnuslarsen
Copy link

commented May 2, 2019

OS Version: I've only tested DEB installations on Ubuntu 18.04, but I'd imagine it happens on all systemd enabled Linux systems
Beat Version: 7.0.0, 7.0.1

In a unit file for any beat BEAT_LOG_OPTS is defaulted to -e
This will make logging options in the configuration file obsolete/ignored.

Examples:
Filebeat systemd unit file (snippet):

[Service]
Environment="BEAT_LOG_OPTS=-e"

filebeat.yml (snippet):

logging:
  level: info
  to_files: true
  to_syslog: false
  json: true
  files:
    path: '/var/log/filebeat'
    name: 'filebeat'
    keepfiles: '7'
    permissions: '0644'

Expected result:
Logging to files in the /var/log/filebeat directory

Actual result:
Logging to /var/log/syslog

Why not keep all the configurations in the configuration file, instead of having it split up in 2 files: systemd drop in file for logging options, and a configuration file for all other options?

Blindly removing the environmental variable BEAT_LOG_OPTS results in the configuration file being used for everything, and for logging to happen in the expected result.
Is there ANY reason not to do this? It's both much simpler AND much cleaner for the user.

@eric-price

This comment has been minimized.

Copy link

commented May 3, 2019

Second this. I had to remove "Environment="BEAT_LOG_OPTS=-e"" from my service file to get logging to go where I specify in the config file. I'm using an rpm on Centos 7.

@jsoriano jsoriano added the discuss label May 7, 2019

@jsoriano

This comment has been minimized.

Copy link
Member

commented May 7, 2019

We wanted to make beats to log by default to stdout/stderr on systemd-based OSs, so logs are collected by journald, for coherence with other services, see #8942.
If having this setting in the unit file is being problematic, an alternative we could consider would be to set this in the default configuration file instead.

@magnuslarsen

This comment has been minimized.

Copy link
Author

commented May 7, 2019

Wouldn't that leave duplicate logs? Making it send to stdout/stderr would log it to /var/log/syslog and journald. Journalbeat picks it up from journald, Filebeat's System module per default picks it up from /var/log/syslog

In #8942, i see this very issue was already discussed. The problem is, that if you don't have Journalbeat installed, your logs aren't going the place where you specify, leaving it in /var/log/syslog regardless of the configuration options, which means--in my case at least--that it didn't work as intended.

Having the default configuration file point to stdout/stderr would make a lot of sense. Then its clear for the user where it's going, and makes the change visible.

Alternatively, the beat could fail/log that the logging options in the configuration file (if any) are overridden by the systemd unit file. Just making it clearer would be ideal.

@watsonian

This comment has been minimized.

Copy link

commented Jun 4, 2019

I just ran into this problem. Spent about an hour trying to figure out why the logging config options were being ignored before realizing -e was overriding them. We have lots of customers who run just Filebeat to ship logs without any other beats and this behavior was very confusing. Particularly when you specify logging.to_syslog: false. The config file was clearly being read (-path.logs in the executing command was reflecting the config setting), but it was ignoring these other logging settings.

@inthecloud247

This comment has been minimized.

Copy link

commented Jun 12, 2019

This is incredibly confusing, and I ran into the same issue as @watsonian and similarly wasted a ton of time trying to debug. If there's a setting like this that overrides the other explicit config options, it would be helpful to at least update the docs to give people a heads-up.

@indreek

This comment has been minimized.

Copy link

commented Jun 17, 2019

Same here!

@indreek

This comment has been minimized.

Copy link

commented Jun 17, 2019

Another point is that if you collect metrics you have no good reason to keep duplicated metrics every x seconds in syslog + elastic. It's a waste of space. So i would like to keep my metrics in separate file and clear them in certain period.

@sanvila

This comment has been minimized.

Copy link

commented Jun 17, 2019

The problem with having -e in the systemd unit is that if you don't like it, you have to change it every time the package is upgraded, because /lib/systemd/system/filebeat.service is overwritten on upgrades.

In theory, it would also be possible to make /etc/systemd/system/multi-user.target.wants/filebeat.service a real file instead of a symlink, but then people would have to track changes to the original file, if any, to keep them in sync.

The packaging system already takes care of preserving changes in /etc/filebeat/filebeat.yml when the package is upgraded, so it would be really nice if that was the only file that had to be modified/customized.

@jsoriano

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

The problem with having -e in the systemd unit is that if you don't like it, you have to change it every time the package is upgraded, because /lib/systemd/system/filebeat.service is overwritten on upgrades.

@sanvila systemd unit files shouldn't be directly modified, you can create a drop-in file to override values.

@sanvila

This comment has been minimized.

Copy link

commented Jun 17, 2019

Ok, didn't know, thanks a lot.

But even in such case, I still think it's slightly better to modify a single file than having to modify a file and create another one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.