Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syslog-ng.service startup failure on #2667

Open
MikeLim7 opened this issue Apr 10, 2019 · 8 comments

Comments

@MikeLim7
Copy link

commented Apr 10, 2019

syslog-ng systemd

Version of syslog-ng

3.20 for RHEL 7
(syslog-ng-3.20.1.233.g502b0f0-1.el7.x86_64)

output of syslog-ng --version

syslog-ng 3 (3.20.1.233.g502b0f0)
Config version: 3.20
Installer-Version: 3.20.1.233.g502b0f0
Revision:
Compile-Date: Apr 8 2019 11:23:43
Module-Directory: //usr/lib64/syslog-ng
Module-Path: //usr/lib64/syslog-ng
Include-Path: /usr/share/syslog-ng/include
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml,afamqp,hook-commands,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

Platform

CentOS 7.6

Issue : Systemd startup errors

Whenever a RHEL/CentOS VM is booted up, syslog-ng daemon fails to start. This can be fixed by configuring syslog-ng.service to start after network daemon

syslog-ng.service - System Logger Daemon
Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit) since Fri 2019-04-05 12:57:24 +08; 4 days ago
Docs: man:syslog-ng(8)
Main PID: 5322 (code=exited, status=2)
Status: "Starting up... (Fri Apr 5 12:57:24 2019"

Apr 05 12:57:24 server.com systemd[1]: Failed to start System Logger Daemon.
Apr 05 12:57:24 server.com systemd[1]: Unit syslog-ng.service entered failed state.
Apr 05 12:57:24 server.com systemd[1]: syslog-ng.service failed.
Apr 05 12:57:24 server.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart.
Apr 05 12:57:24 server.com systemd[1]: Stopped System Logger Daemon.
Apr 05 12:57:24 server.com systemd[1]: start request repeated too quickly for syslog-ng.service
Apr 05 12:57:24 server.com systemd[1]: Failed to start System Logger Daemon.
Apr 05 12:57:24 server.com systemd[1]: Unit syslog-ng.service entered failed state.
Apr 05 12:57:24 server.com systemd[1]: syslog-ng.service failed.

Steps to reproduce

Install syslog-ng on CentOS 7
Enable syslog-ng : systemctl enable syslog-ng
Reboot CentOS 7
View syslog-ng daemon status : systemctl status syslog-ng

Suggestion

Modify /usr/lib/systemd/system/syslog-ng.service and add After=network.target to [Unit] section

[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)
After=network.target

[Service]
Type=notify
ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
ExecReload=/bin/kill -HUP $MAINPID
EnvironmentFile=-/etc/default/syslog-ng
EnvironmentFile=-/etc/sysconfig/syslog-ng
StandardOutput=journal
StandardError=journal
Restart=on-failure

[Install]
WantedBy=multi-user.target

@MikeLim7

This comment has been minimized.

Copy link
Author

commented Apr 10, 2019

syslog-ng systemd

Version of syslog-ng

3.20 for RHEL 7
(syslog-ng-3.20.1.233.g502b0f0-1.el7.x86_64)

output of syslog-ng --version

syslog-ng 3 (3.20.1.233.g502b0f0)
Config version: 3.20
Installer-Version: 3.20.1.233.g502b0f0
Revision:
Compile-Date: Apr 8 2019 11:23:43
Module-Directory: //usr/lib64/syslog-ng
Module-Path: //usr/lib64/syslog-ng
Include-Path: /usr/share/syslog-ng/include
Available-Modules: add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml,afamqp,hook-commands,examples
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

Platform

CentOS 7.6

Issue : Systemd startup errors

Whenever a RHEL/CentOS VM is booted up, syslog-ng daemon fails to start. This can be fixed by configuring syslog-ng.service to start after network daemon

syslog-ng.service - System Logger Daemon
Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit) since Fri 2019-04-05 12:57:24 +08; 4 days ago
Docs: man:syslog-ng(8)
Main PID: 5322 (code=exited, status=2)
Status: "Starting up... (Fri Apr 5 12:57:24 2019"
Apr 05 12:57:24 server.com systemd[1]: Failed to start System Logger Daemon.
Apr 05 12:57:24 server.com systemd[1]: Unit syslog-ng.service entered failed state.
Apr 05 12:57:24 server.com systemd[1]: syslog-ng.service failed.
Apr 05 12:57:24 server.com systemd[1]: syslog-ng.service holdoff time over, scheduling restart.
Apr 05 12:57:24 server.com systemd[1]: Stopped System Logger Daemon.
Apr 05 12:57:24 server.com systemd[1]: start request repeated too quickly for syslog-ng.service
Apr 05 12:57:24 server.com systemd[1]: Failed to start System Logger Daemon.
Apr 05 12:57:24 server.com systemd[1]: Unit syslog-ng.service entered failed state.
Apr 05 12:57:24 server.com systemd[1]: syslog-ng.service failed.

Steps to reproduce

Install syslog-ng on CentOS 7
Enable syslog-ng : systemctl enable syslog-ng
Reboot CentOS 7
View syslog-ng daemon status : systemctl status syslog-ng

Suggestion

Modify /usr/lib/systemd/system/syslog-ng.service and add After=network.target to [Unit] section

[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)
After=network.target
[Service]
Type=notify
ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid
ExecReload=/bin/kill -HUP $MAINPID
EnvironmentFile=-/etc/default/syslog-ng
EnvironmentFile=-/etc/sysconfig/syslog-ng
StandardOutput=journal
StandardError=journal
Restart=on-failure
[Install]
WantedBy=multi-user.target

The After=network.target method is also used by other daemons such as httpd (apache) and mariadb .

@erempel

This comment has been minimized.

Copy link

commented Apr 18, 2019

the rsyslog packaged by Redhat includes a little more in the Unit requirements

[Unit]
Wants=network.target network-online.target
After=network.target network-online.target

Since this is a superset of what MikeLim7 is asking for, I think this would be a little more complete for the systemd way of life.

It would also be more correct to use the systemd local configuration override method
/etc/systemd/system/syslog-ng.service.d/local.conf
and use the
Environment=SYSLOGNG_PIDFILE=/var/run/syslogd.pid
Environment=SYSLOGNG_OPTS=""
ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p $SYSLOGNG_PIDFILE

@bazsi

This comment has been minimized.

Copy link
Collaborator

commented Apr 21, 2019

@czanik can you please check this fix in the contract of rhel 7 packaging?

@erempel @MikeLim7 If it's the right way, can you please send a PR for fixing it?

Thanks.

@MikeLim7

This comment has been minimized.

Copy link
Author

commented Apr 21, 2019

Issue is fixed once the Wants= and After= directive are added; syslog-ng now start up properly every time we reboot or power on.

Agreed with @erempel, we can use RHEL7 rsyslogd as a template for completeness.

@czanik

This comment has been minimized.

Copy link
Member

commented Apr 23, 2019

Originally I had some fears that this change might affect some users negatively. But if the service file for rsyslog has these settings, it's most likely not problematic :)

Added:

Wants=network.target network-online.target
After=network.target network-online.target

to syslog-ng.service for my git snapshot package, did some testing, and it worked fine.

You can test it yourself by using packages from https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/ If I receive positive feedback, I'll also add it to my 3.20 packages and discuss the changes for the official EPEL package (still at version 3.5) with the co-maintainers.

@MikeLim7

This comment has been minimized.

Copy link
Author

commented Apr 23, 2019

Noted and thanks.

Guess can only test once I see a new el7 build at [https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng-githead/builds/]

Anyway, this is the full rsyslog.service for discussion (if needed)

[Unit]
Description=System Logging Service
;Requires=syslog.socket
Wants=network.target network-online.target
After=network.target network-online.target
Documentation=man:rsyslogd(8)
Documentation=http://www.rsyslog.com/doc/

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStart=/usr/sbin/rsyslogd -n $SYSLOGD_OPTIONS
Restart=on-failure
UMask=0066
StandardOutput=null
Restart=on-failure

[Install]
WantedBy=multi-user.target
;Alias=syslog.service

@erempel

This comment has been minimized.

Copy link

commented Apr 23, 2019

Considering the "systemd way" and weighing backwards compatibility for migrating from Redhat 6 I think we can have the best of both worlds.

The current unofficial RPMs for Redhat 6 have the /etc/init.d/syslog-ng file defined with

SYSLOGNG_PID_FILE="/var/run/syslog$SYSLOGNG_SUFFIX.pid"
SYSLOGNG_CONF_FILE="/etc/syslog-ng/syslog-ng$SYSLOGNG_SUFFIX.conf"
SYSLOGNG_PERSIST_FILE="/var/lib/syslog-ng/syslog-ng$SYSLOGNG_SUFFIX.persist"
SYSLOGNG_CONTROL_FILE="/var/lib/syslog-ng/syslog-ng$SYSLOGNG_SUFFIX.ctl"
SYSLOGNG_SUBSYSTEM="syslog-ng$SYSLOGNG_SUFFIX"
SYSLOGNG_OPTIONS=""

with site specific overrides defined in the file /etc/sysconfig/syslog-ng

[ -e /etc/sysconfig/syslog-ng$SYSLOGNG_SUFFIX ] && . /etc/sysconfig/syslog-ng$SYSLOGNG_SUFFIX

The systemd way of doing things is to define values in the service file
/usr/lib/systemd/system/syslog-ng.service

[Service]
Environment=SYSLOGNG_PID_FILE="/var/run/syslog-ng.pid"
Environment=SYSLOGNG_CONF_FILE="/etc/syslog-ng/syslog-ng.conf"
Environment=SYSLOGNG_PERSIST_FILE="/var/lib/syslog-ng/syslog-ng.persist"
Environment=SYSLOGNG_CONTROL_FILE="/var/lib/syslog-ng/syslog-ng.ctl"
Environment=SYSLOGNG_OPTIONS=""

with site specific overrides in the file
/etc/systemd/system/syslog-ng.service.d/local.conf

[Service]
Environment=SYSLOGNG_PID_FILE="/my/local/pid/file/override"

Systemd also provides for using an environment configuration file for compatibility with the old way of overrides in
/etc/sysconfig/syslog-ng

EnvironmentFile=-/etc/sysconfig/syslog-ng

By doing both in the service file, we can support the "correct" systemd way AND be backwards compatible with the previous init.d files from Redhat 6 (System V init).

My recommended systemd service file is attached
syslog-ng.service.txt

@gaborznagy

This comment has been minimized.

Copy link
Contributor

commented Jul 24, 2019

I see that @czanik's COPR repos are using this since version 3.21

Also found that network.target is used in opensuse builds:

Can we agree that this can be put into packaging/rhel/syslog-ng.service?
And possibly to packaging/debian/syslog-ng.systemd.
What do you think @czanik @lbudai @bazsi?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.