rsyslog v 8.24 taking long time to restart / start

[rajubi1981]

Expected behavior

rsyslog should start in minimal time.

Actual behavior

It is taking more than and show the result as below.
[root@local-host ~]# systemd-analyze blame | grep rsys
20.022s rsyslog.service

[root@local-host ~]# time service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service

real 0m29.972s
user 0m0.013s
sys 0m0.011s

But , if we install it in non bare metal then it takes less time.
[root@local-host ~]# systemd-analyze blame | grep rsys
3ms rsyslog.service

[root@localhost ~]# time service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service

real 0m0.031s
user 0m0.014s
sys 0m0.010s
[root@localhost ~]#

We tried upgrading the rsyslog to version rsyslog-8.24.0-16.el7_5.4.x86_64.rpm which is latest as per RHEL RPM repo, and even we that the issue remains.

Then , we tried installing the latest rsyslog (rsyslog-8.35.0-3.el7.x86_64.rpm ) from the link. @ http://rpms.adiscon.com/v8-stable/epel-7/x86_64/RPMS/ and with that, rsyslog does not start. It gives error.

Steps to reproduce the behavior

Install rsyslog in bare metal VM and install rsyslog.

Environment

[root@localhost ~]# rsyslogd -v
rsyslogd 8.24.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.

Can you please help us.
a) Why does it take more time to start rsyslog in bare metal VM ? How should I optimize it.

[davidelang]

could we see your full rsyslog.conf (and any included files)? It could be that you are having name lookup issues, but anything is pure guesswork without seeing the config. David Lang

[rajubi1981]

Hi David,
I am using the default rsyslog.conf file.

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog
# Log cron stuff
cron.*                                                  /var/log/cron
# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local0.*                                                /var/log/boot.log

Post that on restart of rsyslog gives .

[root@DontUseMe ~]# time service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service

real 0m16.618s
user 0m0.016s
sys 0m0.008s
[root@DontUseMe ~]#

The issue is seen with bare metal VM.

Also want to know, how to reload rsyslog.conf changes without restart of rsyslog

[rajubi1981]

One additional thing. If I try to run the rsyslogd in debug mode,

6763.827556657:main thread : rsyslogd 8.24.0 startup, module path '', cwd:/root
6763.827639374:main thread : caller requested object 'net', not found (iRet -3003)
6763.827650321:main thread : Requested to load module 'lmnet'
6763.827655325:main thread : loading module '/usr/lib64/rsyslog/lmnet.so'
6763.827748024:main thread : module lmnet of type 2 being loaded (keepType=0).
6763.827769569:main thread : entry point 'isCompatibleWithFeature' not present in module
6763.827772745:main thread : entry point 'setModCnf' not present in module
6763.827774541:main thread : entry point 'getModCnfName' not present in module
6763.827776041:main thread : entry point 'beginCnfLoad' not present in module
6763.827778959:main thread : source file conf.c requested reference for module 'lmnet', reference count now 1
6763.827793385:main thread : rsyslog runtime initialized, version 8.24.0, current users 1
6763.827827875:main thread : source file rsyslogd.c requested reference for module 'lmnet', reference count now 2
6773.839811777:main thread : GenerateLocalHostName uses 'HP-G9'
6773.839843393:main thread : omfile: using transactional output interface.
6773.839874577:main thread : module builtin:omfile of type 1 being loaded (keepType=0).

We can see from 6763 it jumped to 6773. So it waited for that much time before proceeding further.

[davidelang]

is DNS working at the point that rsyslog is starting, this looks like a DNS timeout as it's trying to find it's hostname We have seen this before. David Lang

[davidelang]

Is your network up and running at the point that rsyslog is trying to start? if not, it could be failing at that point.

[rajubi1981]

Hi David,
Thanks for the help.

[rajubi1981]

We are seeing a strange thing in RHEL7.4
When we set the static hostname with only host name then rsyslog takes more time to start

[root@UseMe123 init.d]# hostnamectl
Static hostname: UseMe123
Pretty hostname: UseMe123.raju.com
Icon name: computer-vm
Chassis: vm
Machine ID: 5ff453b84f8b7de4ec0587e861486e7d
Boot ID: 62408682525f44548cdae1f4a2487ad2
Virtualization: kvm
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.4:GA:server
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64

[root@UseMe123 init.d]# time service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service

real 0m10.043s
user 0m0.015s
sys 0m0.011s

Then when we set the hostname.domainname for static hostname as
hostnamectl set-hostname "UseMe123.raju.com" --static
[root@UseMe123 init.d]# hostnamectl
Static hostname: UseMe123.sunil.com
Transient hostname: UseMe123
Icon name: computer-vm
Chassis: vm
Machine ID: 5ff453b84f8b7de4ec0587e861486e7d
Boot ID: 62408682525f44548cdae1f4a2487ad2
Virtualization: kvm
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.4:GA:server
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64
[root@UseMe123 init.d]#

[root@UseMe123 init.d]# time service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service

real 0m0.032s
user 0m0.014s
sys 0m0.011s

Then time taken is less.
Also, when transient hostname is configured then also rsyslogd restart takes long time.
Can you please help me.

[rajubi1981]

The issue seen is we need to configure as hostname.domainname for /etc/hostname. if we configure only hostname then again it takes more time to restart.

Also, if transient name is present in hostnamectl status then also it takes more time to start.

[root@UseMe1234 ~]# hostnamectl
Static hostname: useme1234.raju.com
Pretty hostname: UseMe1234.raju.com
Transient hostname: UseMe1234
Icon name: computer-vm
Chassis: vm
Machine ID: 5ff453b84f8b7de4ec0587e861486e7d
Boot ID: 8d1e15c91ecd40dc9d8ef49b0d7ae35c
Virtualization: kvm
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.4:GA:server
Kernel: Linux 3.10.0-693.el7.x86_64
Architecture: x86-64

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/ch-configure_host_names#sec-Recommended_Naming_Practices

[davidelang]

what is in /etc/hosts? it looks like it's trying to resolve the pretty hostname, but it's trying to do so by going out to the DNS server, and that isn't up yet so you have to wait for it to timeout. if /etc/hosts has both the short and long names in it you should not have this problem. I'm not familiar with hostnamectl and what it actually does. David Lang

[rajubi1981]

[root@UseMe1234 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@UseMe1234 ~]#

[root@UseMe1234 ~]# cat /etc/hostname
useme1234
[root@UseMe1234 ~]# cat /etc/resolv.conf
search raju.com
nameserver <DNS_IP4>
[root@UseMe1234 ~]#

[root@UseMe1234 ~]# cat /proc/sys/kernel/hostname
UseMe1234
[root@UseMe1234 ~]#

[davidelang]

add UseMe1234 and UseMe1234.raju.com to your /etc/hosts file. rsyslog is trying to lookup your hostname and since it's not in /etc/hosts, it's going to DNS

…

On Tue, 17 Jul 2018, rajubi1981 wrote: Date: Tue, 17 Jul 2018 12:14:39 -0700 From: rajubi1981 ***@***.***> Reply-To: rsyslog/rsyslog <reply+0021b24f9684cbcd4ef0410a7803992217326c57b8e176ba92cf000000011766021 ***@***.***> To: rsyslog/rsyslog ***@***.***> Cc: David Lang ***@***.***>, Comment ***@***.***> Subject: Re: [rsyslog/rsyslog] rsyslog v 8.24 taking long time to restart / start (#2838) ***@***.*** ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 ***@***.*** ~]# ***@***.*** ~]# cat /etc/hostname useme1234 ***@***.*** ~]# cat /etc/resolv.conf search raju.com nameserver <DNS_IP4> ***@***.*** ~]# ***@***.*** ~]# cat /proc/sys/kernel/hostname UseMe1234 ***@***.*** ~]#

[rajubi1981]

One question, if we do not add in the /etc/hosts then in DNS, how do we need to configure to get the hostname resolved.

[davidelang]

On Tue, 17 Jul 2018, rajubi1981 wrote: One question, if we do not add in the /etc/hosts then in DNS, how do we need to configure to get the hostname resolved.

you need to have the network start before rsyslog so that DNS can function.

[dman777]

Why does rsyslog need DNS? I have mine only using localhost.

[davidelang]

by default it does a reverse lookup on the IP addresses of anything it receives also, if you configure it to send anywhere, it's really useful to be able to specify a hostname instead of just an IP address during startup, it tries to lookup the hostname of the machine it's running on, which triggers a DNS lookup by default David Lang

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.