FQDN local hostname

[rgerhards]

Expected behavior

When preserveFQDN="on" ist set, local hostname should also have FQDN.

Actual behavior

After rsyslog start, local hostname is short name only. It changes to FQDN after the first HUP.

Analysis

This issue is a bit tricky. The problem is rooted in that we need to set the hostname very early, before we process the config. We need to do this, so that we can emit error, warning, ... messages when reading the config.

Now, the default is not to preserve the FQDN. The default applies in early stage as we do not have a config yet. This means the domain will be stripped.

After we have read the config and activated it, rsyslog knows the preserveFQDN value and can apply it. As such, the next time the hostname is set, during HUP, the domain is included.

We can re-initialize the hostname as soon as the new config becomes active. That way, the startup message will have the new hostname (with FQDN).

BUT: any (error) message before config activation will still contain the short name only (because we have not yet the info that we need to preserve the FQDN).

To make matters worse, the hostname also depends on whether we have DNS enabled or not. Many users disable DNS at least early in startup, so that rsyslog can be started before DNS (if DNS is not present, we would timeout). In order to obtain the full name (reliably), we need use DNS. I have tested without DNS, and the gethostname() API always returns the non-FQDN name on my test environments. Note that the API description is not very specifiy in regard to FQDN - there may be platforms where it returns FQDN, but this is unspecified. My Ubuntu 20.04 is an example where it does not. If gethostname() does not return the FQDN, we need to obtain it by DNS. This is actually the logic inside our own "rsyslog hostname resolver".

I see one solution to avoid this: make that property configurable via an environment variable (e.g. RSYSLOG_CONF_FORCE_PRESERVE_FQDN=on). You would need to specify that environment variable in the system's startup script or systemd unit file. This is still a bit of a hassle, but it is at least a bit clean. Note that, when selected, you need to ensure that DNS is up before rsyslog is started.

An IMHO suboptimal solution would be to add a positon-dependent config directive to do the same that the environment variable does. This is ugly because it introduces position-dependendy to the config again, which we worked hard to avoid. The reason for that was confusion that is created among users. Also, you would need to ensure this directive is really the first one inside the config. Further, error messages occurring before the config can be read at all (e.g. some issue trying to access the config file) would still be reported by the short name - simply because that is the only one we know by then. Note that this problem does NOT exist with the environment variable approach.

If it would be OK to have the FQDN only after config processing (and message from config processing), we could just re-init the hostname after that processing. This is my preferred approach. @davidelang what do you think?

Environment

• rsyslog version: <= 8.2208.0 scheduled stable build
• platform: all

[davidelang]

prior to the config being parsed, the error messages can only go to stdout correct? (where journald could pick them up and feed them back into rsyslog I guess to be sent elsewhere later) or do they get stored in a queue to be delivered after the config is read? If this is the case, a pass could be made through the queue after the config is read to update the hostname. if it's just sent to stdout, I'm not worried about the hostname being the short one. I agree that we don't want a position dependent parameter in the config, but what about a command line parameter?

…

On Wed, 31 Aug 2022, Rainer Gerhards wrote: If it would be OK to have the FQDN only after config processing (and message from config processing), we could just re-init the hostname after that processing. This is my preferred approach. @davidelang what do you think?

[rgerhards]

Thx @davidelang for the very useful thoughts. I audited the code this morning. It's indeed a bit complicated as well.

Depending on circumstances, messages go out to stdout/stderr. This is mostly the case if something goes fatally wrong. For just "regular" config errors, message go indeed to a rsyslog internal queue and will be re-injected once startup is done.

That said, it looks like we indeed have a way of fixing the hostname. I will carefully look into that. I must differentiate the startup case from HUP. With change on HUP, the in-queue hostname must stay, but with startup, it must be changed.

I'll keep everyone posted.

[rgerhards]

@davidelang Thx again for your thoughts. I was able to change the code so that the config-set hostname (format) now also applies to early boot messages including config processing. See #4977 .

Messages to stderr do not include the hostname, so this is actually a no issue.

[mowgli]

The patch for that was in my update today from 8.2208.0-1devuan1 to 8.2210.0-1devuan1.

After that, the hostname in logfile is gone and is replaced by '[localhost]', what is breaking nearly everything that is analyzing the log files. Even more, it is not possible anymore to see from which host that entry is.

[rgerhards]

That is very strange.

Please create a debug log. Instructions: https://www.rsyslog.com/doc/v8-stable/troubleshooting/howtodebug.html

Be sure that the problem in question occurs at least once during the debug run. Once this is done, please post the debug log (e.g. via pastbin or github). After that, we can review the contents and check how exactly rsyslog processing worked.

[mowgli]

rsyslog.debug.txt
rsyslog.terminal.txt

Direct after start I get the following line in messages:

Oct 20 13:13:10 [localhost] rsyslogd: [origin software="rsyslogd" swVersion="8.2210.0" x-pid="3973" x-info="https://www.rsyslog.com"] start

[mowgli]

Small note: As pointed out in #4825, after pkill -HUP rsyslogd it logs the host name as it should.

[hrak]

Seeing similar behavior in rsyslog-8.2210.0-0adiscon1focal1 on Ubuntu. In our case PreserveFQDN is off. Starting from boot, rsyslog logs with hostname [localhost]. Sending a HUP to rsyslogd indeed makes it log proper hostname. Restarting rsyslog makes it log as [localhost] again until HUP'ed.

Looked a bit at the related PR and isn't this line supposed to read

if(strcmp((char*)pThis->pMsg->pszHOSTNAME, "[localhost]")) { (mind the absence of ! preceding strcmp)

[rgerhards]

I see what happens - I overlooked a border case, also during testing.

The call to actually set the local host name only happens when global parameters are defined (the testbench unfortunately needs this always).

Please add

global(internalmsg.severity="info")

to rsyslog.conf. This is kind of a no-op, as it only sets the severity to the default value. However, now a global param is given and so the local hostname should be queried. So it is a work-around you can use immediately.

Please let me know if that solves the issue for you.

[rgerhards]

@hrak thx for taking the time to look at the PR

Looked a bit at the related PR and isn't this line supposed to read

if(strcmp((char*)pThis->pMsg->pszHOSTNAME, "[localhost]")) { (mind the absence of ! preceding strcmp)

It's not, because as given (with !) it checks for equality with "[localhost]" and, if so, replaces it with the correct hostname.

You'll see the real cure in upcoming PR. Note the work-around above.

[mowgli]

I see what happens - I overlooked a border case, also during testing.

The call to actually set the local host name only happens when global parameters are defined (the testbench unfortunately needs this always).

Please add

global(internalmsg.severity="info")

to rsyslog.conf. This is kind of a no-op, as it only sets the severity to the default value. However, now a global param is given and so the local hostname should be queried. So it is a work-around you can use immediately.

Please let me know if that solves the issue for you.

Yep, it works as a workaround.

[rgerhards]

Yep, it works as a workaround.

thx for the confirmation.

PR is #5004