imtcp Doesn't Listen Until Restarted, Only When Using SSL

[rafecolton]

Expected behavior

Machine A forwards over TCP on port 518 using SSL with the omfwd module and the ossl stream driver. Machine B is listening on port 518 using imtcp with the ossl stream driver, and both machines have proper certificates. Expected behavior is that when both rsyslog instances have started up, logs will flow just fine.

Actual behavior

On Machine B, the receiver, netstat reveals that rsyslog is not even listening on port 518 until after rsyslog has been restarted one time.

Before restarting rsyslog on the receiver, I get this error message on the sender:

[2020-03-11T14:49:04.268156+00:00 agg-log1.local rsyslogd: cannot connect to log1.local:518: Connection refused [v8.1911.0 try https://www.rsyslog.com/e/2027 ]

After restarting the rsyslog service on the receiver, logs flow just fine—I see the output of the sender's logs in /var/log/syslog on the receiver. What's especially strange is that this only happens if I use SSL. If I comment out the SSL block of both the sender and receiver and reboot / re-provision the receiver, it listens and receives on initial boot on expected.

Steps to reproduce the behavior

Provision new machine using configs described below and run tail -F /var/log/syslog. On initial boot of the receiver, sender starts logging the above message. After restarting rsyslog on the receiver, sender's logs can be seen in /var/log/syslog.

In case it's relevant, I included my systemd config. I thought this might be related to it rsyslog waiting on the network, so I tried adding the following with no effect:

After=network-online.target
Wants=network-online.target

NOTE: If you comment out the TLS config parts below, it works just fine on initial boot:

  # TLS configs
  streamDriver="ossl"
  streamDriverMode="1"
  streamDriverAuthMode="x509/certvalid"

Environment

• rsyslog version:

  rsyslogd  8.1911.0 (aka 2019.11) compiled with:
          PLATFORM:                               x86_64-pc-linux-gnu
          PLATFORM (lsb_release -d):
          FEATURE_REGEXP:                         Yes
          GSSAPI Kerberos 5 support:              No
          FEATURE_DEBUG (debug build, slow code): No
          32bit Atomic operations supported:      Yes
          64bit Atomic operations supported:      Yes
          memory allocator:                       system default
          Runtime Instrumentation (slow code):    No
          uuid support:                           Yes
          systemd support:                        Yes
          Config file:                            /etc/rsyslog.conf
          PID file:                               /var/run/rsyslogd.pid
          Number of Bits in RainerScript integers: 64
  
  See https://www.rsyslog.com for more information.
  

• platform: Ubuntu 16.04.6 LTS
• for configuration questions/issues, include rsyslog.conf and included config files

Sender:

# ...
action(
  name="send-to-log1-local"
  type="omfwd"
  target="log1.local"
  protocol="tcp"
  port="518"
  template="RSYSLOG_ForwardFormat"

  # TLS configs
  streamDriver="ossl"
  streamDriverMode="1"
  streamDriverAuthMode="x509/certvalid"

  action.resumeretrycount="-1"
  action.resumeIntervalMax="300"

  # queue config
  queue.type="LinkedList"
  # ...
)
# ...

Receiver:

# ...
# Enables log receiver module
module(
  load="imtcp"
  MaxSessions="500"
  StreamDriver.Name="ossl"
  StreamDriver.Mode="1"
  StreamDriver.AuthMode="x509/certvalid"
)

# listen on port and log to file
input(
  type="imtcp"
  port="518"
  ruleset="imtcp_518"
)
ruleset(
  name="imtcp_518"
  queue.type="LinkedList"
){
  action(
    type="omfile"
    file="/var/log/syslog"
  )
}
# ...

Systemd Service Config:

[Unit]
Description=System Logging Service for Rsyslog
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=https://www.rsyslog.com/doc/
# After=network-online.target # tried this and the line below, did not help
# Wants=network-online.target # tried this and the line above, did not help

[Service]
Type=notify
EnvironmentFile=-/etc/default/rsyslog
ExecStart=/usr/sbin/rsyslogd -n -i/var/run/rsyslogd-518-local.pid -f /etc/rsyslog-.conf $RSYSLOGD_OPTIONS
ExecReload=/sbin/start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/rsyslogd-518-local.pid --name rsyslogd
UMask=0066
StandardOutput=null
Restart=on-failure

# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog-518-local.service

[rafecolton]

Bump. @rgerhards - any thoughts on this?